Lead Data Engineer

Are you an experienced and successful Data Engineer looking for a new, challenging and rewarding career? If so, we have the perfect job for you!

The Data Hub is a centre of excellence for data management and advanced analytics in HM Treasury. Working closely with partners inside and outside of government, the Hub is responsible for the implementation of the Treasury’s data strategy, developing the organisation’s data capability and driving data-led transformation across the department. The Hub also provides strategic coordination and support to teams across the department, bringing data tools and technologies to bear on new and existing data sources. It applies innovative methods to enable rich analysis and support HMT’s departmental objectives, delivering new insights and improving decision-making. To do this, the Hub is organised into two functions:

• Data Management – leads the department in developing its data management architecture, engineering, and data literacy. This includes developing systematic and structural approaches using data science techniques and ensuring that officials at all levels have access to data-related learning and development opportunities.
• Advanced Analytics – a flexible pool of data scientists who are responsible for leading the development of the department’s AI capabilities, and can identify, initiate, and support high-value data science projects across the department.

In this role:

• You will be acquiring data sets from multiple sources at varying levels of maturity. You will have a demonstrable understanding of how to expose data from systems (e.g. from APIs), link data from multiple systems, and deliver streaming services. You will collaborate with others to identify and plan these services, and to review specifications.
• You will maintain and develop AI products used internally and develop new products in line with HMT’s AI roadmap. You will leverage your expertise in Python, cloud systems and AI (specifically LLMs and embeddings) to achieve this. You will ensure the products run reliably and react to any issues rapidly.
• You can produce data models and understand where to use different types of data models. You have experience of a wide range of data engineering tools. You understand industry-recognised data-modelling patterns, how to establish and use agreed standards and tools, design, code, and test programs and scripts for data engineering. You can develop resilient, scalable, and future-proof services for to meet user needs.
• You will actively scope and carry out other projects including web app development and data engineering. You will explore opportunities to work innovatively, recognising, identifying and exploiting opportunities to ensure efficient and effective performance of the organisation, and to support and enable that you will influence the data architecture in place. You can explore new ways of conducting business and organisational processes.
• You will promote best practice in data engineering including acquisition, storage and sharing of data for the benefit of modelling and policy development, working with the Hub to provide overall technical leadership and development opportunities across HMT in data and data science.

As a lead data engineer, you will work on a variety of data engineering projects and own several AI products, maintaining and developing them in response to user feedback. These AI products are used by most HMT staff: improving them will increase productivity across the department.

You will have strong experience using Python for data engineering and web app development; expertise in SQL; considerable experience at least one major Cloud provider (Azure is preferred, however Google Cloud or AWS are acceptable); experience successfully implementing a product and managing customer relationships, acting on feedback and other good product management practices and experience using LLM and Embedding APIs, with a strong understanding of how both work.

We offer a range of benefits, including:

• 25 days annual leave (rising to 30 after 5 years), plus 8 public holidays and the King’s birthday (unless you have a legacy arrangement as an existing Civil Servant).
• Flexible working patterns (part-time, job-share, condensed hours)
• Generous parental and adoption leave packages
• Access to a generous Defined Benefit pension scheme with employer contributions of 28.97%
• Access to a cycle-to-work salary sacrifice scheme and season ticket advances
• A range of active staff networks, based around interests (e.g. analysts, music society, sports and social club) and diversity

We are an equal opportunities employer and welcome applications from all qualified candidates. If you need any reasonable adjustments to take part in the selection process, please let us know.