Lead Data Engineer
Data Insights
Permanent
Office location: ITV White City
Hiring range: £85,000 - £95,000
Your work matters to millions.
Shaping culture is in the DNA of ITV. So, it’s not surprising that you’ll find us in every home in the UK, our productions are famous all over the world and we’re at the forefront of the digital streaming revolution.
When you join us, you enter a fun working environment. With opportunities to learn, to grow and make a real difference. Small enough that your impact’s felt in the business, but big enough that your impact reaches millions of people.
Come develop your skills, change TV and the course of your career. Don’t just watch it. Be part of it. Join ITV.
Your impact sends ripples.
Our data team is responsible for helping us realise the value of data. That could mean anything from up-skilling and training our people, to increasing our revenue through experimentation and personalisation. The list is truly endless. The Innovation strand consists of Data Scientists, Operational Researchers and Behavioural Scientists. As a mix of number crunchers and behavioural experts, they put their curiosity to work in influencing the business. As we enter the fourth industrial revolution, anyone who joins the team will have the opportunity to work on genuinely groundbreaking innovations so that we can continue to blend science into a creative business.
As a Lead Data Engineer, both directly and via your team, you will be responsible for developing, maintaining, and optimising our data pipeline infrastructure using our proprietary central data platform (Symphony), which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.
This is a hands-on leadership role where you will be responsible for a small team of data engineers. You will work in partnership with your agile delivery manager and product manager to size up and breakdown work, and distribute that across your team, and be proactive in bringing that activity to closure.
A key aspect of this role is to inspire and motivate your team and encourage them to give their best effort and contribute enthusiastically to the shared vision.
Key Responsibilities of this role include:
· Deliver technical leadership monitoring productivity and output; setting of objectives and commitments; contributing to reviews and performance assessment of a small number of data engineers
· Ownership and delivery of the design, development, and maintenance of scalable data pipelines, infrastructure and data products using Databricks. Including implementation of efficient and reliable data ingestion, processing, and storage solutions at scale and prototyping new approaches.
· Assist and collaborate with internal and external stakeholders
· Develop and optimise extract, transform, load (ELT) processes ensuring smooth flow of data from various sources into the data platform. Perform data cleansing, validation, and enrichment to support accurate and reliable data analysis.
· Ensure adherence to data architecture principles, best practices and processes, collaborating with data scientists, analysts and stakeholders capturing and defining data requirements, designing appropriate data models and architectures whilst optimising data storage formats and partitioning for efficient data retrieval and analysis.
· Monitor and optimise data pipelines and workflows to ensure high performance and reliability. Identify and resolve performance bottlenecks, data quality issues, and data integration challenges.
· Define the data quality characteristics of your data products and work with the QA team to implement automated quality processes to measure data correctness and report outcomes through our observability infrastructure
· Implement data governance practices and ensure compliance with ITV data privacy and security regulations. Establish data access controls, encryption mechanisms, and data retention policies. Execute and comply with ITV architecture governance processes.
· Ensure all data products conform to all observability requirements and suitable dashboards are in place.
· Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to understand their requirements and provide data engineering support. Document data pipelines, workflows, data presentation implementations and technical specifications for future reference.
· Communicate complex solutions in a clear and understandable way to both technologists and business stakeholders. This will include writing and giving presentations to the wider team
· Stay up-to-date with industry trends and emerging technologies
Skills you’ll need(minimum criteria)
Extensive experience leading small teams of data engineers Experience designing and building Databricks data products Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume with strong knowledge of data engineering concepts, data integration, and ETL processes Ability to write production-grade code including automated testing and able to deploy code via CI/CD platforms (e.g. Github Actions, Jenkins) Experience in Proficiency in working with distributed computing frameworks, such as Apache Spark and data modelling, database systems, and SQL optimisation techniques Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery)
Other things we’re looking for (key criteria)
Knowledge of the UK broadcast industry & broadcast/OTT advertising market and digital marketing and advertising industry Familiar with: Data design techniques for classification and regression as well as knowledge of A/B testing, experimental design, and general statistical modelling Architecture disciplines such as Data Mesh Architecture, Data Architecture, BI Architecture and Enterprise Architecture Data governance, data privacy, and security principles. Tools across other data management capabilities – data engineering, data governance, data quality, metadata management and master data management Designing and building out data visualisation solutions based on technology such as: Tableau, AWS QuickSight, Looker, ThoughtSpot A strong people leader, coach and mentor with exceptional communication skills, delivery focused with a technical outlook and ability to influence stakeholders at all levels.
