National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Big Data Lead

HireWand
Greater London
1 week ago
Create job alert

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Related Jobs

View all jobs

Data Engineer

Senior Data Scientist

Director/Snr Director, Data Science Consulting - Machine Learning/Artificial Intelligence (ML/AI)

Director/Snr Director, Data Science Consulting - Machine Learning/Artificial Intelligence (ML/AI)

Lead Data Scientist - Applied Intelligence

Lead Data Engineer

National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Data Science Jobs Skills Radar 2026: Emerging Tools, Languages & Platforms to Learn Now

The UK’s data science job market is evolving fast—from forecasting models and AI assistants to real-time decision systems. In 2026, data scientists aren’t just expected to build models—they’re responsible for shaping insights that fuel everything from patient care to predictive banking. Welcome to the Data Science Jobs Skills Radar 2026—your essential annual guide to the languages, tools, and platforms driving demand across the UK. Whether you’re entering the job market or reskilling mid-career, this roadmap helps you prioritise the skills that matter most right now.

How to Find Hidden Data Science Jobs in the UK Using Professional Bodies like the RSS, BCS & More

The data science job market in the UK is thriving—but also increasingly competitive. As organisations in finance, healthcare, retail, government, and tech accelerate digital transformation, the demand for data talent has soared. Yet many of the best data science jobs are never posted publicly. They’re shared behind closed doors—within professional networks, at invite-only events, or through member-only mailing lists and specialist interest groups. These “hidden” roles are often filled through referrals, collaborations, or direct outreach to trusted experts. In this guide, we’ll show you how to unlock these hidden opportunities by engaging with key UK professional bodies such as the Royal Statistical Society (RSS), BCS (The Chartered Institute for IT), and Turing Society, plus communities like PyData and AI UK. You’ll learn how to use directories, CPD events, and networks to move beyond job boards—and into roles where you’re approached, not just applying.

How to Get a Better Data Science Job After a Lay-Off or Redundancy

Redundancy can be tough to face, especially in a competitive field like data science. But it’s important to know: your experience, analytical thinking, and modelling skills are still in demand. Across sectors like healthcare, finance, e-commerce, government and AI startups, UK employers continue to seek data scientists who can deliver value through insight, prediction, and automation. This guide will walk you through how to bounce back from redundancy with purpose and clarity—whether you're a data analyst looking to step up, a mid-level data scientist, or a machine learning specialist seeking a better-aligned opportunity.