National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Big Data Lead

HireWand
Greater London
1 week ago
Create job alert

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Related Jobs

View all jobs

Big Data Lead

3 Days Left: Big Data Lead...

Chief Data Officer - Reinsurance

Chief Data Officer - Lloyd's of London

Director/Snr Director, Data Science Consulting - Machine Learning/Artificial Intelligence (ML/AI)

Data Architect

National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Data Science Jobs UK 2025: 50 Companies Hiring Now

Bookmark this guide—refreshed every quarter—so you always know who’s really expanding their data‑science teams. Budgets for predictive analytics, GenAI pilots & real‑time decision engines keep climbing in 2025. The UK’s National AI Strategy, tax relief for R&D & a sharp rise in cloud adoption mean employers need applied scientists, ML engineers, experiment designers, causal‑inference specialists & analytics leaders—right now. Below you’ll find 50 organisations that have advertised UK‑based data‑science vacancies or announced head‑count growth during the past eight weeks. They’re grouped into five quick‑scan categories so you can jump straight to the kind of employer—& culture—that suits you. For every company you’ll see: Main UK hub Example live or recent vacancy Why it’s worth a look (tech stack, mission, culture) Search any employer on DataScience‑Jobs.co.uk to view current ads, or set up a free alert so fresh openings land straight in your inbox.

Return-to-Work Pathways: Relaunch Your Data Science Career with Returnships, Flexible & Hybrid Roles

Returning to work after an extended break can feel like stepping into a whole new world—especially in a dynamic field like data science. Whether you paused your career for parenting, caring responsibilities or another life chapter, the UK’s data science sector now offers a variety of return-to-work pathways. From structured returnships to flexible and hybrid roles, these programmes recognise the transferable skills and resilience you’ve gained and provide mentorship, upskilling and supportive networks to ease your transition back. In this guide, you’ll discover how to: Understand the current demand for data science talent in the UK Leverage your organisational, communication and analytical skills in data science roles Overcome common re-entry challenges with practical solutions Refresh your technical knowledge through targeted learning Access returnship and re-entry programmes tailored to data science Find roles that fit around family commitments—whether flexible, hybrid or full-time Balance your career relaunch with caring responsibilities Master applications, interviews and networking specific to data science Learn from inspiring returner success stories Get answers to common questions in our FAQ section Whether you aim to return as a data analyst, machine learning engineer, data visualisation specialist or data science manager, this article will map out the steps and resources you need to reignite your data science career.

LinkedIn Profile Checklist for Data Science Jobs: 10 Tweaks to Elevate Recruiter Engagement

Data science recruiters often sift through dozens of profiles to find candidates skilled in Python, machine learning, statistical modelling and data visualisation—sometimes before roles even open. A generic LinkedIn profile won’t suffice in this data-driven era. This step-by-step LinkedIn for data science jobs checklist outlines ten targeted tweaks to elevate recruiter engagement. Whether you’re an aspiring junior data scientist, a specialist in MLOps, or a seasoned analytics leader, these optimisations will sharpen your profile’s search relevance and demonstrate your analytical impact.