Engineer the Quantum RevolutionYour expertise can help us shape the future of quantum computing at Oxford Ionics.

View Open Roles

Big Data Lead

HireWand
Greater London
1 month ago
Applications closed

Related Jobs

View all jobs

Senior Data Science Director

Senior Data Science Director

Data Engineer

Senior Commercial Data Analyst - EMEA

Lead Data Engineer (Databricks) - Leeds

Lead Data Engineer

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks: Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud: Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills: DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages: Proficiency in Python, SQL.
  • Data Warehousing & ETL: Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD: Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy: Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development: Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices: Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation: Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management: Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration: Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management: Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring: Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs

Seasonal Hiring Peaks for Data Science Jobs: The Best Months to Apply & Why

The UK's data science sector has matured into one of Europe's most intellectually rewarding and financially attractive technology markets, with roles spanning from junior data analysts to principal data scientists and heads of artificial intelligence. With data science positions commanding salaries from £30,000 for graduate data analysts to £140,000+ for senior principal scientists, understanding when organisations actively recruit can dramatically accelerate your career progression in this intellectually stimulating and rapidly evolving field. Unlike traditional analytical roles, data science hiring follows distinct patterns influenced by business intelligence cycles, research funding schedules, and machine learning project timelines. The sector's unique combination of mathematical rigour, business impact requirements, and cutting-edge technology adoption creates predictable hiring windows that strategic professionals can leverage to advance their careers in extracting insights from tomorrow's data. This comprehensive guide explores the optimal timing for data science job applications in the UK, examining how enterprise analytics strategies, academic research cycles, and artificial intelligence initiatives influence recruitment patterns, and why strategic timing can determine whether you join a pioneering AI research team or miss the opportunity to develop the next generation of intelligent systems.

Jobs

Pre-Employment Checks for Data Science Jobs: DBS, References & Right-to-Work and more Explained

Pre-employment screening in data science reflects the discipline's unique position at the intersection of statistical analysis, machine learning innovation, and strategic business intelligence. Data scientists often have privileged access to comprehensive datasets, proprietary algorithms, and business-critical insights that form the foundation of organisational strategy and competitive positioning. The data science industry operates within complex regulatory frameworks spanning GDPR, sector-specific data protection requirements, and emerging AI governance regulations. Data scientists must demonstrate not only technical competence in statistical modelling and machine learning but also deep understanding of research ethics, data privacy principles, and the societal implications of algorithmic decision-making. Modern data science roles frequently involve analysing personally identifiable information, financial data, healthcare records, and sensitive business intelligence across multiple jurisdictions and regulatory frameworks simultaneously. The combination of analytical privilege, predictive capabilities, and strategic influence makes thorough candidate verification essential for maintaining compliance, security, and public trust in data-driven insights and automated systems.

Careers

Why Now Is the Perfect Time to Launch Your Career in Data Science: The UK's Analytics Revolution

The United Kingdom stands at the forefront of a data science revolution that's reshaping how businesses make decisions, governments craft policies, and society tackles its greatest challenges. From the machine learning algorithms powering London's fintech innovation to the predictive models guiding Manchester's smart city initiatives, Britain's transformation into a data-driven economy has created an unprecedented demand for skilled data scientists that far outstrips the available talent. If you've been contemplating a career transition or seeking to position yourself at the cutting edge of the digital economy, data science represents one of the most intellectually stimulating, financially rewarding, and socially impactful career paths available today. The convergence of big data maturation, artificial intelligence mainstream adoption, business intelligence evolution, and cross-industry digital transformation has created the perfect conditions for data science career success.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.