Senior Data Engineer

Mastek
Greater London
1 month ago
Create job alert

Job Title: Senior Data Engineer

Location: London, UK (3 days in the office)

SC Cleared: Required

Job Type: Full-Time

Experience: 8+ years

Job Summary:

We are seeking a highly skilled and experienced Senior Data Engineer to join our team and contribute to the development and maintenance of our cutting-edge Azure Databricks platform for economic data. This platform is critical for our Monetary Analysis, Forecasting, and Modelling activities. The Senior Data Engineer will be responsible for building and optimising data pipelines, implementing data transformations, and ensuring data quality and reliability. This role requires a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets.

Key Responsibilities:

  • Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data providers) into the Azure Databricks platform.
  • Optimise data pipelines for performance, efficiency, and cost-effectiveness.
  • Implement data quality checks and validation rules within data pipelines.

Data Transformation & Processing:

  • Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies.
  • Develop and maintain data processing logic for cleaning, enriching, and aggregating data.
  • Ensure data consistency and accuracy throughout the data lifecycle.
  • Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services.
  • Implement best practices for Databricks development and deployment.
  • Optimise Databricks workloads for performance and cost.

Data Integration:

  • Integrate data from various sources, including relational databases, APIs, and streaming data sources.
  • Implement data integration patterns and best practices.
  • Work with API developers to ensure seamless data exchange.

Data Quality & Governance:

  • Hands-on experience using Azure Purview for data quality and data governance.
  • Implement data quality monitoring and alerting processes.
  • Work with data governance teams to ensure compliance with data governance policies and standards.
  • Implement data lineage tracking and metadata management processes.
  • Collaborate closely with data scientists, economists, and other technical teams to understand data requirements and translate them into technical solutions.
  • Communicate technical concepts effectively to both technical and non-technical audiences.
  • Participate in code reviews and knowledge sharing sessions.

Automation & DevOps:

  • Implement automation for data pipeline deployments and other data engineering tasks.
  • Work with DevOps teams to implement and build CI/CD pipelines for environmental deployments.
  • Promote and implement DevOps best practices.

Minimum Qualifications:

  • 10+ years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks.
  • Strong proficiency in Python and Spark (PySpark) or Scala.
  • Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns.
  • Extensive experience with Azure data services, including Azure Data Factory, Azure Blob Storage, and Azure SQL Database.
  • Experience working with large datasets and complex data pipelines.
  • Experience with data architecture design and data pipeline optimization.
  • Proven expertise with Databricks, including hands-on implementation experience and certifications.
  • Experience with SQL and NoSQL databases.
  • Experience with data quality and data governance processes.
  • Experience with version control systems (e.g., Git).
  • Experience with Agile development methodologies.
  • Excellent communication, interpersonal, and problem-solving skills.
  • Experience with streaming data technologies (e.g., Kafka, Azure Event Hubs).
  • Experience with data visualisation tools (e.g., Tableau, Power BI).
  • Experience with DevOps tools and practices (e.g., Azure DevOps, Jenkins, Docker, Kubernetes).
  • Experience working in a financial services or economic data environment.
  • Azure certifications related to data engineering (e.g., Azure Data Engineer Associate).

Seniority Level:

Mid-Senior level

Employment Type:

Full-time

Job Function:

IT Services and IT Consulting

#J-18808-Ljbffr

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer - Fabric - £70,000 - London

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Tips for Staying Inspired: How Data Science Pros Fuel Creativity and Innovation

Data science sits at the dynamic intersection of statistics, computer science, and domain expertise, driving powerful innovations in industries ranging from healthcare to finance, and from retail to robotics. Yet, the daily reality for many data scientists can be a far cry from starry-eyed talk of AI and machine learning transformations. Instead, it often involves endless data wrangling, model tuning, and scrutiny over metrics. Maintaining a sense of creativity in this environment can be an uphill battle. So, how do successful data scientists continue to dream big and innovate, even when dealing with the nitty-gritty of data pipelines, debugging code, or explaining results to stakeholders? Below, we outline ten practical strategies to help data analysts, machine learning engineers, and research scientists stay inspired and push their ideas further. Whether you’re just starting out or looking to reinvigorate a long-standing career, these pointers can help you find fresh sparks of motivation.

Top 10 Data Science Career Myths Debunked: Key Facts for Aspiring Professionals

Data science has become one of the most sought-after fields in the tech world, promising attractive salaries, cutting-edge projects, and the opportunity to shape decision-making in virtually every industry. From e-commerce recommendation engines to AI-powered medical diagnostics, data scientists are the force behind innovations that drive productivity and improve people’s lives. Yet, despite the demand and glamour often associated with this discipline, data science is also shrouded in misconceptions. Some believe you need a PhD in mathematics or statistics; others assume data science is exclusively about machine learning or coding. At DataScience-Jobs.co.uk, we’ve encountered a wide array of myths that can discourage talented individuals or mislead those exploring a data science career. This article aims to bust the top 10 data science career myths—providing clarity on what data scientists actually do and illuminating the true diversity and inclusiveness of this exciting field. Whether you’re a recent graduate, a professional looking to pivot, or simply curious about data science, read on to discover the reality behind the myths.

Global vs. Local: Comparing the UK Data Science Job Market to International Landscapes

How to evaluate salaries, opportunities, and work culture in data science across the UK, the US, Europe, and Asia Data science has proven to be more than a passing trend; it is now a foundational pillar of modern decision-making in virtually every industry—from healthcare and finance to retail and entertainment. As the volume of data grows exponentially, organisations urgently need professionals who can transform raw information into actionable insights. This high demand has sparked a wave of new opportunities for data scientists worldwide. In this article, we’ll compare the UK data science job market to those in the United States, Europe, and Asia. We’ll explore hiring trends, salary benchmarks, and cultural nuances to help you decide whether to focus your career locally or consider opportunities overseas or in fully remote roles. Whether you’re a fresh graduate looking for your first data science position, an experienced data professional pivoting from analytics, or a software engineer eager to break into machine learning, understanding the global data science landscape can be a game-changer. By the end of this overview, you’ll be better equipped to navigate the expanding world of data science—knowing which skills and certifications matter most, how salaries differ between regions, and what to expect from distinct work cultures. Let’s dive in.