Be at the heart of actionFly remote-controlled drones into enemy territory to gather vital information.

Apply Now

Senior Data Engineer/ Scientist

Glasgow
2 days ago
Create job alert

Senior Data Engineer - Azure & Databricks Lakehouse

Glasgow (3/4 days onsite) | Exclusive Role with a Leading UK Consumer Business

A rapidly scaling UK consumer brand is undertaking a major data modernisation programme-moving away from legacy systems, manual Excel reporting and fragmented data sources into a fully automated Azure Enterprise Landing Zone + Databricks Lakehouse.
They are building a modern data platform from the ground up using Lakeflow Declarative Pipelines, Unity Catalog, and Azure Data Factory, and this role sits right at the heart of that transformation.
This is a rare opportunity to join early, influence architecture, and help define engineering standards, pipelines, curated layers and best practices that will support Operations, Finance, Sales, Logistics and Customer Care.
If you want to build a best-in-class Lakehouse from scratch-this is the one.

? What You'll Be Doing

Lakehouse Engineering (Azure + Databricks)

Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark, and Spark SQL across a full Medallion Architecture (Bronze ? Silver ? Gold).

Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing), SQL sources, SharePoint and SFTP using ADF + metadata-driven frameworks.

Apply Lakeflow expectations for data quality, schema validation and operational reliability.

Curated Data Layers & Modelling

Build clean, conformed Silver/Gold models aligned to enterprise business domains (customers, subscriptions, deliveries, finance, credit, logistics, operations).

Deliver star schemas, harmonisation logic, SCDs and business marts to power high-performance Power BI datasets.

Apply governance, lineage and fine-grained permissions via Unity Catalog.

Orchestration & Observability

Design and optimise orchestration using Lakeflow Workflows and Azure Data Factory.

Implement monitoring, alerting, SLAs/SLIs, runbooks and cost-optimisation across the platform.

DevOps & Platform Engineering

Build CI/CD pipelines in Azure DevOps for notebooks, Lakeflow pipelines, SQL models and ADF artefacts.

Ensure secure, enterprise-grade platform operation across Dev ? Prod, using private endpoints, managed identities and Key Vault.

Contribute to platform standards, design patterns, code reviews and future roadmap.

Collaboration & Delivery

Work closely with BI/Analytics teams to deliver curated datasets powering dashboards across the organisation.

Influence architecture decisions and uplift engineering maturity within a growing data function.

? Tech Stack You'll Work With

Databricks: Lakeflow Declarative Pipelines, Workflows, Unity Catalog, SQL Warehouses

Azure: ADLS Gen2, Data Factory, Key Vault, vNets & Private Endpoints

Languages: PySpark, Spark SQL, Python, Git

DevOps: Azure DevOps Repos, Pipelines, CI/CD

Analytics: Power BI, Fabric

? What We're Looking For

Experience

5-8+ years of Data Engineering with 2-3+ years delivering production workloads on Azure + Databricks.

Strong PySpark/Spark SQL and distributed data processing expertise.

Proven Medallion/Lakehouse delivery experience using Delta Lake.

Solid dimensional modelling (Kimball) including surrogate keys, SCD types 1/2, and merge strategies.

Operational experience-SLAs, observability, idempotent pipelines, reprocessing, backfills.

Mindset

Strong grounding in secure Azure Landing Zone patterns.

Comfort with Git, CI/CD, automated deployments and modern engineering standards.

Clear communicator who can translate technical decisions into business outcomes.

Nice to Have

Databricks Certified Data Engineer Associate

Streaming ingestion experience (Auto Loader, structured streaming, watermarking)

Subscription/entitlement modelling experience

Advanced Unity Catalog security (RLS, ABAC, PII governance)

Terraform/Bicep for IaC

Fabric Semantic Model / Direct Lake optimisation

Related Jobs

View all jobs

Data Science Engineer- eDV Clearance

Senior Data Engineer - Azure

Senior Data Engineer (with Python, PySpark and AWS)

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer - Commodities - 140,000 + bonuses

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Data Science Recruitment Trends 2025 (UK): What Job Seekers Need To Know About Today’s Hiring Process

Summary: UK data science hiring has shifted from title‑led CV screens to capability‑driven assessments that emphasise rigorous problem framing, high‑quality analytics & modelling, experiment/causality, production awareness (MLOps), governance/ethics, and measurable product or commercial impact. This guide explains what’s changed, what to expect in interviews & how to prepare—especially for product/data scientists, applied ML scientists, decision scientists, econometricians, growth/marketing analysts, and ML‑adjacent data scientists supporting LLM/AI products. Who this is for: Product/decision/data scientists, applied ML scientists, econometrics & causal inference specialists, experimentation leads, analytics engineers crossing into DS, ML generalists with strong statistics, and data scientists collaborating with platform/MLOps teams in the UK.

Why Data Science Careers in the UK Are Becoming More Multidisciplinary

Data science once meant advanced statistics, machine learning models and coding in Python or R. In the UK today, it has become one of the most in-demand professions across sectors — from healthcare to finance, retail to government. But as the field matures, employers now expect more than technical modelling skills. Modern data science is multidisciplinary. It requires not just coding and algorithms, but also legal knowledge, ethical reasoning, psychological insight, linguistic clarity and human-centred design. Data scientists are expected to interpret, communicate and apply data responsibly, with awareness of law, human behaviour and accessibility. In this article, we’ll explore why data science careers in the UK are becoming more multidisciplinary, how these five disciplines intersect with data science, and what job-seekers & employers need to know to succeed in this transformed field.

Data Science Team Structures Explained: Who Does What in a Modern Data Science Department

Data science is one of the most in-demand, dynamic, and multidisciplinary areas in the UK tech and business landscape. Organisations from finance, retail, health, government, and beyond are using data to drive decisions, automate processes, personalise services, predict trends, detect fraud, and more. To do that well, companies don’t just need good data scientists; they need teams with clearly defined roles, responsibilities, workflows, collaboration, and governance. If you're aiming for a role in data science or recruiting for one, understanding the structure of a data science department—and who does what—can make all the difference. This article breaks down the key roles, how they interact across the lifecycle of a data science project, what skills and qualifications are typical in the UK, expected salary ranges, challenges, trends, and how to build or grow an effective team.