Senior Data Engineer/ PowerBI

Head Resourcing
Glasgow
3 weeks ago
Create job alert

Lead Data Engineer - Azure & Databricks Lakehouse

Glasgow (3/4 days onsite) | Exclusive Role with a Leading UK Consumer Business

A rapidly scaling UK consumer brand is undertaking a major data modernisation programme-moving away from legacy systems, manual Excel reporting and fragmented data sources into a fully automated Azure Enterprise Landing Zone + Databricks Lakehouse.
They are building a modern data platform from the ground up using Lakeflow Declarative Pipelines, Unity Catalog, and Azure Data Factory, and this role sits right at the heart of that transformation.
This is a rare opportunity to join early, influence architecture, and help define engineering standards, pipelines, curated layers and best practices that will support Operations, Finance, Sales, Logistics and Customer Care.
If you want to build a best-in-class Lakehouse from scratch-this is the one.

? What You'll Be Doing

Lakehouse Engineering (Azure + Databricks)

Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark, and Spark SQL across a full Medallion Architecture (Bronze ? Silver ? Gold).

Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing), SQL sources, SharePoint and SFTP using ADF + metadata-driven frameworks.

Apply Lakeflow expectations for data quality, schema validation and operational reliability.

Curated Data Layers & Modelling

Build clean, conformed Silver/Gold models aligned to enterprise business domains (customers, subscriptions, deliveries, finance, credit, logistics, operations).

Deliver star schemas, harmonisation logic, SCDs and business marts to power high-performance Power BI datasets.

Apply governance, lineage and fine-grained permissions via Unity Catalog.

Orchestration & Observability

Design and optimise orchestration using Lakeflow Workflows and Azure Data Factory.

Implement monitoring, alerting, SLAs/SLIs, runbooks and cost-optimisation across the platform.

DevOps & Platform Engineering

Build CI/CD pipelines in Azure DevOps for notebooks, Lakeflow pipelines, SQL models and ADF artefacts.

Ensure secure, enterprise-grade platform operation across Dev ? Prod, using private endpoints, managed identities and Key Vault.

Contribute to platform standards, design patterns, code reviews and future roadmap.

Collaboration & Delivery

Work closely with BI/Analytics teams to deliver curated datasets powering dashboards across the organisation.

Influence architecture decisions and uplift engineering maturity within a growing data function.

? Tech Stack You'll Work With

Databricks: Lakeflow Declarative Pipelines, Workflows, Unity Catalog, SQL Warehouses

Azure: ADLS Gen2, Data Factory, Key Vault, vNets & Private Endpoints

Languages: PySpark, Spark SQL, Python, Git

DevOps: Azure DevOps Repos, Pipelines, CI/CD

Analytics: Power BI, Fabric

? What We're Looking For

Experience

5-8+ years of Data Engineering with 2-3+ years delivering production workloads on Azure + Databricks.

Strong PySpark/Spark SQL and distributed data processing expertise.

Proven Medallion/Lakehouse delivery experience using Delta Lake.

Solid dimensional modelling (Kimball) including surrogate keys, SCD types 1/2, and merge strategies.

Operational experience-SLAs, observability, idempotent pipelines, reprocessing, backfills.

Mindset

Strong grounding in secure Azure Landing Zone patterns.

Comfort with Git, CI/CD, automated deployments and modern engineering standards.

Clear communicator who can translate technical decisions into business outcomes.

Nice to Have

Databricks Certified Data Engineer Associate

Streaming ingestion experience (Auto Loader, structured streaming, watermarking)

Subscription/entitlement modelling experience

Advanced Unity Catalog security (RLS, ABAC, PII governance)

Terraform/Bicep for IaC

Fabric Semantic Model / Direct Lake optimisation

Related Jobs

View all jobs

Senior Data Engineer/ PowerBI

Senior Data Engineer - Build Scalable Pipelines & BI (Remote)

Senior Data Engineer - AI Pipelines (Monthly Europe Travel)

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

What Hiring Managers Look for First in Data Science Job Applications (UK Guide)

If you’re applying for data science roles in the UK, it’s crucial to understand what hiring managers focus on before they dive into your full CV. In competitive markets, recruiters and hiring managers often make their first decisions in the first 10–20 seconds of scanning an application — and in data science, there are specific signals they look for first. Data science isn’t just about coding or statistics — it’s about producing insights, shipping models, collaborating with teams, and solving real business problems. This guide helps you understand exactly what hiring managers look for first in data science applications — and how to structure your CV, portfolio and cover letter so you leap to the top of the shortlist.

The Skills Gap in Data Science Jobs: What Universities Aren’t Teaching

Data science has become one of the most visible and sought-after careers in the UK technology market. From financial services and retail to healthcare, media, government and sport, organisations increasingly rely on data scientists to extract insight, guide decisions and build predictive models. Universities have responded quickly. Degrees in data science, analytics and artificial intelligence have expanded rapidly, and many computer science courses now include data-focused pathways. And yet, despite the volume of graduates entering the market, employers across the UK consistently report the same problem: Many data science candidates are not job-ready. Vacancies remain open. Hiring processes drag on. Candidates with impressive academic backgrounds fail interviews or struggle once hired. The issue is not intelligence or effort. It is a persistent skills gap between university education and real-world data science roles. This article explores that gap in depth: what universities teach well, what they often miss, why the gap exists, what employers actually want, and how jobseekers can bridge the divide to build successful careers in data science.

Data Science Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Thinking about switching into data science in your 30s, 40s or 50s? You’re far from alone. Across the UK, businesses are investing in data science talent to turn data into insight, support better decisions and unlock competitive advantage. But with all the hype about machine learning, Python, AI and data unicorns, it can be hard to separate real opportunities from noise. This article gives you a practical, UK-focused reality check on data science careers for mid-life career switchers — what roles really exist, what skills employers really hire for, how long retraining typically takes, what UK recruiters actually look for and how to craft a compelling career pivot story. Whether you come from finance, marketing, operations, research, project management or another field entirely, there are meaningful pathways into data science — and age itself is not the barrier many people fear.