Azure Databricks Engineer

Huxley Associates

London, City And County Of the City Of London, United Kingdom

Last month

£140,000 – £160,000 pa

Salary

£140,000 – £160,000 pa

Posted: 7 Apr 2026 (Last month)

Save job

Create job alert

This is a rare opportunity to apply serious data engineering in a domain where latency, correctness, and reliability carry direct commercial weight.

* Requirements

* 6+ years data engineering in production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts

* ETL/ELT pipeline design and implementation at scale; orchestration with Airflow, Prefect, or equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics

* Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep)

* Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation

* Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads

* MongoDB - document modelling, aggregation pipelines, indexing strategy, replica sets; clear judgment on when document vs relational storage is the right architectural call

* Containerisation: Docker and Kubernetes-based deployment of data workloads; reproducible, environment-agnostic data infrastructure

* Data modelling for analytical workloads - dimensional modelling, data vault, or equivalent; schema evolution, slowly changing dimensions, and downstream impact analysis

* Stream and batch processing patterns; late data handling, watermarking, and backfill strategies; throughput vs latency trade-offs in pipeline design

* Production data observability - data lineage, quality checks, SLA monitoring, alerting on freshness and completeness; treating data correctness as a first-class concern

* CI/CD for data infrastructure - version-controlled pipelines, automated data quality testing, reproducible and auditable deploys

* Ability to work directly with quant researchers, risk managers, and traders - translate business requirements into reliable, well-documented data products

* Nice to Have

* Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management

* Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem

* dbt or equivalent SQL transformation layer; experience building and maintaining dbt projects in a production data warehouse

* Event streaming with Kafka or Confluent Platform - topic design, consumer group management, exactly-once delivery guarantees

* OLAP-optimised stores - ClickHouse, DuckDB, or equivalent; understanding of columnar storage and vectorised query execution

* Energy, commodities, or broader financial markets domain knowledge

* What We're Looking For

* You treat data as a product, not a side effect. You know what it takes to make a pipeline trustworthy - not just running, but observable, tested, and recoverable when something upstream changes at 3am. You think in systems: schema evolution, lineage, freshness SLAs, and the downstream impact of every modelling decision. At ETrading , that data is the foundation of billion-dollar trading decisions. You are the reason it is right.

To find out more about Huxley, please visit

Huxley, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy | Registered office | 8 Bishopsgate, London, EC2N 4BQ, United Kingdom | Partnership Number | OC(phone number removed) England and Wales

Related Jobs

View all jobs

Data Engineer

hireful Bristol, Bristol (county), United Kingdom

£50,000 – £55,000 pa Hybrid

Data Engineer

hireful Devon, United Kingdom

£50,000 – £55,000 pa

Lead Platform Engineer

Synapri London, United Kingdom

£75,000 – £85,000 pa

Data Platform Solutions Architect (Professional Services)

Databricks London, United Kingdom

Hybrid

Resident Solutions Architect (Professional Services)

Databricks London, United Kingdom

Data Platform Solutions Architect (Professional Services) - Emerging Enterprise & DNB

Databricks London, United Kingdom

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Apr 9, 2026

Products

Where to Advertise Data Science Jobs in the UK (2026 Guide)

Advertising data science jobs in the UK requires a different approach to most technical hiring. Data science spans a broad and often misunderstood spectrum — from statistical modelling and experimental design through to machine learning engineering, product analytics and AI research. The strongest candidates identify firmly with specific subdisciplines and are frustrated by adverts that conflate data scientist with data analyst, business intelligence developer or machine learning engineer. General job boards produce high application volumes for data roles but consistently fail to match specialist data science profiles with the right opportunities. This guide, published by DataScienceJobs.co.uk, covers where to advertise data science roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about hiring across different role types.

Apr 5, 2026

Jobs

Data Science Jobs UK 2026: What to Expect Over the Next 3 Years

Data science has spent the past decade being described as the sexiest job of the twenty-first century. By 2026, the reality is both more nuanced and more interesting than that label ever suggested. The discipline has matured, fragmented, deepened, and in some respects reinvented itself — and the jobs market has changed with it in ways that create genuine opportunity for those who understand what employers actually want, and genuine difficulty for those still operating on assumptions formed five years ago. The data science jobs market of 2026 is not simply a larger version of what it was three years ago. The generalist data scientist — equally comfortable wrangling data, building models, and presenting insights to the board — is giving way to a more specialised landscape where employers know exactly what problem they are trying to solve and are looking for candidates with the specific depth to solve it. Machine learning engineering, causal inference, experimentation, AI product development, and domain-specific applied science have all emerged as distinct career tracks within what was previously a single, loosely defined profession. At the same time, the arrival of large language models and the broader AI capability wave has both threatened and created data science roles in equal measure. Some of the work that junior data scientists spent their early careers doing — data cleaning, exploratory analysis, basic model building — is being partially automated by AI tooling. But the demand for practitioners who can evaluate AI systems rigorously, apply statistical thinking to complex business problems, and build the data foundations on which AI depends has grown considerably. The candidates who will thrive over the next three years are those who understand where the discipline is heading — which specialisms are attracting the most investment, which technologies are reshaping what data scientists are expected to build and know, and how to position a data science career that will remain valuable as the field continues to evolve around them. This article breaks down what the UK data science jobs market is likely to look like through to 2028 — covering the titles emerging right now, the technologies driving employer demand, the skills that will matter most, and how to position your career ahead of the curve.

Mar 24, 2026

Jobs

New Data Science Employers to Watch in 2026: UK and International Companies Leading Analytics and AI Innovation

Data science has emerged as one of the most transformative forces across industries, turning raw information into actionable insights, predictive models, and AI-powered solutions. In 2026, the UK is witnessing a surge in organisations where data science is not just a support function but the core of their products and services. For professionals exploring opportunities on www.DataScience-Jobs.co.uk , identifying these employers early can provide a competitive advantage in a market with high demand for advanced analytics and machine learning expertise. This article highlights new and high-growth data science employers to watch in 2026, focusing on UK startups, scale-ups, and global firms expanding their data science operations locally. All of the companies included have recently raised investment, won high-profile contracts, or significantly scaled their analytics teams.