Azure Databricks Engineer

Huxley Associates
London, City And County Of the City Of London, United Kingdom
2 weeks ago
£140,000 – £160,000 pa

Salary

£140,000 – £160,000 pa

Posted
7 Apr 2026 (2 weeks ago)

This is a rare opportunity to apply serious data engineering in a domain where latency, correctness, and reliability carry direct commercial weight.

* Requirements

* 6+ years data engineering in production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts

* ETL/ELT pipeline design and implementation at scale; orchestration with Airflow, Prefect, or equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics

* Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep)

* Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation

* Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads

* MongoDB - document modelling, aggregation pipelines, indexing strategy, replica sets; clear judgment on when document vs relational storage is the right architectural call

* Containerisation: Docker and Kubernetes-based deployment of data workloads; reproducible, environment-agnostic data infrastructure

* Data modelling for analytical workloads - dimensional modelling, data vault, or equivalent; schema evolution, slowly changing dimensions, and downstream impact analysis

* Stream and batch processing patterns; late data handling, watermarking, and backfill strategies; throughput vs latency trade-offs in pipeline design

* Production data observability - data lineage, quality checks, SLA monitoring, alerting on freshness and completeness; treating data correctness as a first-class concern

* CI/CD for data infrastructure - version-controlled pipelines, automated data quality testing, reproducible and auditable deploys

* Ability to work directly with quant researchers, risk managers, and traders - translate business requirements into reliable, well-documented data products

* Nice to Have

* Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management

* Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem

* dbt or equivalent SQL transformation layer; experience building and maintaining dbt projects in a production data warehouse

* Event streaming with Kafka or Confluent Platform - topic design, consumer group management, exactly-once delivery guarantees

* OLAP-optimised stores - ClickHouse, DuckDB, or equivalent; understanding of columnar storage and vectorised query execution

* Energy, commodities, or broader financial markets domain knowledge

* What We're Looking For

* You treat data as a product, not a side effect. You know what it takes to make a pipeline trustworthy - not just running, but observable, tested, and recoverable when something upstream changes at 3am. You think in systems: schema evolution, lineage, freshness SLAs, and the downstream impact of every modelling decision. At ETrading , that data is the foundation of billion-dollar trading decisions. You are the reason it is right.

To find out more about Huxley, please visit

Huxley, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy | Registered office | 8 Bishopsgate, London, EC2N 4BQ, United Kingdom | Partnership Number | OC(phone number removed) England and Wales

Related Jobs

View all jobs

Lead Platform Engineer

Synapri London, United Kingdom
£75,000 – £85,000 pa

Data Architect - Bristol Opportunity

Hays Technology Bristol, United Kingdom
Hybrid

Data Platform Solutions Architect (Professional Services)

Databricks London, United Kingdom
£40,000 – £80,000 pa Hybrid

Resident Solutions Architect (Professional Services)

Databricks London, United Kingdom

Data Platform Solutions Architect (Professional Services) - Emerging Enterprise & DNB

Databricks London, United Kingdom

Staff Software Engineer - Backend

Databricks London, United Kingdom

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Where to Advertise Data Science Jobs in the UK (2026 Guide)

Advertising data science jobs in the UK requires a different approach to most technical hiring. Data science spans a broad and often misunderstood spectrum — from statistical modelling and experimental design through to machine learning engineering, product analytics and AI research. The strongest candidates identify firmly with specific subdisciplines and are frustrated by adverts that conflate data scientist with data analyst, business intelligence developer or machine learning engineer. General job boards produce high application volumes for data roles but consistently fail to match specialist data science profiles with the right opportunities. This guide, published by DataScienceJobs.co.uk, covers where to advertise data science roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about hiring across different role types.

New Data Science Employers to Watch in 2026: UK and International Companies Leading Analytics and AI Innovation

Data science has emerged as one of the most transformative forces across industries, turning raw information into actionable insights, predictive models, and AI-powered solutions. In 2026, the UK is witnessing a surge in organisations where data science is not just a support function but the core of their products and services. For professionals exploring opportunities on www.DataScience-Jobs.co.uk , identifying these employers early can provide a competitive advantage in a market with high demand for advanced analytics and machine learning expertise. This article highlights new and high-growth data science employers to watch in 2026, focusing on UK startups, scale-ups, and global firms expanding their data science operations locally. All of the companies included have recently raised investment, won high-profile contracts, or significantly scaled their analytics teams.

How Many Data Science Tools Do You Need to Know to Get a Data Science Job?

If you’re trying to break into data science — or progress your career — it can feel like you are drowning in names: Python, R, TensorFlow, PyTorch, SQL, Spark, AWS, Scikit-learn, Jupyter, Tableau, Power BI…the list just keeps going. With every job advert listing a different combination of tools, many applicants fall into a trap: they try to learn everything. The result? Long tool lists that sound impressive — but little depth to back them up. Here’s the straight-talk version most hiring managers won’t explicitly tell you: 👉 You don’t need to know every data science tool to get hired. 👉 You need to know the right ones — deeply — and know how to use them to solve real problems. Tools matter, but only in service of outcomes. So how many data science tools do you actually need to know to get a job? For most job seekers, the answer is not “27” — it’s more like 8–12, thoughtfully chosen and well understood. This guide explains what employers really value, which tools are core, which are role-specific, and how to focus your toolbox so your CV and interviews shine.