Data Engineer - Databricks Specialist

The AA
Basingstoke
3 days ago
Create job alert
Location

Basingstoke (hybrid working 3 office days per week)


Employment Type

Permanent, full time


Job Overview

Are you passionate about data and eager to make a significant impact? The AA is a well‐loved brand with a range of driver services that extends far beyond most people realise. We have an enviable set of data assets from breakdown, service, repair, insurance, telematics, digital interactions, car dealers and driving schools. Our growing team is modernising our data infrastructure to a cutting‐edge cloud platform and enabling machine learning and GenAI. Join us at an exciting time and be part of a team that is driving meaningful change for our customers, colleagues and shareholders.


What I will be doing

  • Design, build and operate production‑grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark.
  • Own and evolve Unity Catalog‑based governance to ensure secure, discoverable and well‑managed data assets.
  • Develop and maintain event‑driven data pipelines, integrating closely with backend engineering teams.
  • Implement and support CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments.
  • Create high‑quality, analytics‑ready datasets that deliver actionable business insight at scale.
  • Proactively improve performance, reliability, automation and observability across the Databricks data platform.

What do I need

We are intentionally setting a high bar for Databricks experience. You should be able to demonstrate deep, hands‑on capability, not just theoretical knowledge.


Essential Experience

  • Significant, hands‑on experience with Azure Databricks in a production environment.
  • Proven experience building Spark/Databricks pipelines using Python and PySpark.
  • Strong experience with structured streaming and event‑driven architectures.
  • Practical experience implementing and operating Unity Catalog.
  • Solid understanding of Lakehouse design principles, including dimensional and analytical modelling.
  • Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps.
  • Confidence working with large‑scale data, performance tuning and troubleshooting complex pipelines.

You Have

We understand that “Databricks experience” can mean different things. For this role we mean:



  • Designed and built Databricks pipelines end‑to‑end.
  • Made architectural decisions within Databricks environments.
  • Worked with Spark internals, optimisation techniques and cluster configuration.
  • Operated Databricks solutions in live, business‑critical contexts.

Benefits

  • 25 days annual leave plus bank holidays + holiday buying scheme.
  • Worksave pension scheme with up to 7% employer contribution.
  • Free AA breakdown membership from Day 1 plus 50% discount for family and friends.
  • Discounts on AA products including car and home insurance.
  • Employee discount scheme that gives access to a car salary sacrifice scheme plus great discounts on healthcare, shopping, holidays and more.
  • Company‑funded life assurance.
  • Diverse learning and development opportunities to support career progression.
  • Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family.

Equal Opportunity

We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.


Seniority Level

Not Applicable


Job Function

Information Technology


Industries

Motor Vehicle Manufacturing, Financial Services, and Insurance


#J-18808-Ljbffr

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

How to Write a Data Science Job Ad That Attracts the Right People

Data science plays a critical role in how organisations across the UK make decisions, build products and gain competitive advantage. From forecasting and personalisation to risk modelling and experimentation, data scientists help translate data into insight and action. Yet many employers struggle to attract the right data science candidates. Job adverts often generate high volumes of applications, but few applicants have the mix of analytical skill, business understanding and communication ability the role actually requires. At the same time, experienced data scientists skip over adverts that feel vague, inflated or misaligned with real data science work. In most cases, the issue is not a lack of talent — it is the quality and clarity of the job advert. Data scientists are analytical, sceptical of hype and highly selective. A poorly written job ad signals unclear expectations and immature data practices. A well-written one signals credibility, focus and serious intent. This guide explains how to write a data science job ad that attracts the right people, improves applicant quality and positions your organisation as a strong data employer.

Maths for Data Science Jobs: The Only Topics You Actually Need (& How to Learn Them)

If you are applying for data science jobs in the UK, the maths can feel like a moving target. Job descriptions say “strong statistical knowledge” or “solid ML fundamentals” but they rarely tell you which topics you will actually use day to day. Here’s the truth: most UK data science roles do not require advanced pure maths. What they do require is confidence with a tight set of practical topics that come up repeatedly in modelling, experimentation, forecasting, evaluation, stakeholder comms & decision-making. This guide focuses on the only maths most data scientists keep using: Statistics for decision making (confidence intervals, hypothesis tests, power, uncertainty) Probability for real-world data (base rates, noise, sampling, Bayesian intuition) Linear algebra essentials (vectors, matrices, projections, PCA intuition) Calculus & gradients (enough to understand optimisation & backprop) Optimisation & model evaluation (loss functions, cross-validation, metrics, thresholds) You’ll also get a 6-week plan, portfolio projects & a resources section you can follow without getting pulled into unnecessary theory.

Neurodiversity in Data Science Careers: Turning Different Thinking into a Superpower

Data science is all about turning messy, real-world information into decisions, products & insights. It sits at the crossroads of maths, coding, business & communication – which means it needs people who see patterns, ask unusual questions & challenge assumptions. That makes data science a natural fit for many neurodivergent people, including those with ADHD, autism & dyslexia. If you’re neurodivergent & thinking about a data science career, you might have heard comments like “you’re too distracted for complex analysis”, “too literal for stakeholder work” or “too disorganised for large projects”. In reality, the same traits that can make traditional environments difficult often line up beautifully with data science work. This guide is written for data science job seekers in the UK. We’ll explore: What neurodiversity means in a data science context How ADHD, autism & dyslexia strengths map to common data science roles Practical workplace adjustments you can request under UK law How to talk about your neurodivergence in applications & interviews By the end, you’ll have a clearer sense of where you might thrive in data science – & how to turn “different thinking” into a real career advantage.