Data Engineer - Databricks Specialist

The AA
Basingstoke
3 days ago
Create job alert
Overview

Location: Basingstoke (hybrid working 3 office days per week)

Join Our Data & Analytics Team: Transforming Data into Our Superpower! Are you passionate about data and eager to make a significant impact? The AA is a well-loved brand with a range of driver services much wider than most people realise. We have an enviable set of data assets from breakdown, service, repair, insurance, telematics, digital interactions, car dealers and driving school!

Job Details

Employment Type: Permanent, full time

Additional Benefits: Annual Bonus

Role context

This is the job: At The AA, our purpose is to create confidence for drivers now and for the future. Data plays a critical role in delivering that purpose, and we are investing heavily in a modern, Databricks-centric data platform to unlock the full value of our connected car and insurance data.

This role is for an experienced Databricks Data Engineer. You will be working day-to-day designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines, Unity Catalog governance, and Spark-based data engineering at scale.

This is not a role for someone who has “touched” Databricks occasionally or supported it at the edges. We are looking for someone who has built, owned and evolved Databricks solutions, and who can confidently articulate design decisions, trade-offs, and best practice.

If Databricks is central to how you work and can demonstrate this natural, we want to hear from you.

What will I be doing?
  • Designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark
  • Owning and evolving Unity Catalog–based governance, ensuring secure, discoverable and well-managed data assets
  • Developing and maintaining event-driven data pipelines, integrating closely with backend engineering teams
  • Implementing and supporting CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments
  • Creating high-quality, analytics-ready datasets that deliver actionable business insight at scale
  • Proactively improving performance, reliability, automation and observability across the Databricks data platform
What do I need?

We are intentionally setting a high bar for Databricks experience. You should be able to demonstrate deep, hands-on capability, not just theoretical knowledge.

Essential Experience
  • Significant, hands-on experience with Azure Databricks in a production environment
  • Proven experience building Spark / Databricks pipelines using Python and PySpark
  • Strong experience with structured streaming and event-driven architectures
  • Practical experience implementing and operating Unity Catalog
  • Solid understanding of Lakehouse design principles, including dimensional and analytical modelling
  • Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps
  • Confidence working with large-scale data, performance tuning, and troubleshooting complex pipelines
You Have

What we mean by “Databricks experience”

  • Designed and built Databricks pipelines end-to-end
  • Made architectural decisions within Databricks environments
  • Worked with Spark internals, optimisation techniques and cluster configuration
  • Operated Databricks solutions in live, business-critical contexts

If your Databricks exposure has been limited to minor contributions, proof-of-concepts, or occasional usage alongside other tools, this role is unlikely to be the right fit.

Additional Information

We’re always looking to recognise and reward our employees for the work they do. As a valued member of The AA team, you’ll have access to a range of benefits including:

  • 25 days annual leave plus bank holidays + holiday buying scheme
  • Worksave pension scheme with up to 7% employer contribution
  • Free AA breakdown membership from Day 1 plus 50% discount for family and friends
  • Discounts on AA products including car and home insurance
  • Employee discount scheme that gives you access to a car salary sacrifice scheme plus great discounts on healthcare, shopping, holidays and more
  • Company funded life assurance
  • Diverse learning and development opportunities to support you to progress in your career
  • Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family

Plus, so much more!

We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.


#J-18808-Ljbffr

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

What Hiring Managers Look for First in Data Science Job Applications (UK Guide)

If you’re applying for data science roles in the UK, it’s crucial to understand what hiring managers focus on before they dive into your full CV. In competitive markets, recruiters and hiring managers often make their first decisions in the first 10–20 seconds of scanning an application — and in data science, there are specific signals they look for first. Data science isn’t just about coding or statistics — it’s about producing insights, shipping models, collaborating with teams, and solving real business problems. This guide helps you understand exactly what hiring managers look for first in data science applications — and how to structure your CV, portfolio and cover letter so you leap to the top of the shortlist.

The Skills Gap in Data Science Jobs: What Universities Aren’t Teaching

Data science has become one of the most visible and sought-after careers in the UK technology market. From financial services and retail to healthcare, media, government and sport, organisations increasingly rely on data scientists to extract insight, guide decisions and build predictive models. Universities have responded quickly. Degrees in data science, analytics and artificial intelligence have expanded rapidly, and many computer science courses now include data-focused pathways. And yet, despite the volume of graduates entering the market, employers across the UK consistently report the same problem: Many data science candidates are not job-ready. Vacancies remain open. Hiring processes drag on. Candidates with impressive academic backgrounds fail interviews or struggle once hired. The issue is not intelligence or effort. It is a persistent skills gap between university education and real-world data science roles. This article explores that gap in depth: what universities teach well, what they often miss, why the gap exists, what employers actually want, and how jobseekers can bridge the divide to build successful careers in data science.

Data Science Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Thinking about switching into data science in your 30s, 40s or 50s? You’re far from alone. Across the UK, businesses are investing in data science talent to turn data into insight, support better decisions and unlock competitive advantage. But with all the hype about machine learning, Python, AI and data unicorns, it can be hard to separate real opportunities from noise. This article gives you a practical, UK-focused reality check on data science careers for mid-life career switchers — what roles really exist, what skills employers really hire for, how long retraining typically takes, what UK recruiters actually look for and how to craft a compelling career pivot story. Whether you come from finance, marketing, operations, research, project management or another field entirely, there are meaningful pathways into data science — and age itself is not the barrier many people fear.