Data Engineer (Kafka)

Hays Technology
Sheffield
3 days ago
Create job alert

Sheffield (3 days on-site per week)

Candidates must be eligible for BPSS clearance

We're looking for a highly skilled Senior Data & Observability Engineer to design, build, and evolve large‑scale telemetry and observability capabilities across modern cloud platforms. This role is ideal for someone who thrives in high‑volume, real‑time data environments and enjoys solving complex engineering challenges.

What You'll Do

Design, implement, and maintain scalable data pipelines to ingest and process telemetry (metrics, logs, traces) from OpenShift and Kubernetes platforms.
Stream telemetry via Kafka-managing producers, topics, schemas, and building resilient consumer services for transformation and enrichment.
Engineer multi‑tenant observability data models and routing strategies, ensuring data lineage, quality, and SLAs.
Integrate processed telemetry into Splunk, enabling dashboards, alerting, analytics, and proactive insights (Level 4 Observability maturity).
Implement schema governance, versioning, and compatibility controls using Avro/Protobuf.
Build automated validation, replay, and backfill tools to maintain data reliability and support recovery.
Instrument services with OpenTelemetry and drive consistency in tracing, metrics, and structured logging.
Use LLMs to enhance observability workflows-such as anomaly summarisation, query assistance, and automated runbook creation.
Work closely with platform, SRE, and application teams to integrate telemetry, alerts, and SLOs.
Ensure security, compliance, RBAC, and best practices across pipelines and tooling.
Produce clear documentation covering data flows, schemas, dashboards, and operational processes.What You'll Bring

Strong hands-on experience with Kafka (producers/consumers, schema registry, Kafka Connect, KSQL/KStreams).
Proficiency with OpenShift / Kubernetes telemetry, including OpenTelemetry and Prometheus.
Experience integrating telemetry into Splunk (HEC, UF, sourcetypes, CIM) and building dashboards and alerts.
Solid data engineering skills in Python (or similar) for ETL/ELT, enrichment, and validation.
Understanding of schema formats (Avro, Protobuf, JSON) and compatibility strategies.
Knowledge of observability standards and ability to drive toward proactive, automated monitoring capabilities.
Understanding of telemetry in hybrid cloud and multi-cluster environments.
Strong security awareness-secrets management, RBAC, encryption in transit/at rest.
Great communication skills and the ability to work effectively with cross-functional teams.
Security Requirement
Applicants must be eligible for BPSS clearance (Baseline Personnel Security Standard).

What you need to do now

If you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or call us now.
If this job isn't quite right for you, but you are looking for a new position, please contact us for a confidential discussion about your career.

Hays Specialist Recruitment Limited acts as an employment agency for permanent recruitment and employment business for the supply of temporary workers. By applying for this job you accept the T&C's, Privacy Policy and Disclaimers which can be found at (url removed)

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

What Hiring Managers Look for First in Data Science Job Applications (UK Guide)

If you’re applying for data science roles in the UK, it’s crucial to understand what hiring managers focus on before they dive into your full CV. In competitive markets, recruiters and hiring managers often make their first decisions in the first 10–20 seconds of scanning an application — and in data science, there are specific signals they look for first. Data science isn’t just about coding or statistics — it’s about producing insights, shipping models, collaborating with teams, and solving real business problems. This guide helps you understand exactly what hiring managers look for first in data science applications — and how to structure your CV, portfolio and cover letter so you leap to the top of the shortlist.

The Skills Gap in Data Science Jobs: What Universities Aren’t Teaching

Data science has become one of the most visible and sought-after careers in the UK technology market. From financial services and retail to healthcare, media, government and sport, organisations increasingly rely on data scientists to extract insight, guide decisions and build predictive models. Universities have responded quickly. Degrees in data science, analytics and artificial intelligence have expanded rapidly, and many computer science courses now include data-focused pathways. And yet, despite the volume of graduates entering the market, employers across the UK consistently report the same problem: Many data science candidates are not job-ready. Vacancies remain open. Hiring processes drag on. Candidates with impressive academic backgrounds fail interviews or struggle once hired. The issue is not intelligence or effort. It is a persistent skills gap between university education and real-world data science roles. This article explores that gap in depth: what universities teach well, what they often miss, why the gap exists, what employers actually want, and how jobseekers can bridge the divide to build successful careers in data science.

Data Science Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Thinking about switching into data science in your 30s, 40s or 50s? You’re far from alone. Across the UK, businesses are investing in data science talent to turn data into insight, support better decisions and unlock competitive advantage. But with all the hype about machine learning, Python, AI and data unicorns, it can be hard to separate real opportunities from noise. This article gives you a practical, UK-focused reality check on data science careers for mid-life career switchers — what roles really exist, what skills employers really hire for, how long retraining typically takes, what UK recruiters actually look for and how to craft a compelling career pivot story. Whether you come from finance, marketing, operations, research, project management or another field entirely, there are meaningful pathways into data science — and age itself is not the barrier many people fear.