Senior Data Engineer

Location: London Job Type: Permanent Industry: Cloud & Infrastructure Job reference: LAS/DSAI/7542_1753860819 Posted: about 4 hours ago

Position: Senior Data Engineer

Salary:Negotiable, depending on Experience

Location: Hybrid - Bristol, Manchester, or London
Security Requirement: SC Clearance to start and be willing / able to obtain DV Clearance

We are seeking Data Engineers, with a keen interest / experience in AI /ML and strong proficiency in Python Scripting. To join our Consultancy client working on meaningful projects in the Defence and Security sector.

Required Experience:

End-to-End Data Development: Strong experience with data pipelines, ETL processes, and workflow orchestration, demonstrating best practices across tech stacks.Strong Python SkillsDiverse Data Handling: Familiarity with batch, streaming, real-time, and unstructured data sources.Architectural Design & Systems Thinking: Ability to design and build scalable, high-performance data solutions.Data Modelling & Warehouse Design: Proficiency in data modelling, warehouse design, and database optimization, with examples of logical and physical models.Distributed Data Systems: Experience in deploying, managing, and tuning distributed systems for optimal reliability and performance.Coding & Development Practices: Demonstrated coding expertise with modular, reusable, and efficient code in various languages.Development Lifecycle: Understanding of SDLC, CI/CD pipelines, and version control.Data Governance & Security: Knowledge of data security, governance, metadata management, and master data principles.

Key Responsibilities:

As a Senior Data Engineer, you'll bridge the gap between client needs and technical solutions, creating data pipelines that ingest, transform, and enrich large data volumes. You'll have client-facing responsibilities, delivering high-quality data solutions in multi-disciplinary teams across industries.

In this consulting role, your responsibilities will vary depending on client engagement focus and your skillset, but will often include:

Data Engineering & AI Integration: Apply data engineering tools, integration frameworks, and query engines to create high-quality, standardised data for AI applications and reporting.Data Pipeline Development: Design and implement robust data pipelines and stores in collaboration with other engineers and developers.Innovative Problem Solving: Bring fresh approaches to challenging data engineering problems.Architecture for Scale: Design scalable, complex data architectures that provide cross-team value.Data Modelling & Governance: Establish standards in logical and physical data modelling and data governance.Distributed Computing: Employ parallel processing, streaming, and batch workflows to manage large data volumes effectively.ETL & Workflow Automation: Build ETL processes and automated workflows for efficient data movement.System Optimization: Tune data systems for performance, scalability, and monitoring.Data Security: Apply best practices for information security, including encryption and data anonymity for sensitive data assets.Data Governance & Quality: Manage metadata, data lineage, and data quality standards.

If you're passionate about using data engineering and AI to solve complex problems in the Defence and Security sector, we'd love to hear from you!

People Source Consulting Ltd is acting as an Employment Agency in relation to this vacancy. People Source specialise in technology recruitment across niche markets including Information Technology, Digital TV, Digital Marketing, Project and Programme Management, SAP, Digital and Consumer Electronics, Air Traffic Management, Management Consultancy, Business Intelligence, Manufacturing, Telecoms, Public Sector, Healthcare, Finance and Oil & Gas.