Data Engineer

ESSENTIAL ROLES & RESPONSIBILITIES

• Identify and understand customer data-centric use cases within regulated financial services environments
• Design and implement data ingestion, processing, and transformation pipelines on Azure
• Build and maintain data pipelines for cleaning, normalisation, enrichment, and preparation
• Apply appropriate data modelling techniques and architecture patterns, with a strong focus on medallion architecture
• Orchestrate, monitor, and optimise Azure Databricks jobs and Azure Data Factory pipelines across development, UAT, and production environments
• Configure platforms, clusters, and compute resources to optimise performance, cost, and reliability
• Use automated CI/CD pipelines to manage, deploy, and version data artefacts and pipelines
• Operationalise workflows developed by analysts and data scientists
• Support customers in adopting Azure data, analytics, and machine learning services
• Ensure secure storage, processing, and quality of customer data
• Ensure networking and security best practices are applied when designing and operating data solutions
• Design solutions for processing large volumes of data using batch and streaming approaches
• Collaborate with analytics teams on data visualisation best practices and reporting enablement
• Ensure all solutions are well-documented, including pipelines, schemas, transformations, and operational runbooks

GOVERNANCE & REPORTING

• Maintain accurate documentation of data pipelines, schemas, transformations, and deployment processes
• Support data governance initiatives including lineage, metadata management, and access control
• Contribute to service reporting, risk tracking, and continuous improvement actions
• Ensure data environments are audit-ready and aligned with governance standards

TECHNOLOGY STACK (AZURE)

Cloud Platform:

• Microsoft Azure

Data Engineering & Analytics:

• Azure Databricks (development, UAT, and production)
• Azure Data Factory
• Azure Synapse Analytics (where applicable)

Machine Learning & AI:

• Azure Machine Learning (limited non-production usage)
• Azure Document Intelligence

Databases:

• Microsoft SQL Server / Azure SQL Database (primary platforms)
• PostgreSQL (limited use)
• MySQL (limited use)

Data Processing:

• Batch and streaming data pipelines

Security & Governance:

• Role-based access control (RBAC)
• Data encryption and key management
• Audit logging and monitoring

DevOps:

• CI/CD pipelines for data artefacts and infrastructure

BEHAVIOURAL COMPETENCIES – ORGANISATIONAL & BEHAVIOURAL FIT

• Positive mindset and enthusiasm for learning new technologies
• Collaborative and supportive team player
• Strong sense of ownership and accountability
• Methodical, analytical approach to problem-solving
• Strong understanding of ethical data usage in regulated environments

CRITICAL COMPETENCIES – TECHNICAL FIT

Essential:

• Strong SQL skills
• Programming experience with Python and/or Scala
• Hands-on experience with Azure-based data platforms
• Experience designing, building, and maintaining data pipelines
• Strong understanding of data modelling (relational and analytical), including medallion architecture
• Experience orchestrating and optimising Databricks and Data Factory workloads
• Experience using CI/CD pipelines for data and analytics solutions
• Strong awareness of security, networking best practices, GDPR, and PII handling

Desirable:

• Experience with Azure Databricks in production environments
• Familiarity with Azure Machine Learning and AI services
• Exposure to data visualisation tools (e.g. Power BI)
• Experience with big data frameworks (Spark, Kafka)
• Knowledge of data governance, lineage, and metadata tooling

SHIFT & WORKING PATTERN

• Standard business hours, with participation in an on-call rota as required
• Occasional weekend engineering coverage will be required, typically limited to a small number of planned weekends per year to support business continuity, resilience testing, or disaster recovery activities