Python Data Engineer-Azure

Vallum Associates
Sheffield
1 week ago
Create job alert

Job Title: Python Data Engineer- Azure

Location: Remote (Occasional visits to London)

Duration: 9Months+ Contract Inside IR35

  • Role Summary:
  • We are seeking a versatile and experiencedData Engineerwith a strong foundation inPython,PySpark, and modern data platforms. This role demands hands-on experience withCI/CD automation,unit testing, and working withinAzure environments— both through theAzure Portalandautomated scripts. Exposure todata pipelines,big data file formats, andAzure-native servicesis crucial.
  • Key Responsibilities:
  • Develop and optimize data processing workflows using Python and PySpark.
  • Manage and transform data using SparkSQL, handling data stored in Delta, Parquet, and other file formats.
  • Write and maintain Pytest-based unit tests to ensure pipeline robustness and data quality.
  • Build and maintain CI/CD pipelines using Azure DevOps (ADO) or GitLab for automated deployments.
  • Work within VS Code + Dev Containers for environment management and efficient development cycles.
  • Manage Python dependencies using Poetry.
  • Use OpenTelemetry to enable observability and performance monitoring (exposure is sufficient).
  • Work with Azure tools both via Portal and Automation Scripts.


Skills

Core (Essential)

• Python

• Pytest - Unit testing

• OpenTelemetry (exposure)

• Poetry

• VS Code, Dev Containers

• SQL Querying

• CI/CD tools

• ADO/GitLab

• Pipelines for automation

Data Engineering (Highly desirable)

• PySpark

• SparkSQL

• Data file formats like Delta, parquet

Fabric (Not absolutely required but desirable)

• Fabric Notebooks

• Data Factory pipelines

• Kusto

• Data Flow Gen 2

Generalist Azure Skills (Some generalist Azure knowledge required - flexible on actual tools) (working with these tools via the Azure Portal and via Automation)

• ADLS Gen2

• Entra

• Azure Monitor

• App Service

• Functions

• Purview

• Azure SQL


Priyanka Sharma

Senior Delivery Consultant

Office:

Email:

Related Jobs

View all jobs

Data Engineer (DataBricks,Snowflake,Python) Hull / HYBRID

Python Engineer / Data Scientist

Data Scientist (eDV clearance required)

Principal Data Engineer / Architect

ML / AI Engineer - Python - £60,000 - Remote

Data Engineer - Databricks

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Data Science Jobs for Non‑Technical Professionals: Where Do You Fit In?

Beyond Jupyter Notebooks Ask most people what a data‑science career looks like and they’ll picture Python wizards optimising XGBoost hyper‑parameters. The truth? Britain’s data‑driven firms need storytellers, strategists, ethicists and project leaders every bit as much as they need statisticians. The Open Data Institute’s UK Data Skills Gap 2024 places demand for non‑technical data talent at 42 % of all data‑science vacancies—roles focused on turning model outputs into business value and trustworthy decisions. This guide highlights the fastest‑growing non‑coding roles, the transferable skills many professionals already have, and a 90‑day action plan to land a data‑science job—no pandas required.

McKinsey & Company Data‑Science Jobs in 2025: Your Complete UK Guide to Turning Data into Impact

When CEOs need to unlock billion‑pound efficiencies or launch AI‑first products, they often call McKinsey & Company. What many graduates don’t realise is that behind every famous strategy deck sits a global network of data scientists, engineers and AI practitioners—unified under QuantumBlack, AI by McKinsey. From optimising Formula One pit stops to reducing NHS wait times, McKinsey’s analytics teams turn messy data into operational gold. With the launch of the McKinsey AI Studio in late 2024 and sustained demand for GenAI strategy, the firm is growing its UK analytics headcount faster than ever. The McKinsey careers portal lists 350+ open analytics roles worldwide, over 120 in the UK, spanning data science, machine‑learning engineering, data engineering, product management and AI consulting. Whether you love Python notebooks, Airflow DAGs, or white‑boarding an LLM governance roadmap for a FTSE 100 board, this guide details how to land a McKinsey data‑science job in 2025.

Data Science vs. Data Mining vs. Business Intelligence Jobs: Which Path Should You Choose?

Data Science has evolved into one of the most popular and transformative professions of the 21st century. Yet as the demand for data-related roles expands, other fields—such as Data Mining and Business Intelligence (BI)—are also thriving. With so many data-centric career options available, it can be challenging to determine where your skills and interests best align. If you’re browsing Data Science jobs on www.datascience-jobs.co.uk, you’ve no doubt seen numerous listings that mention machine learning, analytics, or business intelligence. But how does Data Science really differ from Data Mining or Business Intelligence? And which path should you follow? This article demystifies these three interrelated yet distinct fields. We’ll define the core aims of Data Science, Data Mining, and Business Intelligence, highlight where their responsibilities overlap, explore salary ranges, and provide real-world examples of each role in action. By the end, you’ll have a clearer sense of which profession could be your ideal fit—and how to position yourself for success in this ever-evolving data landscape.