Jobs

Tech4 | Senior Data Engineer


Job details
  • Tech4
  • Liverpool
  • 1 week ago

Senior Data Engineer - Python / Data Pipelines / Data Platform / AWS - is required by fast growing, highly successful and and tech focused organisation.


About the job


You will play a crucial role in designing, building, and maintaining their data platform, with a strong emphasis on streaming data, cloud infrastructure, and machine learning operations.


Key Responsibilities:


  • Architect and Implement Data Pipelines:
  • Design, develop, and maintain scalable and efficient data pipelines
  • Optimize ETL processes to ensure seamless data ingestion, processing, and integration across various systems
  • Streaming Data Platform Development:
  • Lead the development and maintenance of a real-time data streaming platform using tools like Apache Kafka, Databricks, Kinesis.
  • Ensure the integration of streaming data with batch processing systems for comprehensive data management
  • Cloud Infrastructure Management:
  • Utilize AWS data engineering services (including S3, Redshift, Glue, Kinesis, Lambda, etc.) to build and manage our data infrastructure
  • Continuously optimize the platform for performance, scalability, and cost-effectiveness
  • Communications:
  • Collaborate with cross-functional teams, including data scientists and BI developers, to understand data needs and deliver solutions
  • Leverage the project management team to coordinate project, requirements, timelines and deliverables, allowing you to concentrate on technical excellence
  • ML Ops and Advanced Data Engineering:
  • Establish ML Ops practices within the data engineering framework, focusing on automation, monitoring, and optimization of machine learning pipelines
  • Data Quality and Governance:
  • Implement and maintain data quality frameworks, ensuring the accuracy, consistency, and reliability of data across the platform
  • Drive data governance initiatives, including data cataloguing, lineage tracking, and adherence to security and compliance standards


Requirements


Experience:

  • 3+ years of experience in data engineering, with a proven track record in building and maintaining data platforms, preferably on AWS
  • Strong proficiency in Python, experience in SQL and PostgreSQL. PySpark, Scala or Java is a plus
  • Familiarity with Databricks and the Delta Lakehouse concept
  • Experience mentoring or leading junior engineers is highly desirable


Skills:

  • Deep understanding of cloud-based data architectures and best practices
  • Proficiency in designing, implementing, and optimizing ETL/ELT workflows
  • Strong database and data lake management skills
  • Familiarity with ML Ops practices and tools, with a desire to expand skills in this area
  • Excellent problem-solving abilities and a collaborative mindset


Nice to Have:


  • Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes)
  • Knowledge of machine learning pipelines and their integration with data platforms


Great training and career development opportunities exist for the right candidate.

Basic salary £60-65,000 + excellent benefits

Office based in Northumberland. Fully remote working available

Sign up for our newsletter

The latest news, articles, and resources, sent to your inbox weekly.

Similar Jobs

Senior Data Engineer

Senior Data Engineer - Python / Data Pipelines / Data Platform / AWS - is required by fast growing, highly successful and and tech focused organisation.About the jobYou will play a crucial role in designing, building, and maintaining their data platform, with a strong emphasis on streaming data, cloud infrastructure,...

Tech4 Northumberland

Senior Data Engineer 85k + 10% Bonus

Senior Data Engineer -My clients are looking for an experienced Senior Data Engineer to join their dynamic team based in London. This is a fantastic opportunity to work in a fast-paced environment with a start-up feel while benefiting from the support and resources of a large, established organisation.The Role:This is...

City of London

Senior Data Engineer - Snowflake - £85,000 - London - Hybrid

Senior Data Engineer - Snowflake - £85,000 - London - HybridCompany Overview:My client is a global leader in the insurance industry, serving millions of customers worldwide. With strong financial foundations established over decades, their commitment to utilising innovative solutions and cutting-edge technologies is a key pillar of their recent success....

City of London

Senior Data Engineer

Senior Data Engineer - Python / Data Pipelines / Data Platform / AWS - is required by fast growing, highly successful and tech focused organisation.About the jobYou will play a crucial role in designing, building, and maintaining their data platform, with a strong emphasis on streaming data, cloud infrastructure, and...

Cramlington

Senior Data Engineer - Remote - £60k

Senior Data Engineer - Remote - £60kExciting opportunity for an experienced cloud data engineer to join an expanding data team who are using data in an exciting and advanced way. They will support your learning and development from a technical and leadership perspective as you help them design, build and...

Newcastle upon Tyne

Senior Data Engineer

£60,000 - £80,000 | Senior Data Engineer | Full Time | Flexible Hybrid | Fintech | AWS | Python | Terraform | Tech for goodYour new company:SR2 have supported this company with their growth from 3 co-founders to the 60+ team of awesome people they are today! If you want...

SR2 | Socially Responsible Recruitment | Certified B Corporation Bristol