Senior Data Engineer

About Quantum:

Quantum is building next-generation AI and trading technologies that harness cutting-edge research and data science. As part of our rapid growth, we are seeking a highly skilled Senior Data Engineer to support our Data & AI team by designing and maintaining robust, scalable, and production-grade data systems. This is greenfield project, allowing for full product ownership and key decision making.

Role Overview:

As a Senior Data Engineer at Quantum, you will be instrumental in building the infrastructure that powers our Data Science, AI and Trading tools. You will work closely with the Data and Technology teams to ensure data accessibility, quality, and scalability - particularly focusing on trading (time-series databases), analytical, and AI pipelines.

This is a highly collaborative role, ideal for someone who thrives on taking full ownership from systems design to implementation in a fast-paced, research-driven environment and wants to be part of building world-class trading system capabilities from the ground up.

Key Responsibilities:

• Develop and Maintain Data Pipelines:

Design, build, and optimise scalable data pipelines to support AI research and production systems, particularly for unstructured, text-heavy and time-series based datasets.

• Data Infrastructure Design:

Architect and implement data ingestion, transformation, storage, and retrieval systems, ensuring they are resilient, high-performing, and fit for future growth.

• Data Quality and Exploration:

Support data exploration efforts by ensuring high data quality, developing validation frameworks, and contributing to continuous data improvement initiatives.

• Collaboration with AI Teams:

Work closely with the Principal Data Scientist to operationalise RAG systems, fine-tune data retrieval processes, and optimise training datasets for AI model development.

• Automation and Optimisation:

Automate ETL (Extract, Transform, Load) processes, reduce manual intervention, and continuously identify opportunities to enhance the efficiency and reliability of data workflows.

• Support Research and Prototyping:

Build and maintain flexible data systems to support rapid experimentation, research validation, and the transition of prototypes into production environments.

• Monitoring and Troubleshooting:

Implement robust monitoring, logging, and alerting for data pipelines to proactively detect issues and maintain high availability and performance.

• Documentation and Best Practices:

Establish and maintain high standards for data engineering documentation, coding practices, and data governance.

Required Skills and Qualifications

• 5+ years of experience in Data Engineering, with a strong background in building data pipelines at scale.
• Proficiency with modern data technologies (e.g Apache Airflow, Spark, Kafka, Snowflake, or similar).
• Strong SQL skills and experience with cloud databases and data warehouses (AWS, GCP, or Azure ecosystems).
• Expertise in working with unstructured data and NLP-related datasets.
• Proficiency in one programming language, preferably Python with experience in data processing libraries such as Pandas, PySpark, or Dask.
• Familiarity with MLOps and deploying AI/ML models into production environments.
• Knowledge of Retrieval-Augmented Generation (RAG) frameworks or interest in learning and supporting RAG systems.
• Experience implementing scalable APIs and integrating data services with AI and analytics platforms.
• Strong understanding of data security, compliance, and governance best practices.
• Excellent collaboration and communication skills, able to work closely with technical and non-technical stakeholders.

Preferred Qualifications

• Experience supporting AI/ML research teams.
• Familiarity with LLM (Large Language Model) pipelines and vector databases (e.g. Pinecone, FAISS).
• Background in data versioning and experiment tracking (e.g DVC, MLflow).
• Familiarity with time-series datasets and databases.

Why Join Quantum?

• Work at the forefront of AI innovation with a team passionate about changing the future of trading and technology.
• Take ownership and make a direct impact from day one.
• Collaborate closely with world-class AI researchers, data scientists, and engineers.
• Opportunity for career growth as part of a rapidly expanding AI and data science team.

Quantum is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.