Databricks Subject Matter Expert (SME)

London (Hybrid Working)

6-Month Contract

As a Databricks SME, you will focus on delivering expert insights and hands-on leadership to build and optimize a robust Unified Data Platform. You’ll provide guidance on data architecture, pipeline development, system performance, and the integration of diverse data sources. Partnering with internal teams and vendors, you’ll ensure that solutions align with business objectives and technical best practices.

Key Responsibilities

• Platform Expertise: Serve as the primary SME for Databricks, driving the adoption and optimization of its capabilities across the organization.
• Architect and Design: Contribute to the design and development of the Unified Data Platform, ensuring it is scalable, efficient, and aligned with organizational goals.
• Pipeline Development: Build and optimize scalable, efficient data pipelines using Databricks and other tools, ensuring consistent code quality and deployment processes.
• System Integration: Guide the integration of Databricks with other systems, enabling unified access to diverse data sources.
• Vendor Collaboration: Collaborate with third-party providers to enhance and scale platform resources effectively.
• Legacy Migration: Provide expertise in migrating legacy data systems to Databricks, ensuring smooth transitions and the decommissioning of outdated infrastructure.

Essential Skills and Experience

• Databricks Mastery: Deep expertise in Databricks, with a proven track record of designing and managing scalable, high-performance data platforms.
• Data Engineering Tools: Advanced proficiency in PySpark and Python for creating and optimizing data pipelines and transformations.
• SQL Proficiency: Expertise in SQL for querying and managing large, complex datasets.
• Azure Ecosystem Knowledge: Extensive experience with Azure data services, including Azure Data Factory, Azure Synapse Analytics, and Azure Storage.
• Data Strategy: Strong ability to bridge the gap between business needs and technical solutions, delivering impactful data architecture strategies.