Job Description

Requirements

Our client, a leading multinational information technology services and consulting company is recruiting for a Kafka Architect to join their business in the United Kingdom.

Job Description:

A Kafka Architect is responsible for designing, implementing, and managing Apache Kafka infrastructure and data pipelines. This role involves collaboration with various teams to understand business requirements and create scalable, efficient, and robust Kafka-based solutions to handle real-time data streaming needs. The Kafka Architect ensures the high availability, security, and performance of the Kafka ecosystem, providing technical leadership and guidance to developers and operations teams.

Key Responsibility : 

Design and Architecture:

Develop and implement the architecture for Apache Kafka infrastructure to meet business and technical requirements.

Design data pipelines and streaming applications using Kafka and associated technologies.

Create and maintain documentation of the Kafka architecture, including data flow diagrams, architecture diagrams, and operational procedures.

Implementation and Deployment:

Set up and configure Kafka clusters, topics, producers, consumers, and connectors.

Ensure secure, high-availability, and fault-tolerant Kafka environments.

Implement best practices for data retention, partitioning, replication, and monitoring.

Monitoring and Optimization:

Monitor Kafka clusters performance and troubleshoot issues related to scalability, latency, and throughput.

Optimize Kafka configurations for better performance and resource utilization.

Implement monitoring and alerting systems for Kafka health and performance using tools like Prometheus, Grafana, or equivalent.

Collaboration and Support:

Work closely with data engineers, developers, and IT operations to integrate Kafka with other systems and applications.

Provide technical support and guidance to development teams on Kafka-related issues.

Conduct training sessions and workshops to upskill teams on Kafka usage and best practices.

Innovation and Improvement:

Stay updated with the latest developments in Kafka and related technologies.

Evaluate and recommend new tools, technologies, and processes to improve the Kafka ecosystem.

Lead proof-of-concept projects to test new solutions and approaches.