Kafka SME

Requirements

We are seeking a highly skilled Kafka Subject Matter Expert (SME) to lead the architecture management and scaling of Kafka environments in 24x7 production settings. The ideal candidate will have a deep understanding of Apache Kafka a proven track record of resolving complex production and nonproduction issues and strong leadership capabilities. As a Kafka SME you will work closely with crossfunctional teams provide technical guidance to junior engineers and ensure the high availability scalability and reliability of our Kafka infrastructure.

Key Responsibilities:

• Kafka Management:
  

  • Design build manage and upgrade Kafka clusters in production and nonproduction environments.
    
  • Ensure the Kafka infrastructure supports 24x7 availability for missioncritical customerfacing applications.
    
  • Monitor Kafka performance and proactively address issues related to scalability availability and performance.
    

• Incident Management:
  

  • Troubleshoot and resolve complex Kafkarelated production incidents in a timely and efficient manner.
    
  • Implement strategies to reduce incident resolution time and improve the reliability of Kafka services.
    
  • Provide root cause analysis and implement longterm fixes for recurring issues.
    

• Leadership & Mentoring:
  

  • Act as a technical leader for Kafkarelated projects and initiatives guiding and mentoring fellow engineers.
    
  • Lead the efforts in managing upgrading and scaling Kafka infrastructure across multiple environments.
    
  • Provide technical direction and input for key business and infrastructure initiatives ensuring best practices are followed.
    

• Infrastructure as Code (IaC):
  

  • Develop and maintain Infrastructure as Code (IaC) using Terraform Helm and optionally Ansible to automate Kafka cluster provisioning scaling and management.
    
  • Ensure that Kafka infrastructure is fully automated and scalable adhering to DevOps and Agile methodologies.
    

• Operational Excellence:
  

  • Establish and follow standard operating procedures for managing Kafka clusters including patching upgrading and scaling.
    
  • Implement Kafka monitoring and alerting tools to ensure continuous health checks and performance monitoring.
    
  • Collaborate with DevOps and development teams to integrate Kafka into CI/CD pipelines and ensure seamless delivery of updates and features.
    

• Documentation & Continuous Improvement:
  

  • Maintain comprehensive documentation for Kafka infrastructure including architectural diagrams operational processes and troubleshooting guides.
    
  • Continuously improve Kafka operations by adopting the latest tools technologies and best practices to drive performance and cost efficiency.
    

Required Skills & Qualifications:

• Kafka Expertise: Extensive handson experience with Apache Kafka including Kafka architecture configuration and performance tuning.
  
• Production Environment Experience: Proven ability to manage Kafka clusters in highdemand 24x7 customerfacing production environments.
  
• Incident Management: Strong experience in diagnosing troubleshooting and resolving complex production and nonproduction incidents in the shortest time possible.
  
• Leadership Skills: Ability to lead a team of engineers provide technical guidance and mentor junior team members.
  
• IaC Expertise: Proficiency in Infrastructure as Code (IaC) using tools like Terraform Helm and optionally Ansible.
  
• Automation & Monitoring: Experience with automating Kafka operations and using monitoring tools to track the health and performance of Kafka clusters.
  
• Problem Solving: Strong analytical and problemsolving skills with the ability to think critically and provide innovative solutions.
  
• SelfMotivation: Highly selforganized motivated and able to work independently with minimal supervision.
  
• Collaboration: Strong communication and collaboration skills with the ability to work in a fastpaced crossfunctional environment.
  

Preferred Qualifications:

• Experience with Ansible: Experience using Ansible for Kafka automation and configuration management is a plus.
  
• Cloud Platforms: Familiarity with deploying and managing Kafka in cloud environments such as AWS Azure or Google Cloud.
  
• DevOps & CI/CD: Handson experience integrating Kafka with CI/CD pipelines and DevOps practices.
  
• Scripting & Automation: Proficiency in scripting languages such as Python Bash or Java for Kafkarelated automation tasks.
  

Benefits