Job Overview: We are seeking an experienced Kafka Engineer with expertise in Confluent Kafka, Java/Scala, and distributed systems. The ideal candidate should be skilled in designing scalable, fault-tolerant Kafka-based data pipelines, troubleshooting messaging issues, and optimizing performance. A strong background in cloud deployments, microservices, and Agile development with an automate-first approach is essential.
Responsibilities:
- Identify and resolve Kafka messaging issues within a justified timeframe.
- Collaborate with business and IT teams to understand business problems and design, implement, and deliver appropriate solutions using Agile methodology within a larger program.
- Work independently to implement solutions across multiple environments (DEV, QA, UAT, PROD).
- Provide technical direction, guidance, and code reviews for other engineers working on the same project.
- Administer distributed Kafka clusters in DEV, QA, UAT, and PROD environments and troubleshoot performance issues.
- Implement and debug subsystems, microservices, and components.
- Follow an automate-first/automate-everything philosophy.
- Demonstrate hands-on experience with programming languages relevant to the role.
Key Skills & Expertise:
- Deep understanding of Confluent Kafka Proficient in Kafka concepts, including producers, consumers, topics, partitions, brokers, and replication mechanisms.
- Programming proficiency Expertise in Java or Scala, with potential Python usage depending on the project.
- System design and architecture Ability to design robust, scalable Kafka-based data pipelines considering data throughput, fault tolerance, and latency.
- Data management skills Knowledge of data serialization formats such as JSON, Avro, and Protobuf, and schema evolution management.
- Kafka Streams API (optional) Familiarity with Kafka Streams for real-time data processing within the Kafka ecosystem.
- Monitoring & troubleshooting Experience with Kafka cluster health monitoring, identifying performance bottlenecks, and troubleshooting issues.
- Cloud integration Experience deploying and managing Kafka on AWS, Azure, or GCP.
- Understanding of distributed systems concepts.
Must-Have Qualifications:
- 8-12 years of experience in software engineering.
- Kafka expertise Deep knowledge of Kafka producers, consumers, topics, partitions, brokers, and replication.
- Programming proficiency Strong in Java or Scala, with potential Python usage.
- System design & architecture Experience in designing high-throughput, scalable Kafka pipelines.
- Cloud & DevOps Experience deploying Kafka on AWS, Azure, or GCP.
- Monitoring & troubleshooting Familiarity with Kafka cluster health monitoring and performance tuning.