Kafka 3.3 replaces ZooKeeper with the KRaft Consensus Protocol

Published on Oct 27, 2022

Apache Software Foundation has released Apache Kafka 3.3.1 with many new features and enhancements. Specifically, this is the first release to mark the production readiness of the KRaft (Kafka Raft) consensus protocol. After several years of development, Kafka 2.8 was released in early access, followed by Kafka 3.0 in preview.

KRaft is a consensus protocol developed for managing metadata directly in Apache Kafka. This simplifies Kafka’s architecture by consolidating metadata management into Kafka itself without the need for a third-party tool such as Apache ZooKeeper. By using KRaft mode, partition scalability and resilience can be improved, while deployments of Apache Kafka can now be simplified.

With KRaft, a new quorum controller ensures that metadata is replicated correctly across the quorum. The active controller stores the metadata in an event-sourced log topic, and the other controllers within the quorum respond to the events that the active controller generates. Periodically, the event log is snapshotted to ensure that it does not grow indefinitely. As opposed to ZooKeeper-based controllers, quorum controllers do not require state loading from ZooKeeper in the event of issues, as internal cluster state is already stored in the metadata topic. Thus, the unavailability window is significantly reduced, improving the worst-case recovery time.

With the new metadata management system, the KRaft consensus and quorum controller can scale Kafka clusters to millions of partitions by improving control plane performance; improving stability, and making it easier to monitor, administer, and support Kafka; allowing Kafka to have a single security model for the entire system and making controller failover near-instantaneous.

According to the Kafka community, ZooKeeper will be deprecated in the next release (3.4) and then removed entirely in version 4.0.

A number of new features have been added to Apache Kafka 3.3, such as metadata log processing error-related metrics, delegation tokens for other users, and a uniform sticky partitioner to improve partitioning performance.

This release adds source/sink metrics for consumed/produced throughput, pause/resume topologies, and consolidates the KStream transform() and process() methods. Exactly-Once support has been added to Kafka Connect.