Kafka in Docker - Prelude Part Two - ZooKeeper and KRaft

In the first article in this series, I covered consensus algorithms and their application to Kafka. This article continues along a similar trajectory albeit with more practical applications.

ZooKeeper

ZooKeeper was the original metadata and coordination layer for Kafka. If you encounter a mature Kafka deployment then it’s likely that they are still using ZooKeeper.

Why Choose ZooKeeper?

• You need compatibility with older Kafka versions. ZooKeeper is required for all Kafka versions before 2.8 and it still widely supported through Kafka 3.x.
• Your existing infrastructure already relies on ZooKeeper. Many organizations already run ZooKeeper clusters for Hadoop, HBase, etc. In these instances it makes senses to keep using ZooKeeper to operational complexity.
• You’re in a production environment where stability is more important than modernization. ZooKeeper-backed Kafka has been around over a decade and is battle-tested at scale.
• You’re using a Kafka image or management tool that assumes ZooKeeper. As an example, some popular images and UI tools (like Kafka Manager, Confluent Platform) assume that ZooKeeper is present.

To summarize: ZooKeeper is still reliable, stable, and supported.

KRaft

KRaft (Kafka Raft) removes the ZooKeeper dependency by embedding Raft-based consensus directly into Kafka itself.

Why Choose KRaft?

• You want a simpler Kafka deployment as there is no need to run a separate ZooKeeper cluster. This simplifies configuration, operations, and monitoring.
• You’re starting fresh with Kafka 3.4+. KRaft became production-ready as of Kafka 3.5 and Kafka 4.0 drops ZooKeeper entirely.
• You’re optimizing for containerized or cloud-native deployments. With fewer moving parts, KRaft is more appealing in environments like Docker and Kubernetes.
• You want faster failover and better scalability for metadata operations. The internal Raft based controller is more scalable and responsive in large environments.

To summarize: KRaft is modern, simplified, and the future of Kafka.

Choosing the Docker Image For Your Needs

There are a lot of Kafka Docker images floating around and it can be overwhelming to pick the right one for your needs. This is because certain images will lock you into using ZooKeeper or KRaft. So, if you need ZooKeeper but you choose an image that uses KRaft you’re gonna have a bad time.

Popular Kafka Docker Images

From here we should be armed with enough knowledge to actually install Kafka in Docker. This will be the subject of the next article in this series.