Is Zookeeper is must in Kafka

Yes, Zookeeper is very much required for running Kafka for the following reasons,

Role of Zookeeper in Kafka

Zookeeper in Kafka

* Zookeeper as a general purpose distributed process coordination system so kafka use Zookeeper to help manage and co-ordinate.

* Most recent version of Kafka will not work without Zookeeper.

* Zookeeper mainly used to track status of kafka cluster nodes, Kafka topics, and partitions etc.

* It maintain and co-ordination between different nodes in a cluster.

* Major role of Zookeeper is periodically commit offsets i.e in case of node failure it can recover the data from the previously committed offset.

* Zookeeper serve some of the vital roles like,

  1. Leader detection
  2. Configuration management
  3. synchronization
  4. Detecting when a new node joins or leaves the cluster, etc.

* Kafka uses zookeeper for Electing a controller –The controller is one of the brokers and is responsible for maintaining the leader relationship among all the partitions.

* It is responsible for Topic configuration like,

  • Which topics exist.
  • How many partitions each topic has.
  • Where are the replicas.
  • Who is the preferred leader.
  • Which type of configuration overrides are set for each topic.