Is Zookeeper is must in Kafka
Yes, Zookeeper is very much required for running Kafka for the following reasons,
Role of Zookeeper in Kafka
* Zookeeper as a general purpose distributed process coordination system so kafka use Zookeeper to help manage and co-ordinate.
* Most recent version of Kafka will not work without Zookeeper.
* Zookeeper mainly used to track status of kafka cluster nodes, Kafka topics, and partitions etc.
* It maintain and co-ordination between different nodes in a cluster.
* Major role of Zookeeper is periodically commit offsets i.e in case of node failure it can recover the data from the previously committed offset.
* Zookeeper serve some of the vital roles like,
- Leader detection
- Configuration management
- Detecting when a new node joins or leaves the cluster, etc.
* Kafka uses zookeeper for Electing a controller –The controller is one of the brokers and is responsible for maintaining the leader relationship among all the partitions.
* It is responsible for Topic configuration like,
- Which topics exist.
- How many partitions each topic has.
- Where are the replicas.
- Who is the preferred leader.
- Which type of configuration overrides are set for each topic.