Moving Kafka off ZooKeeper to KRaft without downtime

Why this cannot wait much longer

Since Kafka 4.0, ZooKeeper mode no longer exists. Every security fix and feature from here on lands in 4.x. A cluster still on ZooKeeper is therefore pinned to a release line that will stop receiving fixes. The upgrade to 4.x cannot start until the metadata has moved off ZooKeeper, and that move is the risky part.

KRaft replaces ZooKeeper with a small group of Kafka controllers that keep the cluster's metadata (topics, partitions, configurations, ACLs) in an internal, replicated log. Fewer moving parts, faster failover, and one system to secure and monitor instead of two.

Before you start: the inventory

Most of the effort is here, not in the migration itself. Before touching production, we go through this list:

• Version. Upgrade to the latest 3.x release (3.9.x) first, as its own change, and let it settle. Migration on very old 3.x releases is not recommended.
• Anything that talks to ZooKeeper directly. Old admin scripts, homegrown tools that read znodes, monitoring that scrapes ZooKeeper, and automation that creates SCRAM users through ZooKeeper. After migration, these either fail or, worse, change ZooKeeper data that no longer matters.
• ACLs. Export them and check for malformed entries. ACLs are copied as-is, and a malformed one can stop the new controllers from starting.
• SCRAM users and configuration overrides. Count them before and after; you will verify the counts later.
• Clients. Kafka 4.0 removes old protocol versions, so Java clients need to be on 2.1 or newer before the later 4.0 upgrade. This does not block the migration, but finding a 1.x client in a corner of the estate now is cheaper than finding it during the 4.0 roll.
• Java. Kafka 4.0 brokers need Java 17. Plan the runtime upgrade alongside.
• Disks. If brokers use several log directories, any failed directory must be repaired before that broker can migrate.

Sizing the controllers

Run three controllers (the quorum survives one failure) or five (it survives two), on dedicated machines or pods, spread across availability zones. They do not need much CPU or memory, but they write the metadata log constantly, so give them fast, dedicated disks. Avoid running controllers and brokers in the same process (combined mode) for production. And open the controller listener port between every broker and every controller before you begin; a missing firewall rule is the most common reason a broker will not rejoin after its restart.

The procedure

The order matters, and each phase should be its own change window, with checks before moving on.

1. Find the cluster ID

   The new controllers must join the existing cluster, so they are formatted with its ID. Read it from ZooKeeper:

   bin/zookeeper-shell.sh zk-1:2181 get /cluster/id
   # {"version":"1","id":"WZEKwK-bS62oT3ZOSU0dgw"}

2. Start the KRaft controllers in migration mode

   Each controller gets a node.id that does not clash with any existing broker.id. Use a static controller list for the migration.

   # controller.properties
   process.roles=controller
   node.id=3000
   controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
   controller.listener.names=CONTROLLER
   listeners=CONTROLLER://:9093
   
   # migration settings
   zookeeper.metadata.migration.enable=true
   zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
   
   # so the controllers can reach the brokers' internal listener
   inter.broker.listener.name=INTERNAL
   listener.security.protocol.map=CONTROLLER:SSL,INTERNAL:SSL

   bin/kafka-storage.sh format --config controller.properties --cluster-id WZEKwK-bS62oT3ZOSU0dgw
   bin/kafka-server-start.sh controller.properties

   The controllers start and wait. Nothing has changed for clients yet.

3. Enable migration on the brokers

   Add the migration settings to every broker and do a rolling restart, one broker at a time, waiting for under-replicated partitions to return to zero before the next.

   # added to each broker's server.properties (broker.id stays as it is)
   inter.broker.protocol.version=3.9
   zookeeper.metadata.migration.enable=true
   controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
   controller.listener.names=CONTROLLER
   listener.security.protocol.map=CONTROLLER:SSL,INTERNAL:SSL,EXTERNAL:SSL

   When the last broker has restarted, the active controller copies all metadata from ZooKeeper into its log and takes over as the cluster's controller. Look for this line in the active controller's log:

   Completed migration of metadata from ZooKeeper to KRaft

   The cluster is now in dual-write mode: the KRaft controllers are in charge, and they also write every metadata change back to ZooKeeper. That is what keeps rollback possible.

4. Verify before going further

   Compare topic and partition counts, ACL counts, SCRAM users and configuration overrides with the inventory. Run a canary producer and consumer through the whole procedure and watch their error rate and latency. Check that under-replicated partitions are zero and that controller failover works (stop the active controller and watch another take over).

5. Move the brokers to KRaft mode

   On each broker, replace broker.id with node.id (same number), add process.roles=broker, remove the ZooKeeper and migration settings, and keep the controller settings. If you use ACLs, switch the authorizer in the same change:

   process.roles=broker
   node.id=1
   controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
   controller.listener.names=CONTROLLER
   authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
   # removed: broker.id, zookeeper.connect,
   #          zookeeper.metadata.migration.enable, inter.broker.protocol.version

   Roll the brokers one at a time again. Watch for authorization failures in client logs straight after each restart; they are the first sign of an ACL that did not carry over the way you expected.

6. Wait, then finalize

   Run in this state for a week or two, through a normal release cycle. When you are confident, remove zookeeper.metadata.migration.enable and zookeeper.connect from the controllers and restart them one by one. After this step there is no way back to ZooKeeper. ZooKeeper can then be decommissioned, after you have confirmed that nothing still connects to it.

7. Upgrade to Kafka 4.x as a separate change

   Only now is the cluster eligible for 4.x. Treat it as its own upgrade with its own rehearsal, because 4.0 brings its own changes: Java 17 on brokers, removed client protocol versions, and the new consumer rebalance protocol, which is on by default on the server but which clients must opt into.

Rolling back

Until finalization, every phase can be reversed, but the steps are manual and differ by phase:

Write these steps into your runbook with your real hostnames, and rehearse at least the "brokers in KRaft mode" rollback on staging. It is the one people get wrong under pressure.

How we rehearse it

• Build a staging cluster with a production-like number of topics, partitions and ACLs. Metadata volume, not message volume, decides how long the copy takes.
• Run canary producers and consumers through the whole rehearsal, and record their errors and latency during each roll.
• Time each phase, so the production change windows are realistic.
• Do the full rollback from "brokers in KRaft mode" at least once, then migrate forward again.
• Search every repository and scheduled job for ZooKeeper connection strings, and fix them before the production run, not after.