Analytics PipelineClickHouse & Materialized Views
Guide ClickHouse Core Architecture & Analytics Fundamentals

Recovering from ClickHouse Keeper Quorum Loss

When a ClickHouse Keeper ensemble loses quorum — two of three nodes down, a split brain, a corrupted log — every ReplicatedMergeTree table backed by it flips read-only within seconds and inserts start failing with KEEPER_EXCEPTION. Recovery is a specific sequence: confirm the quorum is actually gone using the 4-letter-word commands, restore or reconfigure the ensemble to a healthy majority, then repair the ClickHouse replicas that went read-only during the outage. This page walks the quorum math, the detection queries, and the full recovery path.

Prerequisites

Quorum math: why a minority ensemble is dead

Keeper commits a write only when a strict majority of the configured ensemble acknowledges it. An ensemble of N nodes needs floor(N/2) + 1 alive to make progress and tolerates the loss of floor((N-1)/2). A 3-node ensemble survives 1 failure and needs 2 alive; a 5-node ensemble survives 2 and needs 3. Lose one past that and the survivors cannot elect a leader, so they refuse all writes — which is exactly what read-only ClickHouse tables are reacting to. This is also why even-numbered ensembles are a trap: 4 nodes tolerate only 1 failure, the same as 3, while costing an extra machine and a wider split-brain surface.

Keeper quorum states: healthy majority, tolerated single failure, and quorum loss A three-node Keeper ensemble needs two nodes for a majority. Three alive or two alive keeps a leader and read-write tables. One alive is a minority, no leader is elected, and replicated tables go read-only. 3 alive — majority leader elected · writes commit tables read-write 1 down — still majority 2 of 3 alive · leader holds tables read-write · tolerated 2 down — quorum lost 1 of 3 · minority · no leader tables read-only Majority needed = floor(N / 2) + 1 N = 3 → need 2 alive, tolerate 1 failure. N = 5 → need 3 alive, tolerate 2 failures. Even N never helps: 4 nodes tolerate only 1 failure, the same as 3, with a wider split-brain surface.

Step 1 — Confirm the ensemble has actually lost quorum

Do not restart anything until you know the Keeper ensemble is leaderless rather than just slow. The mntr 4-letter word reports each node’s role and whether a leader exists. Send it to every Keeper node.

bash
# Ask each node its state. On a healthy ensemble exactly one reports leader.
for host in keeper-1 keeper-2 keeper-3; do
  echo "== $host =="
  echo mntr | nc -q1 "$host" 9181 | grep -E 'zk_server_state|zk_followers|zk_synced_followers'
done

Expected output during quorum loss (no node is leader):

text
== keeper-1 ==
zk_server_state	follower
== keeper-2 ==   (unreachable)
== keeper-3 ==   (unreachable)

stat gives the same role plus outstanding request counts, and ruok returns imok only if the node process is up (it says nothing about quorum). A Keeper ensemble where no node reports zk_server_state leader and followers cannot sync is quorum-lost.

Step 2 — See the read-only fallout on the ClickHouse side

From any ClickHouse node, confirm the tables that went read-only and read the Keeper exception they are raising. This is the blast radius you are recovering.

sql
SELECT database, table, is_readonly, is_session_expired, zookeeper_exception
FROM system.replicas
WHERE is_readonly = 1
ORDER BY database, table;

Every replicated table on the ensemble should show is_readonly = 1 with a zookeeper_exception mentioning connection loss or session expiry. Inserts against them are failing right now — clients should already be rerouting per your fallback policy.

Step 3 — Restore the ensemble to a healthy majority

There are two recovery paths depending on how many Keeper nodes survived.

Path A — a majority of hosts can be brought back. Start the down Keeper processes so a majority is alive again. They rejoin, elect a leader, and replay their logs. No data operation is needed; quorum returns on its own once floor(N/2)+1 nodes are up.

bash
systemctl start clickhouse-keeper   # on each recoverable node
echo mntr | nc -q1 keeper-1 9181 | grep zk_server_state   # expect one 'leader'

Path B — only a minority survives and the down nodes are gone for good. You must force the surviving node into a new single-node quorum, then grow the ensemble back. On the survivor, reconfigure it as a standalone one-node ensemble and restart, then add fresh nodes with reconfig.

xml
<!-- keeper survivor: temporarily a 1-node ensemble so it can elect itself leader -->
<keeper_server>
    <server_id>1</server_id>
    <raft_configuration>
        <server><id>1</id><hostname>keeper-1</hostname><port>9234</port></server>
    </raft_configuration>
</keeper_server>
bash
# After the survivor is leader again, dynamically add replacement nodes:
clickhouse-keeper-client -h keeper-1 -p 9181 -q "reconfig add '2=keeper-2:9234'"
clickhouse-keeper-client -h keeper-1 -p 9181 -q "reconfig add '3=keeper-3:9234'"

Path B rolls the ensemble forward from the survivor’s log, so any writes that never reached the survivor are lost — which is why replicas may need re-registering in Step 4.

Step 4 — Repair the ClickHouse replicas

Once a leader exists, the ClickHouse tables do not always clear on their own — especially after Path B, where the Keeper metadata may be older than the local parts. Re-establish sessions first, and rebuild metadata only where it is genuinely gone.

sql
-- Re-open sessions on every replicated table on this node.
SYSTEM RESTART REPLICAS;

-- For any table still read-only because its Keeper path was rolled back, rebuild it from local parts.
SYSTEM RESTORE REPLICA analytics.events_repl;

The RESTART / RESTORE decision is the same one covered in depth in executing replica failover DDL — run RESTORE on exactly one replica per table and let the peers re-attach.

Verification

Confirm the ensemble is a healthy majority and every ClickHouse table is writable again. First the Keeper side:

bash
echo mntr | nc -q1 keeper-1 9181 | grep -E 'zk_server_state|zk_synced_followers|zk_outstanding_requests'

A recovered 3-node ensemble shows one leader, zk_synced_followers 2, and zk_outstanding_requests 0. Then the ClickHouse side:

sql
SELECT count() AS still_readonly
FROM system.replicas
WHERE is_readonly = 1;

still_readonly must be 0. Finally prove writes commit end-to-end by inserting a canary row and reading it back:

sql
INSERT INTO analytics.events_repl (event_id, event_ts) VALUES (generateUUIDv4(), now64(3));
SELECT count() FROM analytics.events_repl WHERE event_ts > now() - INTERVAL 1 MINUTE;

A non-zero count that survives a SYSTEM SYNC REPLICA confirms the ensemble is committing again.

Gotchas and edge cases

  • Never rm the survivor’s log to “start clean.” In Path B the surviving node’s log is the only record of committed writes. Wiping it and starting fresh discards every acknowledged transaction, and the ClickHouse replicas then converge on an empty Keeper state, silently dropping recent parts. Snapshot the log directory before any reconfiguration.
  • ruok/imok is not a quorum check. A node answering imok only means its process is alive; it can be a follower that has lost the leader and is refusing writes. Always decide quorum from zk_server_state and zk_synced_followers in mntr, never from ruok.
  • Distributed DDL stays frozen until Keeper recovers. The ON CLUSTER queue lives in Keeper, so any schema change issued during the outage silently queues and only executes once quorum returns — sometimes replaying a stale statement. Drain system.distributed_ddl_queue deliberately after recovery rather than assuming it is empty.
  • An even-sized ensemble buys nothing. Teams often add a 4th Keeper node “for safety” and reduce availability: 4 nodes still tolerate only 1 failure but double the chance of a split brain. Keep ensembles at 3 or 5, and watch member health through monitoring replication lag with system.replicas so you replace a dead node before a second one takes quorum with it.

Up: Fallback Routing & High Availability