Raft, and Raft under the hood

4 · Raft: agreeing on the boring things

The control plane is an embedded Raft group (openraft), running inside every rift-cluster-server process and talking over the cluster port. Raft keeps a replicated log. One node is the leader; it appends entries and copies them to followers. An entry is committed once a majority has it on disk; after that it can never be lost or reordered, and every node applies committed entries in the same order.

Followers expect a heartbeat from the leader every 50 ms. A follower that hears nothing for a random 150–300 ms becomes a candidate: it increments the term, votes for itself, and asks for votes. A node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own. A majority of votes makes a leader. The randomness keeps two candidates from splitting the vote forever.

The simulation below runs that algorithm. Time is slowed about 15× and network delay is exaggerated so you can watch messages travel.

What the simulation leaves out, and what the real cluster adds on top:

• Log and vote are fsynced into redb before they're acknowledged. “Committed” therefore means on disk on a majority, which is why a full power loss replays every committed entry.
• A restarting member gets a 3 s grace before campaigning, and the leader runs a per-peer liveness ticker that bypasses the peer-health gate (D-22). Without both, a voter that restarts can campaign, bump its term, and never reconcile.
• Apply cannot fail. The leader validates a config before appending it. Applying an entry only writes the state machine's tables and calls the engine's incremental reconciler. A port that fails to bind on one node is reported as status on that node; it never stalls the log.
• The voter cap is 9. Further nodes join as learners: they serve traffic and replicate the log, but don't vote and own no flow keys (D-93), so consensus latency stays flat up to the 16-node ceiling.

5 · Raft under the hood

Section 4 is the algorithm. This section is what the cluster actually puts on the wire to run it. There are two moving parts. openraft is the engine: it owns terms, votes, logs, timers and the decision of what to send to whom. A small network layer in this crate carries those messages between processes, signs them, and decides how hard to try.

One port, signed HTTP

Every node binds one cluster port: --cluster-bind, which has no default because exposing it should be a deliberate choice. It tells peers where to reach it with --cluster-advertise, which defaults to the bind address and may be a hostname. Everything between nodes is a JSON POST over plain HTTP/1.1 on that port, from a pooled client with TCP_NODELAY, a 500 ms connect timeout and a 2 s request timeout:

There is no TLS on this port. It is meant for a private network, and a public bind address is refused unless acknowledged. What it does have is authentication on every single request. The sender signs it with HMAC-SHA256 under the shared --cluster-secret, over five length-prefixed fields: a timestamp, a random nonce, the method, the path with its query, and the whole body. The receiver checks in a fixed order, and the order is part of the design: cheap checks that need no secret come first, and the nonce is recorded last, so an unauthenticated sender can't fill the replay cache. A request sent to a member, rather than to an address, also names its recipient: ?to=<node id>, inside the signed path. After the signature checks, and before any handler runs, the receiver refuses a request named for another node (D-96). Build a request below and tamper with it.

Two details are easy to miss. The protocol check runs before the body is read, so a node on a different major version is turned away with 426 before it costs any parsing. And the nonce cache (100,000 entries, each kept for 61 s, twice the 30 s skew allowance plus one) fails closed: if it is full of live entries, new requests are refused rather than admitted unchecked. A node run with --cluster-insecure skips signing and verification entirely and says so loudly at startup.

Finding each other: seeds and the membership

Nodes don't discover each other by broadcasting. A new node knows one thing, its --cluster-seeds, and uses them for exactly one request: join. From then on, the source of truth for “who is in the cluster, and where” is the Raft membership itself. Every member's advertise address is stored, verbatim, in the committed membership entry, so every node reads the same peer list at the same log index.

• Addresses are re-resolved on every send. A hostname is looked up again each time a message goes out, and every address it yields is tried in the resolver's order (D-28). A pod that comes back with a new IP behind the same name is reachable at once.
• The leader is learned, not configured. Each node knows the current leader from its own Raft state. A request that needs the leader and reaches a follower is answered with the leader's address: a 421 for a join or a retire, a typed ForwardTo reply for a forwarded admin write. The sender re-sends there, at most three hops. That is how a join seeded at a follower, an admin write that landed anywhere and a retire all find the leader.
• An address is where to dial, not who answers. A pod rescheduled without its volume can come back at a dead member's address under a new id. Every Raft message and member call names the member it is for, so that node refuses traffic meant for the dead member instead of answering for it, and the leader logs once that another node answers there (D-96). Requests that dial an address on purpose to find out who is there, such as a join through a seed or the retire probe, name no recipient.

The timers

These values are fixed, not operator knobs, and a test pins them (D-42): the isolation window, the probe and the snapshot timeouts are all derived from them, and changing one alone breaks the arithmetic that keeps the others safe.

An election, message by message

The second heartbeat

openraft's heartbeat is an ordinary AppendEntries with no entries. It works until the leader has something big in flight to a follower: a multi-megabyte append, or a snapshot being installed chunk by chunk. While that transfer is running, openraft sends that follower nothing else, and a follower that hears nothing for longer than its election timeout campaigns. Its term then runs ahead of the leader's, the leader refuses it, and the pair never reconcile.

So the network layer runs its own liveness ticker for each peer. Every 50 ms, if this node is the leader right now (read live, not cached) and nothing has gone to that peer in the last 50 ms, it sends a minimal AppendEntries carrying only the leader's vote and no log position. On the follower the vote is checked before the log, so this refreshes its election timer and can never truncate anything. It goes through the probe path, which bypasses the peer-health gate (D-22): a heartbeat refused locally because the peer “recently failed” is exactly how a restarted voter would never hear the leader. It waits at most 150 ms for a reply (D-83), because its job is done once it arrives.

Replicating entries

• Batches. Up to 300 entries per AppendEntries. A follower that answers 413 makes the leader halve the batch immediately.
• One transfer per peer. A large append runs detached with its own deadline (2 s plus 1 s per MiB), and a re-send of the same batch joins it rather than starting a second copy.
• Finding the match point. If a follower's log disagrees at prev_log_id, it answers conflict, and the leader binary-searches back to the last index they agree on, then sends from there. The follower truncates its unagreed tail.
• Committing. Once a majority has an entry, the leader advances its commit index; followers learn it from the next AppendEntries, heartbeat or not.
• Snapshots. A follower that needs purged entries gets the snapshot in 1 MiB chunks, each with 10 s to land. Between chunks, only the liveness ticker keeps its election timer fed.