Write-ahead logs and convergence

13 · Two write-ahead logs

Every node keeps two redb files in its state directory, and both are write-ahead logs in the strict sense: nothing is acknowledged until the record that makes it recoverable is on disk, to the degree that write's contract demands. The degree is what differs. raft.redb, the control plane's log, pays an fsync on every append. flow.redb, the data plane's, pays one fsync per 50 ms for everything written in between. They are separate files on purpose: a redb file has one commit lock, and a flow-state group fsync must never queue behind, or in front of, a Raft append.

The one redb rule both depend on

A redb write transaction commits at a durability level. Immediate: the data is on disk when commit() returns, because redb fsyncs before returning. None: the commit is visible to readers and correctly ordered, but sits in the OS page cache, and becomes durable only when a later Immediate commit lands. That second half is the whole trick behind group commit: many cheap None commits, then one Immediate commit makes all of them durable together.

The Raft log, one entry on one follower

The entry itself is small JSON: a ControlRequest carrying the op_id, the operation (PutImposter, a stub patch, a delete…), and issued_at_secs, which is the state machine's only clock. Nothing in apply reads the local time, so every node computes the same result from the same entry.

Snapshots keep the log from growing forever. Every 5,000 entries openraft builds one from the state machine; the payload is written to a temporary file, fsynced, renamed into place, the directory is fsynced, and only then is a row naming it committed, so a crash at any point leaves either the old snapshot or the new one, never half of one. The 1,000 most recent entries are kept and older ones purged. A follower that needs a purged entry is sent the snapshot instead (section 14).

The third table in raft.redb, pending_intents, is the R4 log from section 6: the receiving node writes each admin request there with an Immediate commit before forwarding it, and clears it once the outcome is committed. It is local to that node: it is never replicated and never part of a snapshot.

The flow log, under load

Flow writes arrive at mock-traffic rates, so the writer never fsyncs per write in the default mode. The figure runs that writer: every acknowledged write is a bar on the top lane, red until a group fsync covers it and green after. The gauge is the node's real metric, rift_cluster_flow_wal_lag_ops: acknowledged writes that are committed at None but not yet fsynced. It is exactly what a power cut on this node, right now, would take off this disk.

Two things keep the async window honest. Each write was also pushed to the key's two replicas, which run their own writers and their own 50 ms tickers, so a write is lost from the cluster only if all three holders lose power inside the same interval (section 12's figure). And replicas persist in the origin's mode: the durability travels with the replication message, so an imposter that asked for none is not quietly written to disk by its replicas, and one that asked for sync is not demoted.

14 · How the fleet converges

“Consistent” means two different things here, because the two planes make different promises.

Config converges by replaying one log. Every node applies the same committed entries in the same order, and apply is deterministic: the leader validated the entry before appending it, and the only clock apply reads is the entry's own. So two nodes at the same applied index hold identical config, and the revision a client sees is the log index. Two nodes can only ever differ by lag, one behind the other, never by content. Lag is closed three ways:

1. Backfill. The leader tracks how far each follower's log matches its own and resends from there. On a mismatch it walks back until the logs agree, and the follower truncates its unagreed tail. A node that restarts, or comes back from a partition, catches up this way at heartbeat speed.
2. Snapshot install. If the entries a follower needs have been purged, the leader sends its latest snapshot. The follower replaces every state-machine table in one durable transaction, then continues by backfill from the snapshot's index.
3. Admission. A new node joins as a learner and catches up by the same two mechanisms. The leader's 1 s sweep promotes it once its replication lag is within 16 entries, while there are fewer than 9 voters.

The barrier from section 6 is what turns will converge into R1's has converged by the time you see the 2xx. Drive it below: write, slow a node's apply, crash one, or add an empty node and watch it catch up from a snapshot.

Flow state converges toward its owner. There is no log to replay. Each key has one writer, the owner at the current m_idx, and every other copy follows it. The owner pushes each accepted write to the key's two successors, fire-and-forget, and acknowledges without waiting for them. Two mechanisms close the gaps a lost push leaves:

• Anti-entropy. Every 5 s each node pulls, from their owners, the flows it holds but doesn't own, and merges by the highest (m_idx, v, origin).
• Adoption. The first time a new owner touches a flow after a membership change, it pulls that flow from the other holders and merges before serving.

A strong read always goes to the owner, so a replica's lag is never visible to it. Only an imposter that opted into readConsistency: "local" can see it, by up to one push or one 5 s pull.

What you can rely on