Membership, and what survives · How RiftCluster Works

15 · Joining, leaving, retiring

Membership changes only by a node joining or leaving, or by an operator retiring a member that is gone (D-95). Each transition is a committed log entry that the node does not drive itself. Click a stage.

From one node to a fleet

A cluster is never “configured” as three nodes. It is founded by one node and grown one join at a time, and every step is a committed membership entry. Step through a three-node fleet forming. The Fleet column tracks the number that matters: how many voters a majority needs, and how many failures the fleet survives.

Two details in that sequence carry most of the safety. Founding is explicit: only a node started with --cluster-allow-solo and no seeds will found a cluster, and a node that has seeds and can't reach them gives up after 30 s instead of founding its own. Two nodes that each founded a cluster would be two fleets that can never merge, so that mistake is refused at start rather than repaired afterwards. The joiner asks; the leader decides: the joining node only ever sends one signed request, and the leader writes the membership entries. That is D-21: what can enter the fleet is bounded by what an operator chose to start, with the cluster secret.

Past three, every join is the same step. Nodes four through nine become voters (a five-voter fleet survives two failures). The tenth and later stay learners, which serve traffic but neither vote nor own flows. A joiner that needs a large snapshot is answered as a learner with catching_up: true rather than holding its join open; the leader's 1 s sweep promotes it once its replication is within 16 entries.

Cold start, the payoff

A full-cluster restart needs no special handling: every node reopens its redb, the group re-forms from the persisted vote, log and snapshot, elects a leader, and replays. A deletion is a log entry, so it can't resurrect. An empty-disk node is a learner catching up from a snapshot. If every disk is empty there is no group to re-form, and nothing serves until an operator re-initializes: a loud refusal, as R3 demands.

The case that needs care is a single member restarting, not the whole fleet. That is what the 3 s restart grace, the leader's 50 ms retries and the liveness ticker in section 4 exist for.

16 · What survives what

The volatile rows are decisions, not gaps: each would cost hot-path writes to preserve state whose value ends with the test run.

State	Lives in	Node crash	Full restart
Membership, node ids	Raft log + vote (`raft.redb`, fsync)	survives	survives
Imposter configs, `enabled`, revisions	Raft state machine	survives	survives
Route table, session key	Raft state machine	survives	survives
Admin intents + op-id dedup	Raft SM + receiving node's `pending_intents`	survives	survives
Flow state @ `sync`	FlowShard, fsync per ack	survives	zero loss
Flow state @ `async` (default)	FlowShard, group fsync per 50 ms	survives	≤ 1 interval, only if all 3 holders die
Flow state @ `none`	memory	via replicas	lost (opted)
Sequence cursors	memory	reset (D-8)	reset
Request journal, counters	memory, per node	that node's lost	lost
proxyOnce `Pending`	owner memory	re-claimable	lost
proxyOnce `Recorded` + stubs	config (Raft SM)	survives	survives

The whole design in four sentences

R1 · visible everywhereThe write barrier holds the 2xx until every Ready member has applied the entry (up to 2 s); a straggler is named unapplied=, and a member that can't be heard from unreachable=, in Rift-Cluster-Warnings.

R2 · next request sees itOwner-routed reads; if the owner is unreachable, a fast, honest 503, not a stale answer.

R3 · survives restartAbsolute at sync. At async, the loss window is one fsync interval, and only if all three holders die inside it.

R4 · never lostThe intent is parked on disk before forwarding; a 503 carries an op-id whose fate you can query.