Flow ownership and stateful stubs

9 · Who owns a flow

Every flow key has exactly one authoritative owner, computed, not negotiated, by rendezvous (highest-random-weight) hashing:

owner(key) = argmax over voters n of xxh64(n.id ‖ class ‖ key)

Each voter gets a score for the key; the highest score wins; the next two are the key's replica successors. The voter list comes from the committed membership, so every node at the same applied index computes the same answer. The key is the scoped flow id: the store prefixes it with the imposter's namespace, i8080: by default, so two imposters reading the same header value don't share state.

The figure below runs the real function: node ids are derived from their names the way --cluster-node-name derives them, and each chip sits under its owner. Remove a node and watch which flows move.

Removing a node moves only the flows it owned. Every other flow keeps its owner, because each voter's score is independent of the others. Adding a node moves only the flows it now wins. No coordination happens: ownership transfer is not a protocol. It is a deterministic consequence of a committed membership entry, effective at that entry's index (m_idx).

10 · The state behind a stateful stub

Section 9 says who owns a flow. This section is about what a flow holds, and what each kind of stateful stub does to it.

A flow is a small key-value map, named by a flow id. The imposter decides where the id comes from (_rift.flowState.flowIdSource): "imposter_port", the default, gives the whole imposter one flow, and "header:X-Flow-Id" gives every distinct header value its own, falling back to the port when the header is absent. The store then scopes the id, i8080: by default, so two imposters that see the same header value don't share state; contextScope: "fleet" opts into one shared f: namespace instead. The scoped id is the flow's key on the ring, so one owner holds every key of a flow, which is what lets a scenario read and update it atomically.

The path of one stateful operation

1. The engine calls its flow-store interface, synchronously, on a blocking thread. That interface is the seam where the cluster plugs in; the engine itself doesn't know there is a cluster.
2. A bounded bridge carries the call into the node's cluster runtime. It needs a permit, and if none is free the call is shed at once rather than queued; it has a 2 s deadline. This is what keeps stateful traffic from starving stateless traffic when an owner is slow.
3. The node computes the owner from its ring at its current m_idx. If that is itself, the operation runs locally; otherwise it is one signed RPC to the owner.
4. The owner checks, in order: the caller's m_idx matches its own (else fenced); it really is the owner; it is not isolated (D-17); the flow has been adopted at this m_idx. Then, under a per-flow lock, it applies the operation, stamps the new version (m_idx, v+1, origin), writes its shard (memory now, disk per the durability mode), pushes the result to the two successors, and answers.
5. Any failure in steps 2–4 reaches the engine as backend unavailable, which the stub turns into an honest 503 rather than a guess.

Everything else in the next three sections is how those copies survive: replicas and fencing when ownership moves, the shard on disk, and anti-entropy when a push is lost.

11 · Replicas, handoff and fencing

The owner serializes every write and every correctness-bearing read for its keys. Each accepted write is pushed asynchronously to the two HRW successors, fire-and-forget, with a 5 s anti-entropy pull as backstop. The replicas exist for handoff and durability, not for read scaling.

Every stored value carries a fencing tuple (m_idx, v, origin): the membership index under which the writer owned the key, the owner's per-key write counter, and the writer's node id to break exact ties. Whoever adopts a key takes the highest tuple. A deposed owner's writes carry a smaller m_idx, so they lose by arithmetic.

That leaves one dangerous window: an old owner, partitioned away, that hasn't yet learned it was deposed. The isolated-owner rule (D-17) closes it at the owner itself. A node that can't see a current leader, or a leader not acknowledged by a quorum recently, reports is_isolated() and refuses owner-side writes and strong reads. A new owner is on the quorum side by definition, so the two can't both serve.

The rule is strict. Any node whose current leader is unknown refuses, so an ordinary leader election pauses owner-side flow operations on nodes that lost sight of the leader. In practice that pause is about 13–40 ms per election.

proxyOnce: one upstream call, cluster-wide

proxyOnce must call the real upstream once per request signature, record the answer, and replay it forever. Three nodes taking the same first hit at the same moment need an arbiter, and the ring provides one: the claim is owned like a flow, keyed by (port, signature). The owner runs a small state machine: Unclaimed → Pending (the winner gets a token and calls upstream) → Recorded, only after the recorded stub's config write commits. Pending lives only in the owner's memory, so a crash makes the signature claimable again. If no owner can serialize the claim, the request is refused with 503 and upstream is never called (D-66).

12 · Flow state on disk

R3 says flow state survives a full restart, but a quorum round per scenario transition at 20–40k RPS would be an outage. Each node instead runs a FlowShard: an embedded redb file (flow.redb, separate from the Raft store so the two fsync policies never contend) holding every key the node owns or replicates. Each key already lives on three nodes, so it lands on three disks with no new replication machinery. A single writer task per node batches mutations. Each imposter picks how hard to push:

Recovery reuses machinery that already exists. A restarted node reopens redb, drops expired entries, and serves as a replica source; adoption pulls from recovered disk exactly as from live memory. A restart is just a very long partition. Bounds keep the tier honest: a per-entry TTL (5 min default) and at most 100k flows per node, shedding whole flows by LRU, never single keys, because a half-evicted scenario would be torn state.