How RiftCluster works · sections 6–8
Writes, mock requests and routing
6 · Life of an admin write
Admin writes are POST /imposters, a stub edit, disable/enable, a proxy recording, a route-table change. They carry R1, R3 and R4, so the path is specified end to end. Step through the normal path first, then pick a failure from the menu.
Three steps carry the guarantees:
- Park before forward (R4). The node that received the request writes the intent to its own disk (
pending_intents) before doing anything else. From then on, whatever dies, some disk knows the client asked. A recovery loop replays parked intents to whoever is leader.
- Commit means fsynced on a majority (R3). Not “in memory on a majority”.
- The barrier (R1). Raft's commit guarantees durability, not that followers have applied the entry. Once the leader answers, the node that accepted the request polls every member's applied index (every 25 ms, for up to 2 s) and holds the
2xx until every member that reports itself Ready is at ≥ N. Each member answers with its applied index and its own /readyz verdict (D-97). A node that is joining, catching up or draining answers ready: false and is left out without being named: the balancer isn't sending it traffic, so read-your-write doesn't depend on it. Each poll is a single attempt capped at 250 ms. A member that hasn't answered once within 500 ms, such as a crashed voter that stays a member until it is retired, is named unreachable= and the write stops waiting for it. A member that answers but is still behind at the 2 s cap is named unapplied=. Each poll names the member it asks about (D-96), so a node that now holds a dead member's address cannot confirm on that member's behalf; its refusal makes that member unreachable= at once.
The op-id makes retries safe. Appending a stub twice is not idempotent by content, so every mutation carries an op_id (the client's Idempotency-Key, or one minted for it). The dedup map lives inside the replicated state machine, so it survives failover and full restart. A replayed op returns its original outcome and revision and applies nothing.
One more safety net covers the gaps. If a mock request finds no matching imposter or stub and the node's applied index trails the commit index it last heard, the node waits up to 500 ms for apply and retries the match once. It only fires on requests that were about to fail anyway, so it costs nothing on the hot path.
7 · Life of a mock request
A mock request crosses three zones with very different costs. The stateless zone is unmodified open-source engine code: prefilter indexes, predicates, templates, scripts. The cluster doesn't exist there, which is why throughput survives clustering (the standing gate is ≤ 2% regression with clustering compiled in). The owner zone costs one LAN round-trip to the flow's owner per stateful operation, unless this node is the owner. The local-append zone records the request in this node's own journal and never waits on anyone.
Which stages a request touches, and what they costpick a stub type
entryentrystatelessownerstateless*statelessownerlocal append
When the owner is gone, the read path neither hangs nor guesses. If local health tracking already marks the owner unhealthy, the operation fails at once instead of burning a 2 s timeout per request. A scenario-gated request with an unreachable owner answers 503: a test that fails loudly beats a test that passes on a stale answer. Owner calls from the engine's sync code wait on a bounded bridge, so when an owner black-holes, stateful requests shed while stateless traffic keeps full speed.
Two deliberate exceptions. An imposter can opt into readConsistency: "local" for script reads (speed over freshness, its choice). And cycling responses fall back to the node-local cursor when the cursor owner can't answer, counted in rift_cluster_sequence_fallbacks_total: blocking every cyclic response during a blip is worse than a possibly repeated index (D-10).
8 · How a request finds its imposter
Every node runs every imposter, so routing never asks which node: the load balancer settles that. It asks which imposter on this node. There are four doors, and all of them end in the same in-process call into the engine:
| Door | Looks like | Chooses the imposter by |
|---|
| Its own port | http://node:8080/orders/7 | The TCP port: the Mountebank default. Every node binds every imposter port. |
| A space on a shared port | one port, stubs tagged space: "team-a" | The port picks the imposter; the request's flow id then decides which stubs are eligible, because a stub only matches when the flow id equals its space. |
| The gateway | /__rift/8080/orders/7 | The port in the path. The request is handed to imposter 8080 as /orders/7 without touching a socket, so it works even where that port failed to bind. It answers on the admin port and on the front door, with no admin credential. |
| The front door | a --front-door listener | A route table: host, path prefix, headers and method pick a target port, optionally stripping the prefix or rewriting the host. |
The route table is control-plane state, so it behaves like any config. PUT /front-door/routes becomes one Raft op, validated before it enters the log: a table with two enabled routes that match identically at the same priority is refused as ambiguous. Applying it rewrites the node's route rows and stamps the table revision (routes@N, usable with If-Match), and each node compiles the table into an ordered list and swaps it in atomically as part of that apply. Every node therefore routes by the same table at the same log index, and no request ever sees half a table.
Try it against a small table. The figure runs the same rules as the router: enabled routes ordered by priority (higher first), then exact host before wildcard before none, then longer path prefix, then more header matches, then id; the first route whose every clause matches wins. A path prefix matches on whole segments, so /orders matches /orders/7 but not /ordersX.
A request that matches no route still gets one more chance: a path starting with /__rift/ goes through the gateway. Anything else is a 404 carrying x-rift-front-door: no-route, so a missing route is distinguishable from an imposter that answered 404 itself.
The admin port, under --cluster
The public admin address belongs to the admin front, and the upstream admin API moves behind it to loopback (D-79). The front decides each request in order. /__rift/* is mock traffic, dispatched in-process, with no credential required or forwarded (D-92). The console and the session routes are served by the front itself. Every write that changes config (imposters, stubs, routes, spaces) is terminated here and turned into a Raft op, which is how section 6 begins. Reads and everything else are proxied to the local core. One credential, --api-key, guards all of it (D-73).
Proxy stubs: when the imposter calls the real service
A stub can answer by forwarding to a real upstream (proxy.to). proxyTransparent forwards every time and records nothing. proxyAlways forwards every time and keeps adding what it sees to a recorded stub placed after it. proxyOnce calls upstream once per request signature, records the answer as a new stub placed before itself, and from then on that stub answers and upstream is never called again.
On one node that is easy. On a fleet, three nodes can take the same first request at once, and “once” has to mean once cluster-wide. The claim is owned like a flow: its key is the port plus a hash of the request signature, and its HRW owner arbitrates it (D-66). The recording is not flow state at all: it is a config change, committed through Raft as a single op that writes the stub and the “recorded” marker together (D-40). Every node applies it like any other stub edit.