A cluster that agrees enough, and never loses a write.
Several ai-memory daemons in one rack or data center replicate every HTTPS write over mutual TLS with Ed25519-signed pushes. A write commits locally first, then waits for a W-of-N quorum; a miss is answered with 202 and the row stays safe. Peers that were away catch up by pulling, erasures travel through a durable outbox, and a replicated message wakes its recipient on the receiving host.
Three peers, one signed mesh.
Each peer is a full ai-memory serve with its own store and its own wake hub. With --quorum-writes W and a --quorum-peers list, every HTTPS write fans out to every peer and is acknowledged once the local commit plus W−1 peer acknowledgements land inside --quorum-timeout-ms.
The 202 contract. On a quorum miss the local row is already durable and is never rolled back, so the daemon answers 202 Accepted with {quorum_met:false, acks, needed, reason, durability:"local"} and queues the push for redelivery, never a 5xx. Sources: src/replication.rs, src/federation/sync.rs, src/federation/quorum.rs, src/handlers/parity.rs, src/federation/signing.rs, src/federation/peer_attestation.rs.
Partitions cost latency, not data.
A peer that misses a push gets it later, twice over: the push dead-letter queue replays it, and the peer's own catch-up loop pulls everything it missed with GET /api/v1/sync/since. When the row that arrives is a message, the receiving node publishes a local wake only if the apply actually moved the row forward, so replays never wake anybody.
Sources: src/federation/push_dlq.rs, src/federation/receive.rs (catch-up loop), src/federation/applied_wake.rs, src/federation/erasure_outbox.rs, src/handlers/federation_receive.rs.
One write, one recall.
- Write on peer A. The row commits on A. A signs the canonical body (
X-Memory-Sig) and pushes it over mutual TLS to B and C. - Receive on B. B checks the client certificate fingerprint, the signature against A's enrolled Ed25519 key, the nonce, and that A may write the namespace it claims; then it applies the row and acknowledges.
- Quorum. With
--quorum-writes 2, A's commit plus B's acknowledgement meet quorum and the client gets its success response. C, if slow, gets the row from the DLQ replay or its next catch-up pull. - Recall on C. Recall is local to the peer that serves it: C answers from its own store. Immediately after a write, a recall on another peer can be a catch-up interval behind; read from the writing peer when you need the newest row.
Three peers in one rack.
# Each peer: an Ed25519 identity, and every other peer's public key enrolled
ai-memory identity generate --agent-id dc1/rack4/peer-a
ai-memory identity import --agent-id dc1/rack4/peer-b --pub peer-b.pub
ai-memory identity import --agent-id dc1/rack4/peer-c --pub peer-c.pub
# Each peer: TLS, the mTLS fingerprint allowlist, quorum fan-out and catch-up
# (a non-loopback bind also requires api_key in config.toml)
ai-memory --db /var/lib/ai-memory/ai-memory.db serve \
--host 0.0.0.0 --port 9077 \
--tls-cert /etc/ai-memory/peer-a.crt --tls-key /etc/ai-memory/peer-a.key \
--mtls-allowlist /etc/ai-memory/peer-fingerprints.allow \
--federation-identity dc1/rack4/peer-a \
--quorum-writes 2 \
--quorum-peers https://peer-b.dc1:9077,https://peer-c.dc1:9077 \
--quorum-timeout-ms 2000 \
--quorum-client-cert /etc/ai-memory/peer-a.crt --quorum-client-key /etc/ai-memory/peer-a.key \
--quorum-ca-cert /etc/ai-memory/ca.crt \
--catchup-interval-secs 30Flags verified against src/daemon_runtime.rs (ServeArgs) and src/cli/identity.rs. Peer attestation JSON, enrolment posture and rotation are in the federation guide. Hub-spoke alternative: one PostgreSQL + Apache AGE hub daemon with spokes that push through /sync/push and pull through /sync/since, as described in enterprise deployment §4.2.
Governance, wake plane and attestation at Tier 3.
Replicated decisions
Namespace standards, pending actions and pending decisions replicate through the same quorum fan-out, so an approval on one peer resolves the action everywhere (src/federation/sync.rs).
Local hubs, federated rows
Hubs never bridge hosts. A message crosses the cluster as a replicated inbox row, and applied_wake on the receiving peer turns a fresh apply into a local wake.
Three layers
The mTLS allowlist authenticates the transport, the signature and nonce authenticate the push, and peer attestation checks the claimed sender and its namespace scope on /sync/push and /sync/since.
When Tier 3 stops fitting.
| Dimension | Tier 3 behaviour | Move up when |
|---|---|---|
| Consistency | Quorum writes plus catch-up; no Raft or Paxos, no gossip discovery. Peers are listed explicitly. | Always true; size quorum for the failures you must survive. |
| Graph invalidation | Knowledge-graph invalidations reach other peers asynchronously. | Read from the writing peer when you need the result immediately. |
| Surfaces | HTTPS writes fan out. MCP-over-stdio writes join the quorum only when mcp_federation_forward_url points the MCP server at the local daemon; writes that skip the quorum (MCP without the forward URL, CLI) reach peers through the catch-up pull from /sync/since, every 30 s by default (a peer receives only rows in namespaces its attestation allowlist grants (deny by default) that are visible to its identity; scope=private rows stay on the node unless marked federation_share:true or the operator sets AI_MEMORY_FED_SYNC_TRUST_PEER=1). MCP and CLI erasures reach peers through the erasure outbox drained by the daemon. | Set the forward URL, or route agents that need replication through HTTPS. |
| Per-peer store | SQLite peers each keep one writer at a time and an in-process vector index. | Write load or corpus size grows: PostgreSQL tier. |
| Wake reach | Wakes are per host; the cross-host leg is federation, and the poll remains the guarantee. | No change needed at larger tiers. |
Reasoning for each: architectural limits and federation.