Back-filled by scripts/backfill_legacy_runs.sh — historical run predates campaign.meta.json emission.
Run focus
What this campaign tested: First end-to-end run after r5 harness fixes (Phase B via local HTTP + tail -1 guard, Phase C via SSH-hop to node-4:127.0.0.1:9077). Scenario 1 Phase A: each of ai:alice / ai:bob / ai:charlie drives 10 memory_store calls through the agent-framework-natural MCP stdio path (grok-CLI → MCP stdio → ai-memory). Phase B: each agent reads the OTHER two agents' namespaces via local ai-memory HTTP. Phase C: SSH-hop into node-4 (memory-only aggregator) and query its local ai-memory for each agent's namespace, asserting 10 rows per namespace with metadata.agent_id correctly stamped.
What it demonstrated: (1) Agent-driven MCP writes succeed end-to-end — grok-CLI accepts the natural-language prompt, selects the memory_store tool, the stdio ai-memory process writes to /var/lib/ai-memory/a2a.db, rebuilds HNSW from 1→9 entries across the session sequence. All 10 writes per agent completed without error. (2) The harness is clean — no shell arithmetic crashes, no public-IP firewall blocks, JSON emit reaches stdout correctly (and now that the workflow split stdout from stderr, scenario-1.json is pure JSON). (3) Substrate finding: the MCP stdio path and the federation serve daemon are SEPARATE processes on each node. MCP writes go through the local DB driver and exit; they never publish to the serve daemon's quorum-write coordinator, so the W=2/N=4 fanout never fires. Memory-only node-4 saw zero of the 30 writes.
AI NHI analysis · Claude Opus 4.7
INFRA GREEN / SCENARIO RED — all 30 agent-driven writes landed to LOCAL sqlite on each node; 0 rows replicated to peers. Phase B recall from other agents' namespaces returned 0; Phase C at node-4 memory-only aggregator returned 0 across all three namespaces. This is a real substrate-level interaction gap, not a harness bug — the scoreboard is honest.
Three librarians each stamped every book they received into their own local catalog — all thirty stamps landed perfectly. But the shared library union catalog never got updated because the stamping process was a temp worker who went home before signing the "send to HQ" form. Each local catalog is correct. The union catalog is empty. The test caught a real gap in how the libraries talk to each other, not a bug in the stamps.
A2A gate just surfaced a concrete interaction gap between two interfaces of our own product — the one-shot MCP stdio path doesn't coordinate with the always-on federation daemon on the same node. This is exactly the class of bug that only shows up under real agent → MCP → federation → peer exercise; it's invisible to unit tests and invisible to the ship-gate's API-only harness. Product-level fix, tracks as a v0.6.0.1 candidate in ai-memory-mcp. DO spend for this finding: under $0.30. The a2a-gate is paying for itself on its first green-infra dispatch.
drive_agent.sh store invokes `grok --no-sandbox --format json --max-tool-rounds 20 -p "Store a memory in namespace scenario1-<agent> titled w<i>-<agent>..."`. Grok binds to MCP via /root/.grok/user-settings.json which points at `ai-memory mcp` as a stdio server. The stdio server is a short-lived child: it accepts one JSON-RPC memory_store request, writes to /var/lib/ai-memory/a2a.db through the local DB driver, emits 'ai-memory: building HNSW index (N vectors)...' on next-session startup (proves the write persisted to sqlite), and exits. The sibling `ai-memory serve --quorum-writes 2 --quorum-peers <other-3>` daemon on 0.0.0.0:9077 is a DIFFERENT process with its own write-pending queue + fanout worker — it never observes the MCP-path write and therefore never quorum-writes to peers. Phase B reports 0 (local serve DB has no replicated rows for other agents' namespaces); Phase C reports 0 at node-4 (memory-only node never received any fanout). Both are correct — the federation layer is genuinely empty. Fix candidates: (a) MCP stdio tool dispatch should call the local serve daemon's HTTP API (POST /api/v1/memories) rather than writing directly to sqlite, so the fanout coordinator sees every write uniformly regardless of ingress path. (b) ai-memory serve could host MCP as an in-process tool rather than spawning a separate stdio child. Option (a) is the smaller patch.
Two parallel tracks. (1) File the substrate issue in ai-memory-mcp and target v0.6.0.1. (2) For the next a2a-gate dispatch (r7), add a scenario-1 variant or a scenario-1b that exercises the FEDERATION-AWARE write path: drive_agent.sh acquires the agent token and POSTs to http://127.0.0.1:9077/api/v1/memories on the local serve daemon. This split preserves scenario-1's MCP-native exercise (still important for agent-framework compatibility evidence) while giving us a green-path federation run in parallel — so when the substrate fix lands, scenario-1 converges on green without losing the paired framework-compatibility signal.
Every scenario that produced a JSON report in this campaign, in testbook order. Click a row's scenario id to jump to its full report below. See the Every test performed page for the authoritative catalog.
| ID | Title | Result | Reason |
|---|---|---|---|
| S1 | Per-agent write + read (MCP stdio) | ? |
Reasons: ai:alice recalled 0 < 20 via MCP | ai:bob recalled 0 < 20 via MCP | ai:charlie recalled 0 < 20 via MCP | cross-cluster identity check failed — see node-4 dump above
{
"scenario": 1,
"pass": false,
"agent_group": "openclaw",
"expected_per_reader": 20,
"per_agent": {
"ai:alice": {
"recall": 0
},
"ai:bob": {
"recall": 0
},
"ai:charlie": {
"recall": 0
}
},
"reasons": [
"ai:alice recalled 0 < 20 via MCP",
"ai:bob recalled 0 < 20 via MCP",
"ai:charlie recalled 0 < 20 via MCP",
"cross-cluster identity check failed — see node-4 dump above"
]
}
[scenario-1 openclaw] phase A: each agent writes 10 memories via MCP [scenario-1 openclaw] agent ai:alice on node-a (165.227.187.55) ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: no embeddings for HNSW index, using linear scan ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (1 vectors)... ai-memory: HNSW index ready (1 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (2 vectors)... ai-memory: HNSW index ready (2 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (3 vectors)... ai-memory: HNSW index ready (3 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (4 vectors)... ai-memory: HNSW index ready (4 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (5 vectors)... ai-memory: HNSW index ready (5 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (6 vectors)... ai-memory: HNSW index ready (6 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (7 vectors)... ai-memory: HNSW index ready (7 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (8 vectors)... ai-memory: HNSW index ready (8 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (9 vectors)... ai-memory: HNSW index ready (9 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped [scenario-1 openclaw] agent ai:bob on node-b (138.197.127.195) ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: no embeddings for HNSW index, using linear scan ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (1 vectors)... ai-memory: HNSW index ready (1 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (2 vectors)... ai-memory: HNSW index ready (2 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (3 vectors)... ai-memory: HNSW index ready (3 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (4 vectors)... ai-memory: HNSW index ready (4 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (5 vectors)... ai-memory: HNSW index ready (5 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (6 vectors)... ai-memory: HNSW index ready (6 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (7 vectors)... ai-memory: HNSW index ready (7 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (8 vectors)... ai-memory: HNSW index ready (8 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (9 vectors)... ai-memory: HNSW index ready (9 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped [scenario-1 openclaw] agent ai:charlie on node-c (167.99.62.222) ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: no embeddings for HNSW index, using linear scan ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (1 vectors)... ai-memory: HNSW index ready (1 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (2 vectors)... ai-memory: HNSW index ready (2 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (3 vectors)... ai-memory: HNSW index ready (3 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (4 vectors)... ai-memory: HNSW index ready (4 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (5 vectors)... ai-memory: HNSW index ready (5 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (6 vectors)... ai-memory: HNSW index ready (6 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (7 vectors)... ai-memory: HNSW index ready (7 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (8 vectors)... ai-memory: HNSW index ready (8 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped ai-memory: loaded config from /root/.config/ai-memory/config.toml ai-memory: requested tier = semantic ai-memory: embedder loaded (all-MiniLM-L6-v2 (384-dim, local)) ai-memory: building HNSW index (9 vectors)... ai-memory: HNSW index ready (9 entries) ai-memory MCP server started (stdio, tier=semantic) ai-memory MCP server stopped [scenario-1 openclaw] settle 15s for W=2/N=4 convergence [scenario-1 openclaw] phase B: each agent recalls the OTHER two agents' namespaces [scenario-1 openclaw] ai:alice recalled 0 rows from the other two namespaces [scenario-1 openclaw] ai:bob recalled 0 rows from the other two namespaces [scenario-1 openclaw] ai:charlie recalled 0 rows from the other two namespaces [scenario-1 openclaw] phase C: cross-cluster identity verification via node-4 (SSH hop) [scenario-1 openclaw] ns=scenario1-ai:alice count=0 wrong_agent_id=0 [scenario-1 openclaw] !! expected 10 rows, got 0 [scenario-1 openclaw] ns=scenario1-ai:bob count=0 wrong_agent_id=0 [scenario-1 openclaw] !! expected 10 rows, got 0 [scenario-1 openclaw] ns=scenario1-ai:charlie count=0 wrong_agent_id=0 [scenario-1 openclaw] !! expected 10 rows, got 0