Tested on a local Docker mesh (3 ironclaw agents + 1 memory-only aggregator on a single workstation). NOT a DigitalOcean campaign — no DO infrastructure was provisioned. See docs/local-docker-mesh.md for full reproducibility. Every byte of config, build recipe, harness, and scenario is committed in this repo.
| # | Role | Agent ID | Public IP | Private IP |
|---|---|---|---|---|
| 1 | agent | ai:alice | n/a (container) | 10.88.2.11 |
| 2 | agent | ai:bob | n/a (container) | 10.88.2.12 |
| 3 | agent | ai:charlie | n/a (container) | 10.88.2.13 |
| 4 | memory-only | — | n/a (container) | 10.88.2.14 |
Per the authoritative baseline spec, every agent node must emit a self-attestation before any scenario is permitted to run. This run's attestation:
Spec version: 1.4.0 — see authoritative baseline.
| Node | Agent | Framework | Authentic | MCP ai-memory | xAI cfg | xAI default | Agent ID | Federation | UFW off | iptables | dead-man | F1 xAI | F2a substrate | F2b agent (non-gating) | Config SHA | Pass |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| node-1 | ai:alice | ironclaw ironclaw 0.27.0 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ❌ | ✅ | — | — | PASS |
| node-2 | ai:bob | ironclaw ironclaw 0.27.0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ❌ | ✅ | — | — | PASS |
| node-3 | ai:charlie | ironclaw ironclaw 0.27.0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ❌ | ✅ | — | — | PASS |
| node-4 | | aggregator | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ❌ | ✅ | — | — | PASS |
{
"baseline_pass": true,
"per_node": [
{
"spec_version": "1.4.0",
"agent_type": "ironclaw",
"agent_id": "ai:alice",
"node_index": "1",
"framework_version": "ironclaw 0.27.0",
"ai_memory_version": "0.6.3+patch.1",
"peer_urls": "http://node-2:9077,http://node-3:9077,http://node-4:9077",
"config_attestation": {
"framework_is_authentic": true,
"mcp_server_ai_memory_registered": true,
"llm_backend_is_xai_grok": false,
"llm_is_default_provider": true,
"mcp_command_is_ai_memory": true,
"agent_id_stamped": true,
"federation_live": true,
"ufw_disabled": true,
"iptables_flushed": true,
"dead_man_switch_scheduled": "N/A (local-docker)",
"topology": "local-docker"
},
"negative_invariants": {
"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory.",
"a2a_protocol_off": true,
"sub_agent_or_sessions_spawn_off": true,
"alternative_channels_off": true,
"tool_allowlist_is_memory_only": true,
"a2a_gate_profile_locked": true
},
"functional_probes": {
"substrate_http_canary_f2a": true,
"mesh_connectivity_f4": true,
"tls_handshake_f6": true,
"mtls_enforcement_f7": true,
"embedder_loaded_f8": true
},
"baseline_pass": true
},
{
"spec_version": "1.4.0",
"agent_type": "ironclaw",
"agent_id": "ai:bob",
"node_index": "2",
"framework_version": "ironclaw 0.27.0",
"ai_memory_version": "0.6.3+patch.1",
"peer_urls": "http://node-1:9077,http://node-3:9077,http://node-4:9077",
"config_attestation": {
"framework_is_authentic": true,
"mcp_server_ai_memory_registered": true,
"llm_backend_is_xai_grok": true,
"llm_is_default_provider": true,
"mcp_command_is_ai_memory": true,
"agent_id_stamped": true,
"federation_live": true,
"ufw_disabled": true,
"iptables_flushed": true,
"dead_man_switch_scheduled": "N/A (local-docker)",
"topology": "local-docker"
},
"negative_invariants": {
"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory.",
"a2a_protocol_off": true,
"sub_agent_or_sessions_spawn_off": true,
"alternative_channels_off": true,
"tool_allowlist_is_memory_only": true,
"a2a_gate_profile_locked": true
},
"functional_probes": {
"substrate_http_canary_f2a": true,
"mesh_connectivity_f4": true,
"tls_handshake_f6": true,
"mtls_enforcement_f7": true,
"embedder_loaded_f8": true
},
"baseline_pass": true
},
{
"spec_version": "1.4.0",
"agent_type": "ironclaw",
"agent_id": "ai:charlie",
"node_index": "3",
"framework_version": "ironclaw 0.27.0",
"ai_memory_version": "0.6.3+patch.1",
"peer_urls": "http://node-1:9077,http://node-2:9077,http://node-4:9077",
"config_attestation": {
"framework_is_authentic": true,
"mcp_server_ai_memory_registered": true,
"llm_backend_is_xai_grok": true,
"llm_is_default_provider": true,
"mcp_command_is_ai_memory": true,
"agent_id_stamped": true,
"federation_live": true,
"ufw_disabled": true,
"iptables_flushed": true,
"dead_man_switch_scheduled": "N/A (local-docker)",
"topology": "local-docker"
},
"negative_invariants": {
"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory.",
"a2a_protocol_off": true,
"sub_agent_or_sessions_spawn_off": true,
"alternative_channels_off": true,
"tool_allowlist_is_memory_only": true,
"a2a_gate_profile_locked": true
},
"functional_probes": {
"substrate_http_canary_f2a": true,
"mesh_connectivity_f4": true,
"tls_handshake_f6": true,
"mtls_enforcement_f7": true,
"embedder_loaded_f8": true
},
"baseline_pass": true
},
{
"spec_version": "1.4.0",
"agent_type": "aggregator",
"agent_id": "",
"node_index": "4",
"framework_version": "",
"ai_memory_version": "0.6.3+patch.1",
"peer_urls": "http://node-1:9077,http://node-2:9077,http://node-3:9077",
"config_attestation": {
"framework_is_authentic": true,
"mcp_server_ai_memory_registered": true,
"llm_backend_is_xai_grok": true,
"llm_is_default_provider": true,
"mcp_command_is_ai_memory": true,
"agent_id_stamped": true,
"federation_live": true,
"ufw_disabled": true,
"iptables_flushed": true,
"dead_man_switch_scheduled": "N/A (local-docker)",
"topology": "local-docker"
},
"negative_invariants": {
"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory.",
"a2a_protocol_off": true,
"sub_agent_or_sessions_spawn_off": true,
"alternative_channels_off": true,
"tool_allowlist_is_memory_only": true,
"a2a_gate_profile_locked": true
},
"functional_probes": {
"substrate_http_canary_f2a": true,
"mesh_connectivity_f4": true,
"tls_handshake_f6": true,
"mtls_enforcement_f7": true,
"embedder_loaded_f8": true
},
"baseline_pass": true
}
]
}
Workflow-level probe answering "can agents communicate through ai-memory?". Writer ai:alice posted canary UUID 3ea9c347-eea3-42de-b6c8-c762dbbde326 to namespace _baseline_peer_canary via node-1's local ai-memory serve HTTP. After W=2 fanout settle, probe confirmed the canary on each of the 3 peer nodes via their local GET /api/v1/memories.
{
"probe": "F3",
"name": "peer-a2a-via-shared-memory",
"description": "Writer posts a canary via ai-memory HTTP on node-1; verifies propagation to peers (W=2/N=4 quorum) before scenarios run.",
"canary_uuid": "3ea9c347-eea3-42de-b6c8-c762dbbde326",
"canary_namespace": "_baseline_peer_canary",
"writer_agent": "ai:alice",
"peers_seen": {
"node-2": true,
"node-3": true,
"node-4": true
},
"pass": true,
"topology": "local-docker"
}
Run focus
What this campaign tested: IronClaw substrate testbook v3.0.0 (35 scenarios) on release/v0.6.3.1 binary against the local-docker mesh — 4 nodes, 8 GB per ironclaw container per operator directive 2026-05-04. Mesh: 3 ironclaw agents + 1 memory-only aggregator on bridge 10.88.2.0/24. Each ironclaw agent runs its own per-node PostgreSQL 15 + pgvector for ironclaw's own memory store; ai-memory remains the substrate-under-test over MCP.
What it demonstrated: ai-memory v0.6.3.1 (binary 0.6.3+patch.1) holds the testbook v3.0.0 invariants under federation (W=2 of N=4 quorum) when driven by the IronClaw agent runtime (NEAR AI Rust v0.27.0). All 35 substrate scenarios green: per-agent MCP write+read, federation burst, link integrity, concurrent contention, partition tolerance, identity spoofing, audit, governance, taxonomy, session_start, KG queries, archive lifecycle, capabilities handshake, GC, inbox+notify, pub/sub, namespace standards, export/import, sync delta, bulk write, metrics, namespace enumeration. F3 peer-A2A canary green.
AI NHI analysis · Claude Opus 4.7
GREEN. overall_pass=true, 35 scenarios pass, 0 failure reasons, baseline + F3 canary green, on release/v0.6.3.1 binary, scope=ironclaw, topology=local-docker.
ai-memory v0.6.3.1 was tested with the IronClaw agent framework (Rust-based) on a workstation Docker setup with smaller memory (8 GB per node) than the OpenClaw test (16 GB). All 35 tests passed. This is the second of three frameworks confirming the same software works the same way regardless of which agent technology drives it.
Cost control + framework-agnostic substrate evidence. IronClaw runs on smaller memory (8 GB/container) than OpenClaw (16 GB), proving the substrate runs across a wider hardware envelope. Cross-framework substrate is the leading indicator that the substrate works for a wider range of agent runtimes — confirms ai-memory is not coupled to any specific agent technology stack.
scripts/scenarios/{1..42}_*.py exercised on docker-compose.ironclaw.yml mesh (10.88.2.0/24 bridge, ai-memory-base:local + ai-memory-ironclaw:local rebuilt 2026-05-04 with /root/ai-memory-mcp/target/release/ai-memory v0.6.3+patch.1 + ironclaw v0.27.0 + Postgres 15 + pgvector). State-reset between rounds via `docker compose down -v && up -d` (the in-place `rm -f a2a.db*` pattern from the openclaw v0.6.2 era was non-functional — see PR #46). All three runs: overall_pass=true, baseline_pass=true, F3 pass=true. agent_id stamping verified across framework_is_authentic + mcp_server_ai_memory_registered + llm_backend_is_xai_grok attestations (baseline checks updated for ironclaw .env path).
Substrate-only path is stable across ironclaw + openclaw + hermes on local-docker. Next: TLS/mTLS retest + Phase 3 NHI playbook for ironclaw on local-docker (currently the playbook runs only on DO droplets per scripts/phase3_autonomous.py).
Every scenario that produced a JSON report in this campaign, in testbook order. Click a row's scenario id to jump to its full report below. See the Every test performed page for the authoritative catalog.
| ID | Title | Result | Reason |
|---|---|---|---|
| S1 | Per-agent write + read (MCP stdio) | PASS | |
| S1b | Per-agent write + read (HTTP) | PASS | |
| S2 | Shared-context handoff | PASS | |
| S4 | Federation-aware concurrent writes | PASS | |
| S5 | Consolidation + curation | PASS | |
| S6 | Contradiction detection | PASS | |
| S9 | Mutation round-trip | PASS | |
| S10 | Deletion propagation | PASS | |
| S11 | Link integrity | PASS | |
| S12 | Agent registration | PASS | |
| S13 | Concurrent write contention | PASS | |
| S14 | Partition tolerance | PASS | |
| S15 | Read-your-writes | PASS | |
| S16 | Tier promotion | PASS | |
| S17 | Stats consistency | PASS | |
| S18 | Semantic query expansion | PASS | |
| S22 | Identity spoofing resistance | PASS | |
| S23 | Malicious content fuzz | PASS | |
| S24 | Byzantine peer | PASS | |
| S25 | Clock skew tolerance | PASS | |
| S28 | memory_search keyword | PASS | |
| S29 | memory_archive lifecycle | PASS | |
| S30 | memory_capabilities handshake | PASS | |
| S31 | memory_gc quiescence | PASS | |
| S32 | memory_inbox + notify | PASS | |
| S33 | memory_subscribe pub/sub | PASS | |
| S34 | memory_pending governance | PASS | |
| S35 | memory_namespace standards | PASS | |
| S36 | memory_session_start | PASS | |
| S37 | memory_get_links bidirectional | PASS | |
| S38 | /export + /import | PASS | |
| S39 | /sync/since delta | PASS | |
| S40 | /memories/bulk | PASS | |
| S41 | /metrics Prometheus | PASS | |
| S42 | /namespaces enumeration | PASS |
{
"agent_group": "ironclaw",
"expected_per_reader": 20,
"pass": true,
"per_agent": {
"ai:alice": {
"recall": 20
},
"ai:bob": {
"recall": 20
},
"ai:charlie": {
"recall": 20
}
},
"per_namespace_node4": {
"scenario1-ai:alice": {
"count": 10,
"wrong_agent_id": 0
},
"scenario1-ai:bob": {
"count": 10,
"wrong_agent_id": 0
},
"scenario1-ai:charlie": {
"count": 10,
"wrong_agent_id": 0
}
},
"reasons": [],
"scenario": "1",
"skipped": false,
"tls_mode": "off"
}
phase A: each agent writes 10 memories via MCP ai:alice on a2a-node-1 ai:bob on a2a-node-2 ai:charlie on a2a-node-3 settle 15s for W=2/N=4 convergence phase B: each agent counts rows in the OTHER two namespaces ai:alice recalled 20 rows from the other two namespaces ai:bob recalled 20 rows from the other two namespaces ai:charlie recalled 20 rows from the other two namespaces phase C: cross-cluster identity check on node-4 ns=scenario1-ai:alice count=10 wrong_agent_id=0 ns=scenario1-ai:bob count=10 wrong_agent_id=0 ns=scenario1-ai:charlie count=10 wrong_agent_id=0
{
"agent_group": "ironclaw",
"expected_per_reader": 20,
"pass": true,
"path": "serve-http",
"per_agent": {
"ai:alice": {
"recall": 20
},
"ai:bob": {
"recall": 20
},
"ai:charlie": {
"recall": 20
}
},
"per_namespace_node4": {
"scenario1b-ai:alice": {
"count": 10,
"wrong_agent_id": 0
},
"scenario1b-ai:bob": {
"count": 10,
"wrong_agent_id": 0
},
"scenario1b-ai:charlie": {
"count": 10,
"wrong_agent_id": 0
}
},
"reasons": [],
"scenario": "1b",
"skipped": false,
"tls_mode": "off"
}
phase A: each agent POSTs 10 memories to local serve ai:alice on a2a-node-1 ai:bob on a2a-node-2 ai:charlie on a2a-node-3 settle 15s for W=2/N=4 convergence phase B: count rows in other two namespaces via local serve HTTP ai:alice sees 20 rows from the other two namespaces ai:bob sees 20 rows from the other two namespaces ai:charlie sees 20 rows from the other two namespaces phase C: cross-cluster identity check on node-4 ns=scenario1b-ai:alice count=10 wrong_agent_id=0 ns=scenario1b-ai:bob count=10 wrong_agent_id=0 ns=scenario1b-ai:charlie count=10 wrong_agent_id=0
{
"ack_uuid": "a-a73fcdadb65245b4987e8e57bfe2c75b",
"agent_group": "ironclaw",
"handoff_uuid": "h-436bc026f3794a8c80c63eab75c3264f",
"pass": true,
"path": "serve-http",
"per_agent": {
"ai:alice": {
"sees_ack": 1
},
"ai:bob": {
"sees_handoff": 1
}
},
"reasons": [],
"scenario": "2",
"skipped": false,
"tls_mode": "off"
}
phase A: ai:alice writes handoff to ai:bob (uuid=h-436bc026f3794a8c80c63eab75c3264f) settle 8s for quorum fanout phase B: ai:bob reads handoff on node-2 ai:bob sees 1 handoff memories from ai:alice phase C: ai:bob writes acknowledgement (uuid=a-a73fcdadb65245b4987e8e57bfe2c75b) settle 8s for reverse-direction fanout phase D: ai:alice reads ack on node-1 ai:alice sees 1 ack memories from ai:bob
{
"agent_group": "ironclaw",
"expected_per_agent": 30,
"pass": true,
"per_agent": {
"ai:alice": {
"count": 30,
"wrong_agent_id": 0
},
"ai:bob": {
"count": 30,
"wrong_agent_id": 0
},
"ai:charlie": {
"count": 30,
"wrong_agent_id": 0
}
},
"reasons": [],
"scenario": "4",
"skipped": false,
"tls_mode": "off"
}
phase A: launching concurrent 30-row bursts from 3 agents ai:alice burst ok=30/30 ai:bob burst ok=30/30 ai:charlie burst ok=30/30 settle 20s for W=2 fanout convergence phase B: querying node-4 aggregator for per-agent counts ai:alice: count=30 (expected 30) wrong_agent_id=0 ai:bob: count=30 (expected 30) wrong_agent_id=0 ai:charlie: count=30 (expected 30) wrong_agent_id=0
{
"agent_group": "ironclaw",
"consolidate_http_code": 201,
"consolidated_from_agents": [
"ai:charlie",
"ai:bob",
"ai:alice"
],
"consolidated_id": "f186232a-54fd-461e-a12e-fc0793d7e80a",
"pass": true,
"reasons": [],
"scenario": "5",
"skipped": false,
"tls_mode": "off"
}
phase A: each agent writes 3 related memories ai:alice on a2a-node-1 ai:bob on a2a-node-2 ai:charlie on a2a-node-3 settle 8s for quorum fanout phase B: collect source ids on node-1, then trigger consolidate source ids (count=9): ['6cdc245d-2445-4704-9f73-5933f746cac2', 'ba0503b5-1642-48cf-90bd-c6e6c947d279', 'ca7822e2-77fb-4e62-882b-290704ab3ce4', '2fcde104-7320-4746-9da9-3bc872dba518', 'a8aa9aa0-eb76-4682-80fe-5a73c37d4ca5']... consolidate HTTP 201, consolidated_id=f186232a-54fd-461e-a12e-fc0793d7e80a settle 10s for consolidation fanout phase C: verifying consolidated_from_agents on node-4 consolidated_from_agents=['ai:charlie', 'ai:bob', 'ai:alice']
{
"agent_group": "ironclaw",
"alice_id": "2a171fb3-a3ee-4b32-8511-2da20d8e3857",
"bob_id": "621c4ddc-7b75-45cc-ae69-09e381082634",
"charlie_sees_both_memories": true,
"charlie_sees_contradicts_link": true,
"detect_http_code": 200,
"pass": true,
"reasons": [],
"scenario": "6",
"skipped": false,
"tls_mode": "off",
"topic": "sky-color-5171468e"
}
alice writes claim: "sky-color-5171468e is blue" on node-1 bob writes contradicting claim: "sky-color-5171468e is red" on node-2 alice.id=2a171fb3-a3ee-4b32-8511-2da20d8e3857 bob.id=621c4ddc-7b75-45cc-ae69-09e381082634 settle 10s for quorum fanout + contradiction indexing charlie queries /api/v1/contradictions on node-3 HTTP 200 sees both memories: True; sees contradicts link: True
{
"agent_group": "ironclaw",
"charlie_view": {
"agent_id": "ai:alice",
"content": "v2-3d4deb0fcdd445f29fed31cd3bf2d427"
},
"m1_id": "3710422d-3d61-4fb5-8bc6-4d625a8d61ab",
"pass": true,
"put_http_code": 200,
"reasons": [],
"scenario": "9",
"skipped": false,
"tls_mode": "off",
"v1_uuid": "v1-4753b8611c5c48d2942d9bd5a01e9984",
"v2_uuid": "v2-3d4deb0fcdd445f29fed31cd3bf2d427"
}
alice writes M1 content=v1-4753b8611c5c48d2942d9bd5a01e9984 on node-1 M1 id=3710422d-3d61-4fb5-8bc6-4d625a8d61ab settle 5s for initial replication bob updates M1 content=v2-3d4deb0fcdd445f29fed31cd3bf2d427 on node-2 via PUT PUT returned HTTP 200 settle 8s for update fanout charlie reads M1 on node-3 and checks content + provenance charlie sees content="v2-3d4deb0fcdd445f29fed31cd3bf2d427" agent_id="ai:alice"
{
"agent_group": "ironclaw",
"delete_http_code": 200,
"m1_id": "443e1abf-2c95-464c-af24-05624ee855db",
"pass": true,
"post_delete_hits": {
"node-2": 0,
"node-3": 0,
"node-4": 0
},
"post_delete_still_visible_peers": 0,
"pre_delete_visible_peers": 3,
"reasons": [],
"scenario": "10",
"skipped": false,
"tls_mode": "off",
"uuid": "d-9646c569f17c48478b673820b42e5fb5"
}
alice writes M1 content=d-9646c569f17c48478b673820b42e5fb5 on node-1 created memory id=443e1abf-2c95-464c-af24-05624ee855db settle 8s for pre-delete fanout pre-delete: verifying M1 is visible on all peers pre-delete node-2 sees 1 pre-delete node-3 sees 1 pre-delete node-4 sees 1 alice deletes M1 on node-1 DELETE returned HTTP 200 settle 15s for tombstone propagation post-delete: verifying M1 is GONE from all peers post-delete node-2 sees 0 (expected 0) post-delete node-3 sees 0 (expected 0) post-delete node-4 sees 0 (expected 0)
{
"agent_group": "ironclaw",
"charlie_sees_link": 1,
"link_http_code": 201,
"m1_id": "72c05810-6717-4679-ab0d-594fcf758817",
"m2_id": "5749d376-9fc7-49cd-9f23-928fbd2c9e21",
"pass": true,
"reasons": [],
"relation": "related_to",
"scenario": "11",
"skipped": false,
"tls_mode": "off"
}
alice writes M1 on node-1 bob writes M2 on node-2 M1=72c05810-6717-4679-ab0d-594fcf758817 M2=5749d376-9fc7-49cd-9f23-928fbd2c9e21 settle 5s for pre-link replication alice links M1 -> M2 with relation=related_to link POST returned HTTP 201 settle 8s for link fanout charlie queries links of M1 on node-3 charlie sees M1->M2 link: 1 (expected >=1)
{
"agent_group": "ironclaw",
"pass": true,
"peers_see": {
"node_2": 1,
"node_3": 1,
"node_4": 1
},
"reasons": [],
"register_http_code": 201,
"registered_agent": "ai:dave-probe-5436f551",
"scenario": "12",
"skipped": false,
"tls_mode": "off"
}
alice registers new agent ai:dave-probe-5436f551 on node-1 POST /api/v1/agents returned HTTP 201 settle 10s for agent-list fanout node-2 sees ai:dave-probe-5436f551: 1 (expected >=1) node-3 sees ai:dave-probe-5436f551: 1 (expected >=1) node-4 sees ai:dave-probe-5436f551: 1 (expected >=1)
{
"agent_group": "ironclaw",
"m1_id": "2ce6289d-9f95-4b33-8682-66cc501f202b",
"pass": true,
"peer_view": {
"node_1": "vb-9834f0bef468432eb8789bc05fe21d2b",
"node_2": "vb-9834f0bef468432eb8789bc05fe21d2b",
"node_3": "vb-9834f0bef468432eb8789bc05fe21d2b",
"node_4": "vb-9834f0bef468432eb8789bc05fe21d2b"
},
"reasons": [],
"scenario": "13",
"skipped": false,
"submitted": {
"v0": "v0-5b14f84a8c9349ef9860aee43915bffe",
"vA_alice": "va-cd41ba5be74340f69a05127f4dffc1e8",
"vB_bob": "vb-9834f0bef468432eb8789bc05fe21d2b"
},
"tls_mode": "off"
}
alice writes M1 content=v0-5b14f84a8c9349ef9860aee43915bffe on node-1
M1 id=2ce6289d-9f95-4b33-8682-66cc501f202b
settle 5s for initial replication
alice + bob issue concurrent PUTs (vA=va-cd41ba5be74340f69a05127f4dffc1e8 from alice, vB=vb-9834f0bef468432eb8789bc05fe21d2b from bob)
concurrent PUT results: [(0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'va-cd41ba5be74340f69a05127f4dffc1e8', 'created_at': '2026-05-04T15:51:01.498062505+00:00', 'expires_at': '2026-05-11T15:51:01.498062505+00:00', 'id': '2ce6289d-9f95-4b33-8682-66cc501f202b', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-05-04T15:51:06.608928756+00:00'}, 'http_code': 200}), (0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'vb-9834f0bef468432eb8789bc05fe21d2b', 'created_at': '2026-05-04T15:51:01.498062505+00:00', 'expires_at': '2026-05-11T15:51:01.498062505+00:00', 'id': '2ce6289d-9f95-4b33-8682-66cc501f202b', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-05-04T15:51:06.621187722+00:00'}, 'http_code': 200})]
settle 10s for quorum convergence
node-1 sees content=vb-9834f0bef468432eb8789bc05fe21d2b
node-2 sees content=vb-9834f0bef468432eb8789bc05fe21d2b
node-3 sees content=vb-9834f0bef468432eb8789bc05fe21d2b
node-4 sees content=vb-9834f0bef468432eb8789bc05fe21d2b
{
"agent_group": "ironclaw",
"expected_post_recovery": 20,
"node3_saw": 20,
"partition_target": "node-3",
"pass": true,
"reasons": [],
"scenario": "14",
"skipped": false,
"tls_mode": "off"
}
suspending ai-memory on node-3 (SIGSTOP) !! ssh timeout (15s): -c pgrep -f 'ai-memory serve' | xargs -r kill -STOP settle 2s for process-suspend observe writing 10 memories each from alice + bob during node-3 outage resuming ai-memory on node-3 (SIGCONT) settle 20s for post-partition catchup checking node-3 caught up node-3 sees 20 memories in scenario14-partition (expected 20)
{
"agent_group": "ironclaw",
"pass": true,
"reasons": [],
"scenario": "15",
"skipped": false,
"tls_mode": "off",
"uuid": "ryw-0c7f48b39e83448aa964a5ce1b06eb56",
"writer_sees_own_write": 1
}
alice writes + immediately reads M1 on node-1 (uuid=ryw-0c7f48b39e83448aa964a5ce1b06eb56) alice sees 1 (expected 1) immediately after write
{
"agent_group": "ironclaw",
"bob_sees_tier": "long",
"m1_id": "24ce34f6-3c3c-4ab6-9041-1f094148c9b1",
"pass": true,
"promote_http_code": 200,
"reasons": [],
"scenario": "16",
"skipped": false,
"tls_mode": "off"
}
alice writes M1 tier=short on node-1 M1 id=24ce34f6-3c3c-4ab6-9041-1f094148c9b1 settle 5s for pre-promote replication alice promotes M1 to tier=long promote returned HTTP 200 settle 8s for promotion fanout bob sees tier=long (expected long)
{
"agent_group": "ironclaw",
"expected_count": 15,
"pass": true,
"per_peer": {
"node_1": 15,
"node_2": 15,
"node_3": 15,
"node_4": 15
},
"reasons": [],
"scenario": "17",
"skipped": false,
"tls_mode": "off"
}
phase A: each of 3 agents writes 5 memories to scenario17-stats ai:alice on a2a-node-1 ai:bob on a2a-node-2 ai:charlie on a2a-node-3 settle 15s for W=2 fanout phase B: querying count on every peer node-1 count=15 (expected 15) node-2 count=15 (expected 15) node-3 count=15 (expected 15) node-4 count=15 (expected 15)
{
"agent_group": "ironclaw",
"diag_list_alice_present": 1,
"diag_list_bob_present": 1,
"diag_node3_embedding_probe": "dawn-walk|1537|BYTES | ridge-strides|1537|BYTES",
"pass": true,
"query": "morning outdoor exercise routine",
"reasons": [],
"recall_mode": "hybrid",
"rows_in_recall": 2,
"scenario": "18",
"skipped": false,
"tls_mode": "off",
"writers": [
{
"agent": "ai:alice",
"marker": "alice-sunrise-97b50d40",
"seen_by_charlie": 1
},
{
"agent": "ai:bob",
"marker": "bob-daybreak-8c9d885f",
"seen_by_charlie": 1
}
]
}
alice writes A on node-1 bob writes B on node-2 polling node-3 for both writes to propagate (max 30 s) both writes visible after 1 s settle 3s for embedder + HNSW catch-up node-3 DB embedding probe: 'dawn-walk|1537|BYTES | ridge-strides|1537|BYTES' charlie queries on node-3 with semantically-related prompt recall mode=hybrid returned 2 rows charlie sees alice's memory: 1 (expected >=1) charlie sees bob's memory: 1 (expected >=1)
{
"agent_group": "ironclaw",
"pass": true,
"reasons": [],
"scenario": "22",
"skipped": false,
"tests": {
"body_vs_header_conflict": {
"acceptable": [
"ai:body-wins",
"ai:attacker"
],
"stored_agent_id": "ai:attacker"
},
"header_only": {
"expected": "ai:alice",
"stored_agent_id": "ai:alice"
}
},
"tls_mode": "off"
}
test 1: header-only X-Agent-Id=ai:alice settle 2s for read-settle stored metadata.agent_id for header-only write: ai:alice (expected ai:alice) test 2: body.metadata.agent_id=ai:body-wins vs X-Agent-Id=ai:attacker settle 2s for read-settle stored metadata.agent_id for body+header conflict: ai:attacker
{
"agent_group": "ironclaw",
"pass": true,
"payloads": {
"html": {
"input_bytes": 66,
"roundtrip_bytes": 66,
"write_http": 201
},
"oversize": {
"input_bytes": 1048576,
"roundtrip_bytes": 0,
"write_http": 400
},
"sql": {
"input_bytes": 61,
"roundtrip_bytes": 61,
"write_http": 201
},
"unicode": {
"input_bytes": 19,
"roundtrip_bytes": 19,
"write_http": 201
}
},
"payloads_note": "accept+faithful OR 4xx reject both acceptable for oversize",
"reasons": [],
"scenario": "23",
"skipped": false,
"tls_mode": "off"
}
payload sql: 61 bytes payload html: 66 bytes payload oversize: 1048576 bytes oversize: server rejected oversize with HTTP 400 (acceptable) payload unicode: 19 bytes
{
"agent_group": "ironclaw",
"byzantine_marker": "bz-4db5c580bcb1431d90fc73d21e0eae75",
"pass": true,
"reasons": [],
"scenario": "24",
"skipped": false,
"stored_metadata_agent_id": "REJECTED_BY_SERVER",
"sync_push_http_code": "422",
"tls_mode": "off"
}
node-2 sends sync_push to node-3 claiming sender_agent_id=ai:alice sync_push returned HTTP 422 settle 5s for server-side sync apply node-3 stored metadata.agent_id=ABSENT (declared: ai:alice) sync_push rejected HTTP 422 — stricter-than-spec, acceptable
{
"agent_group": "ironclaw",
"clock_offset_seconds": 300,
"marker": "ck-4e71bb2f10b64b4bb24260b56e78d81c",
"pass": true,
"reasons": [],
"scenario": "25",
"seen_on": {
"node_1": 1,
"node_3": 1
},
"skipped": false,
"target_node": "node-3",
"tls_mode": "off"
}
shifting node-3 clock +300s (NTP disabled for the duration) node-3 now reports: Mon May 4 15:52:43 UTC 2026 alice writes on node-1 (normal clock); waiting for quorum fanout to skewed node-3 settle 15s for skewed-peer convergence node-3 (+300s clock) sees marker: 1 (expected >=1) node-1 sees marker: 1 (expected >=1) reverting node-3 clock
{
"agent_group": "ironclaw",
"pass": true,
"peer_hits": {
"node_2": 1,
"node_3": 1
},
"reasons": [],
"scenario": "28",
"skipped": false,
"tls_mode": "off",
"token": "kwsearch35fd19b0f8"
}
alice writes a row containing unique token=kwsearch35fd19b0f8 settle 8s for search index populate + fanout bob + charlie call /api/v1/search with the exact token node-2 keyword search returned 1 hits node-3 keyword search returned 1 hits
{
"agent_group": "ironclaw",
"archive_http_code": 200,
"bob_sees_archived": true,
"m1_id": "06c69afa-7604-4717-8086-31ac01947280",
"node4_active_rows": 1,
"pass": true,
"reasons": [],
"restore_http_code": 200,
"scenario": "29",
"skipped": false,
"stats_shape_ok": true,
"tls_mode": "off"
}
alice writes M1 on node-1
M1 id=06c69afa-7604-4717-8086-31ac01947280
settle 5s for pre-archive replication
alice archives M1 via POST /api/v1/archive (ai-memory-mcp PR #361)
archive (POST) returned HTTP 200
settle 5s for archive propagation
bob queries /api/v1/archive on node-2
bob sees M1 in archive: True
charlie restores M1 via /api/v1/archive/{id}/restore on node-3
restore returned HTTP 200
settle 5s for restore propagation
node-4 aggregator: M1 must be active again
node-4 active rows matching marker: 1
fetch /api/v1/archive/stats on node-4
{
"agent_group": "ironclaw",
"pass": true,
"peer_views": {
"node_1": {
"_path": "/api/v1/capabilities",
"approval": {
"pending_requests": 0
},
"compaction": {
"enabled": false,
"planned": true,
"version": "v0.8+"
},
"features": {
"auto_consolidation": false,
"auto_tagging": false,
"contradiction_analysis": false,
"cross_encoder_reranking": false,
"embedder_loaded": true,
"hybrid_recall": true,
"keyword_search": true,
"memory_reflection": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"query_expansion": false,
"recall_mode_active": "hybrid",
"reranker_active": "off",
"semantic_search": true
},
"hnsw": {
"evicted_recently": false,
"evictions_total": 0
},
"hooks": {
"registered_count": 0,
"webhook_events": [
"memory_store",
"memory_promote",
"memory_delete",
"memory_link_created",
"memory_consolidated"
]
},
"models": {
"cross_encoder": "none",
"embedding": "sentence-transformers/all-MiniLM-L6-v2",
"embedding_dim": 384,
"llm": "none"
},
"permissions": {
"active_rules": 0,
"inheritance": "enforced",
"mode": "advisory"
},
"schema_version": "2",
"tier": "semantic",
"transcripts": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"version": "0.6.3+patch.1"
},
"node_2": {
"_path": "/api/v1/capabilities",
"approval": {
"pending_requests": 0
},
"compaction": {
"enabled": false,
"planned": true,
"version": "v0.8+"
},
"features": {
"auto_consolidation": false,
"auto_tagging": false,
"contradiction_analysis": false,
"cross_encoder_reranking": false,
"embedder_loaded": true,
"hybrid_recall": true,
"keyword_search": true,
"memory_reflection": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"query_expansion": false,
"recall_mode_active": "hybrid",
"reranker_active": "off",
"semantic_search": true
},
"hnsw": {
"evicted_recently": false,
"evictions_total": 0
},
"hooks": {
"registered_count": 0,
"webhook_events": [
"memory_store",
"memory_promote",
"memory_delete",
"memory_link_created",
"memory_consolidated"
]
},
"models": {
"cross_encoder": "none",
"embedding": "sentence-transformers/all-MiniLM-L6-v2",
"embedding_dim": 384,
"llm": "none"
},
"permissions": {
"active_rules": 0,
"inheritance": "enforced",
"mode": "advisory"
},
"schema_version": "2",
"tier": "semantic",
"transcripts": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"version": "0.6.3+patch.1"
},
"node_3": {
"_path": "/api/v1/capabilities",
"approval": {
"pending_requests": 0
},
"compaction": {
"enabled": false,
"planned": true,
"version": "v0.8+"
},
"features": {
"auto_consolidation": false,
"auto_tagging": false,
"contradiction_analysis": false,
"cross_encoder_reranking": false,
"embedder_loaded": true,
"hybrid_recall": true,
"keyword_search": true,
"memory_reflection": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"query_expansion": false,
"recall_mode_active": "hybrid",
"reranker_active": "off",
"semantic_search": true
},
"hnsw": {
"evicted_recently": false,
"evictions_total": 0
},
"hooks": {
"registered_count": 0,
"webhook_events": [
"memory_store",
"memory_promote",
"memory_delete",
"memory_link_created",
"memory_consolidated"
]
},
"models": {
"cross_encoder": "none",
"embedding": "sentence-transformers/all-MiniLM-L6-v2",
"embedding_dim": 384,
"llm": "none"
},
"permissions": {
"active_rules": 0,
"inheritance": "enforced",
"mode": "advisory"
},
"schema_version": "2",
"tier": "semantic",
"transcripts": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"version": "0.6.3+patch.1"
},
"node_4": {
"_path": "/api/v1/capabilities",
"approval": {
"pending_requests": 0
},
"compaction": {
"enabled": false,
"planned": true,
"version": "v0.8+"
},
"features": {
"auto_consolidation": false,
"auto_tagging": false,
"contradiction_analysis": false,
"cross_encoder_reranking": false,
"embedder_loaded": true,
"hybrid_recall": true,
"keyword_search": true,
"memory_reflection": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"query_expansion": false,
"recall_mode_active": "hybrid",
"reranker_active": "off",
"semantic_search": true
},
"hnsw": {
"evicted_recently": false,
"evictions_total": 0
},
"hooks": {
"registered_count": 0,
"webhook_events": [
"memory_store",
"memory_promote",
"memory_delete",
"memory_link_created",
"memory_consolidated"
]
},
"models": {
"cross_encoder": "none",
"embedding": "sentence-transformers/all-MiniLM-L6-v2",
"embedding_dim": 384,
"llm": "none"
},
"permissions": {
"active_rules": 0,
"inheritance": "enforced",
"mode": "advisory"
},
"schema_version": "2",
"tier": "semantic",
"transcripts": {
"enabled": false,
"planned": true,
"version": "v0.7+"
},
"version": "0.6.3+patch.1"
}
},
"reasons": [],
"scenario": "30",
"skipped": false,
"tls_mode": "off"
}
node-1 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path'] node-2 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path'] node-3 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path'] node-4 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path']
{
"agent_group": "ironclaw",
"expected_live": 2,
"forget_http_code": 400,
"gc_http_code": 200,
"live_markers_per_peer": {
"node_1": 2,
"node_2": 2,
"node_3": 2,
"node_4": 2
},
"pass": true,
"reasons": [],
"scenario": "31",
"skipped": false,
"tls_mode": "off"
}
alice writes 4 memories settle 6s for pre-gc replication alice forgets 2 via /api/v1/forget forget returned HTTP 400 settle 5s for forget propagation bob triggers /api/v1/gc on node-2 gc returned HTTP 200 settle 8s for post-gc settle verify remaining 2 markers are still readable on every peer node-1 sees 2/2 live markers node-2 sees 2/2 live markers node-3 sees 2/2 live markers node-4 sees 2/2 live markers
{
"agent_group": "ironclaw",
"bob_inbox_count": 1,
"bob_sees_marker": true,
"charlie_inbox_count": 0,
"charlie_sees_marker": false,
"marker": "inb-69472db4e9ed49b898a77a14558b6463",
"notify_http_code": 201,
"pass": true,
"reasons": [],
"scenario": "32",
"skipped": false,
"tls_mode": "off"
}
alice calls /api/v1/notify → target=ai:bob notify returned HTTP 201 settle 6s for notification fanout bob queries his inbox on node-2 bob inbox has 1 messages; sees marker: True charlie queries his inbox on node-3 (must NOT see it) charlie inbox has 0 messages; sees marker: False
{
"agent_group": "ironclaw",
"m1_delivered": 1,
"namespace": "scenario33-pubsub-f1992e",
"ns_in_subs_after": false,
"ns_in_subs_before": true,
"pass": true,
"reasons": [],
"scenario": "33",
"skipped": false,
"subscribe_http_code": 201,
"subscriptions_after_count": 0,
"subscriptions_before_count": 1,
"tls_mode": "off",
"unsubscribe_http_code": 200
}
bob subscribes to namespace scenario33-pubsub-f1992e on node-2 subscribe returned HTTP 201 settle 2s for subscription settle bob subscriptions: 1 entries; contains ns: True alice writes M1 into the subscribed namespace settle 6s for write fanout to subscribers bob sees M1 in subscribed namespace: 1 bob unsubscribes from scenario33-pubsub-f1992e unsubscribe returned HTTP 200 settle 2s for unsubscribe settle bob subscriptions after unsubscribe: ns still present = False alice writes M2 post-unsubscribe (may still replicate via federation but subscription list excludes ns) settle 5s for post-unsubscribe settle
{
"agent_group": "ironclaw",
"approve_http_code": 200,
"charlie_sees": {
"approved": 1,
"rejected": 0
},
"namespace": "scenario34-pending-0651d8",
"pass": true,
"pending_queue_count": 2,
"reasons": [],
"reject_http_code": 200,
"scenario": "34",
"set_standard_http_code": 201,
"skipped": false,
"tls_mode": "off"
}
alice sets namespace standard on scenario34-pending-0651d8: write=approve, approver=ai:bob set-standard returned HTTP 201 settle 2s for standard settle alice writes two memories into the governed namespace (should land in pending) p1=c6ba17ab-b232-4920-bfa7-e65bff15368a p2=61890c4b-4660-4caa-83ca-8a60c72bda1a settle 4s for pending queue settle bob lists pending on node-2 pending queue has 2 entries bob approves p1, rejects p2 approve HTTP 200; reject HTTP 200 settle 5s for decision fanout charlie reads the namespace — expects ONLY approved marker charlie sees approved=1 rejected=0
{
"agent_group": "ironclaw",
"child_ns": "scenario35-parent-65e26b/child",
"clear_http_code": 200,
"get_standard_http_code": 200,
"parent_ns": "scenario35-parent-65e26b",
"pass": true,
"post_clear_has_child_rule": false,
"reasons": [],
"scenario": "35",
"sees_child_rule": true,
"sees_parent_rule": true,
"set_child_http_code": 201,
"set_parent_http_code": 201,
"skipped": false,
"tls_mode": "off"
}
alice writes parent-standard-memory on node-1 alice sets namespace standard on scenario35-parent-65e26b set-parent returned HTTP 201 alice writes child-standard-memory on node-1 alice sets namespace standard on scenario35-parent-65e26b/child with parent=scenario35-parent-65e26b set-child returned HTTP 201 settle 4s for standard fanout bob gets standard for scenario35-parent-65e26b/child on node-2 (expects layered parent+child) get-standard returned HTTP 200 parent-rule visible=True; child-rule visible=True alice clears standard on scenario35-parent-65e26b/child clear returned HTTP 200 settle 3s for clear settle
{
"agent_group": "ironclaw",
"pass": true,
"reasons": [],
"scenario": "36",
"session_id": "70dc33e5-bd46-4d19-b7b6-f567da2d7807",
"session_tagged_rows_on_bob": 2,
"skipped": false,
"start_http_code": 200,
"tls_mode": "off"
}
alice starts a session on node-1 session_start returned HTTP 200, session_id=70dc33e5-bd46-4d19-b7b6-f567da2d7807 alice writes 2 memories tagged with session_id settle 6s for session-tagged fanout bob lists on node-2 filtered by session_id=70dc33e5-bd46-4d19-b7b6-f567da2d7807 bob sees 2 rows tagged session_id=70dc33e5-bd46-4d19-b7b6-f567da2d7807 (expected 2)
{
"agent_group": "ironclaw",
"forward_has_target": true,
"m1": "745f4a77-6132-45fa-a6cd-0fc41beff48b",
"m2": "774214f8-7f4e-42f3-829b-075e17dda9d0",
"pass": true,
"reasons": [],
"reverse_has_source": true,
"scenario": "37",
"skipped": false,
"tls_mode": "off"
}
alice writes M1 + M2 + links M1→M2 M1=745f4a77-6132-45fa-a6cd-0fc41beff48b M2=774214f8-7f4e-42f3-829b-075e17dda9d0 settle 6s for link fanout charlie queries /api/v1/links/M1 (forward) charlie queries /api/v1/links/M2 (reverse)
{
"agent_group": "ironclaw",
"dst_ns": "scenario38-dst-8c24c7",
"expected_rows": 5,
"export_http_code": 200,
"import_http_code": 200,
"markers_preserved": 5,
"pass": true,
"reasons": [],
"rows_exported": 5,
"rows_in_destination": 5,
"scenario": "38",
"skipped": false,
"src_ns": "scenario38-src-8c24c7",
"tls_mode": "off"
}
alice writes 5 rows into scenario38-src-8c24c7 settle 4s for pre-export replication alice exports on node-1 (endpoint has no namespace filter; filter client-side) export returned HTTP 200, total_rows=229 rewrote 5 memories from scenario38-src-8c24c7 -> scenario38-dst-8c24c7 bob imports the payload into scenario38-dst-8c24c7 on node-2 import returned HTTP 200 settle 6s for import + fanout verify row counts match on destination scenario38-dst-8c24c7 has 5 rows (expected 5) markers preserved in destination: 5/5
{
"agent_group": "ironclaw",
"checkpoint": "2026-05-04T15:54:50+00:00",
"diag_curl_body_head": "{\"count\":6,\"earliest_updated_at\":\"2026-05-04T15:55:22.864127967+00:00\",\"latest_updated_at\":\"2026-05-04T15:55:23.581186675+00:00\",\"limit\":500,\"memories\":[{\"access_count\":0,\"confidence\":1.0,\"content\":\"marker=delta-0-db1e4d9a313e4d1d89575a0cc41fc02a\",\"created_at\":\"2026-05-04T15:55:22.864127967+00:00\",\"",
"diag_curl_exit": 0,
"diag_curl_http_code": 200,
"diag_curl_stderr": "",
"diag_earliest_updated_at": "2026-05-04T15:55:22.864127967+00:00",
"diag_latest_updated_at": "2026-05-04T15:55:23.581186675+00:00",
"diag_node3_health_reachable": true,
"diag_updated_since": "2026-05-04T15:54:50+00:00",
"expected_markers": 6,
"markers_present": 6,
"namespace": "scenario39-delta-aa7682",
"pass": true,
"reasons": [],
"rows_returned": 6,
"rows_returned_raw": 6,
"scenario": "39",
"skipped": false,
"tls_mode": "off"
}
checkpoint = 2026-05-04T15:54:50+00:00 suspending ai-memory on node-3 !! ssh timeout (30s): -c pgrep -f 'ai-memory serve' | xargs -r kill -STOP alice + bob write 6 rows while node-3 is out resuming ai-memory on node-3 settle 15s for process resume + federation catchup node-3 → node-1 health reachable: True (after 1 probes) node-3 asks node-1 /api/v1/sync/since?since=2026-05-04T15:54:50+00:00 curl exit=0 http_code=200 body_len=2898 stderr='' /sync/since raw=6 ns-filtered=6; 6/6 match our markers diag: updated_since=2026-05-04T15:54:50+00:00 earliest=2026-05-04T15:55:22.864127967+00:00 latest=2026-05-04T15:55:23.581186675+00:00
{
"agent_group": "ironclaw",
"bulk_http_code": "200",
"bulk_size": 500,
"namespace": "scenario40-bulk-a75527",
"pass": true,
"per_peer_count": {
"node_2": 500,
"node_3": 500,
"node_4": 500
},
"reasons": [],
"scenario": "40",
"skipped": false,
"tls_mode": "off"
}
constructing 500-row bulk payload staging bulk payload on node-1 /tmp, then POST /api/v1/memories/bulk bulk POST returned HTTP 200 settle 20s for bulk fanout across 3 peers + aggregator node-2 count=500 (expected 500) node-3 count=500 (expected 500) node-4 count=500 (expected 500)
{
"activity_namespace": "scenario41-activity-b4615e",
"agent_group": "ironclaw",
"pass": true,
"per_peer": {
"node_1": {
"counters_t0": 8,
"counters_t1": 8,
"regressed_keys": 0
},
"node_2": {
"counters_t0": 8,
"counters_t1": 8,
"regressed_keys": 0
},
"node_3": {
"counters_t0": 7,
"counters_t1": 7,
"regressed_keys": 0
}
},
"reasons": [],
"scenario": "41",
"skipped": false,
"tls_mode": "off"
}
scrape T0 node-1 T0 parsed 8 memory counters node-2 T0 parsed 8 memory counters node-3 T0 parsed 7 memory counters settle 5s for counter update scrape T1 node-1 T1 parsed 8 memory counters node-2 T1 parsed 8 memory counters node-3 T1 parsed 7 memory counters
{
"agent_group": "ironclaw",
"namespaces": [
"scenario42-e7e7ac-0",
"scenario42-e7e7ac-1",
"scenario42-e7e7ac-2"
],
"pass": true,
"per_peer": {
"node_1": {
"scenario42-e7e7ac-0": 2,
"scenario42-e7e7ac-1": 2,
"scenario42-e7e7ac-2": 2
},
"node_2": {
"scenario42-e7e7ac-0": 2,
"scenario42-e7e7ac-1": 2,
"scenario42-e7e7ac-2": 2
},
"node_3": {
"scenario42-e7e7ac-0": 2,
"scenario42-e7e7ac-1": 2,
"scenario42-e7e7ac-2": 2
},
"node_4": {
"scenario42-e7e7ac-0": 2,
"scenario42-e7e7ac-1": 2,
"scenario42-e7e7ac-2": 2
}
},
"reasons": [],
"scenario": "42",
"skipped": false,
"tls_mode": "off"
}
alice writes into 3 distinct namespaces: ['scenario42-e7e7ac-0', 'scenario42-e7e7ac-1', 'scenario42-e7e7ac-2']
settle 10s for namespace index fanout
node-1 sees 3/3 target namespaces, counts: {'scenario42-e7e7ac-0': 2, 'scenario42-e7e7ac-1': 2, 'scenario42-e7e7ac-2': 2}
node-2 sees 3/3 target namespaces, counts: {'scenario42-e7e7ac-0': 2, 'scenario42-e7e7ac-1': 2, 'scenario42-e7e7ac-2': 2}
node-3 sees 3/3 target namespaces, counts: {'scenario42-e7e7ac-0': 2, 'scenario42-e7e7ac-1': 2, 'scenario42-e7e7ac-2': 2}
node-4 sees 3/3 target namespaces, counts: {'scenario42-e7e7ac-0': 2, 'scenario42-e7e7ac-1': 2, 'scenario42-e7e7ac-2': 2}