Campaign a2a-ironclaw-v3r18-mtls-release-v0.6.2 FAIL

{
	"agent_group": "ironclaw",
	"expected_per_reader": 20,
	"pass": false,
	"per_agent": {
		"ai:alice": {
			"recall": 0
		},
		"ai:bob": {
			"recall": 0
		},
		"ai:charlie": {
			"recall": 0
		}
	},
	"per_namespace_node4": {
		"scenario1-ai:alice": {
			"count": 0,
			"wrong_agent_id": 0
		},
		"scenario1-ai:bob": {
			"count": 0,
			"wrong_agent_id": 0
		},
		"scenario1-ai:charlie": {
			"count": 0,
			"wrong_agent_id": 0
		}
	},
	"reason": "ai:alice recalled 0 < 20 via MCP; ai:bob recalled 0 < 20 via MCP; ai:charlie recalled 0 < 20 via MCP; cross-cluster identity check failed — see per_ns",
	"reasons": [
		"ai:alice recalled 0 < 20 via MCP",
		"ai:bob recalled 0 < 20 via MCP",
		"ai:charlie recalled 0 < 20 via MCP",
		"cross-cluster identity check failed — see per_ns"
	],
	"scenario": "1",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: each agent writes 10 memories via MCP
  ai:alice on 165.227.185.137
  !! drive_agent store failed for ai:alice i=1: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=2: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=3: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=4: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=5: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=6: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=7: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=8: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=9: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:alice i=10: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  ai:bob on 165.227.112.16
  !! drive_agent store failed for ai:bob i=1: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=2: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=3: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=4: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=5: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=6: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=7: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=8: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=9: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:bob i=10: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  ai:charlie on 165.227.104.184
  !! drive_agent store failed for ai:charlie i=1: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=2: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=3: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=4: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=5: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=6: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=7: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=8: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=9: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
  !! drive_agent store failed for ai:charlie i=10: error: unrecognized subcommand 'chat'

  tip: a similar subcommand exists: 'channels'

Usage: ironclaw [OPTIONS] [COMMAND]

For more information, try '--help'.
/root/drive_agent.sh: line 140: 1: title
settle 15s for W=2/N=4 convergence
phase B: each agent counts rows in the OTHER two namespaces
  ai:alice recalled 0 rows from the other two namespaces
  ai:bob recalled 0 rows from the other two namespaces
  ai:charlie recalled 0 rows from the other two namespaces
phase C: cross-cluster identity check on node-4
  ns=scenario1-ai:alice count=0 wrong_agent_id=0
  !! expected 10 rows, got 0
  ns=scenario1-ai:bob count=0 wrong_agent_id=0
  !! expected 10 rows, got 0
  ns=scenario1-ai:charlie count=0 wrong_agent_id=0
  !! expected 10 rows, got 0

raw file

{
	"agent_group": "ironclaw",
	"expected_per_reader": 20,
	"pass": true,
	"path": "serve-http",
	"per_agent": {
		"ai:alice": {
			"recall": 20
		},
		"ai:bob": {
			"recall": 20
		},
		"ai:charlie": {
			"recall": 20
		}
	},
	"per_namespace_node4": {
		"scenario1b-ai:alice": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1b-ai:bob": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1b-ai:charlie": {
			"count": 10,
			"wrong_agent_id": 0
		}
	},
	"reasons": [],
	"scenario": "1b",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: each agent POSTs 10 memories to local serve
  ai:alice on 165.227.185.137
  ai:bob on 165.227.112.16
  ai:charlie on 165.227.104.184
settle 15s for W=2/N=4 convergence
phase B: count rows in other two namespaces via local serve HTTP
  ai:alice sees 20 rows from the other two namespaces
  ai:bob sees 20 rows from the other two namespaces
  ai:charlie sees 20 rows from the other two namespaces
phase C: cross-cluster identity check on node-4
  ns=scenario1b-ai:alice count=10 wrong_agent_id=0
  ns=scenario1b-ai:bob count=10 wrong_agent_id=0
  ns=scenario1b-ai:charlie count=10 wrong_agent_id=0

raw file

{
	"ack_uuid": "a-cde64729dfa4423fa85f8478c1e30d1f",
	"agent_group": "ironclaw",
	"handoff_uuid": "h-5b6a89a9c21444339009cb035583ddd9",
	"pass": true,
	"path": "serve-http",
	"per_agent": {
		"ai:alice": {
			"sees_ack": 1
		},
		"ai:bob": {
			"sees_handoff": 1
		}
	},
	"reasons": [],
	"scenario": "2",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: ai:alice writes handoff to ai:bob (uuid=h-5b6a89a9c21444339009cb035583ddd9)
settle 8s for quorum fanout
phase B: ai:bob reads handoff on node-2
  ai:bob sees 1 handoff memories from ai:alice
phase C: ai:bob writes acknowledgement (uuid=a-cde64729dfa4423fa85f8478c1e30d1f)
settle 8s for reverse-direction fanout
phase D: ai:alice reads ack on node-1
  ai:alice sees 1 ack memories from ai:bob

raw file

{
	"agent_group": "ironclaw",
	"expected_per_agent": 30,
	"pass": true,
	"per_agent": {
		"ai:alice": {
			"count": 30,
			"wrong_agent_id": 0
		},
		"ai:bob": {
			"count": 30,
			"wrong_agent_id": 0
		},
		"ai:charlie": {
			"count": 30,
			"wrong_agent_id": 0
		}
	},
	"reasons": [],
	"scenario": "4",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: launching concurrent 30-row bursts from 3 agents
  ai:alice burst ok=30/30
  ai:bob burst ok=30/30
  ai:charlie burst ok=30/30
settle 20s for W=2 fanout convergence
phase B: querying node-4 aggregator for per-agent counts
  ai:alice: count=30 (expected 30) wrong_agent_id=0
  ai:bob: count=30 (expected 30) wrong_agent_id=0
  ai:charlie: count=30 (expected 30) wrong_agent_id=0

raw file

{
	"agent_group": "ironclaw",
	"consolidate_http_code": 201,
	"consolidated_from_agents": [
		"ai:charlie",
		"ai:bob",
		"ai:alice"
	],
	"consolidated_id": "104c41a9-03ca-4791-9356-d184c8c34d68",
	"pass": true,
	"reasons": [],
	"scenario": "5",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: each agent writes 3 related memories
  ai:alice on 165.227.185.137
  ai:bob on 165.227.112.16
  ai:charlie on 165.227.104.184
settle 8s for quorum fanout
phase B: collect source ids on node-1, then trigger consolidate
  source ids (count=9): ['acc37d8b-46b1-475c-b057-2f8e0dad49a1', 'ac4493f5-acce-4c81-8039-88791bb7140e', '5f85e2c3-822a-4437-b3f6-7787e78dd426', 'f15cfe22-d366-4b5a-87b2-640f9ef6f82c', 'e800a189-8224-425d-a570-fa3a44abd288']...
  consolidate HTTP 201, consolidated_id=104c41a9-03ca-4791-9356-d184c8c34d68
settle 10s for consolidation fanout
phase C: verifying consolidated_from_agents on node-4
  consolidated_from_agents=['ai:charlie', 'ai:bob', 'ai:alice']

raw file

{
	"agent_group": "ironclaw",
	"alice_id": "de645926-17e7-43d3-b5d9-eafaf7276b0f",
	"bob_id": "fc00459f-d1d1-4802-a1e2-1e16ac6106a2",
	"charlie_sees_both_memories": true,
	"charlie_sees_contradicts_link": true,
	"detect_http_code": 200,
	"pass": true,
	"reasons": [],
	"scenario": "6",
	"skipped": false,
	"tls_mode": "mtls",
	"topic": "sky-color-36887d1d"
}

raw file

alice writes claim: "sky-color-36887d1d is blue" on node-1
bob writes contradicting claim: "sky-color-36887d1d is red" on node-2
  alice.id=de645926-17e7-43d3-b5d9-eafaf7276b0f bob.id=fc00459f-d1d1-4802-a1e2-1e16ac6106a2
settle 10s for quorum fanout + contradiction indexing
charlie queries /api/v1/contradictions on node-3
  HTTP 200
  sees both memories: True; sees contradicts link: True

raw file

{
	"agent_group": "ironclaw",
	"charlie_view": {
		"agent_id": "ai:alice",
		"content": "v2-21227bd308824742843990355476b7b8"
	},
	"m1_id": "0597401c-02ef-4ec1-bb18-c25ed151583b",
	"pass": true,
	"put_http_code": 200,
	"reasons": [],
	"scenario": "9",
	"skipped": false,
	"tls_mode": "mtls",
	"v1_uuid": "v1-800eb62f90cf40abb4ed3eb3c56d0511",
	"v2_uuid": "v2-21227bd308824742843990355476b7b8"
}

raw file

alice writes M1 content=v1-800eb62f90cf40abb4ed3eb3c56d0511 on node-1
  M1 id=0597401c-02ef-4ec1-bb18-c25ed151583b
settle 5s for initial replication
bob updates M1 content=v2-21227bd308824742843990355476b7b8 on node-2 via PUT
  PUT returned HTTP 200
settle 8s for update fanout
charlie reads M1 on node-3 and checks content + provenance
  charlie sees content="v2-21227bd308824742843990355476b7b8" agent_id="ai:alice"

raw file

{
	"agent_group": "ironclaw",
	"delete_http_code": 200,
	"m1_id": "b27b30e5-3410-4e16-a100-36dbdd3e4f4e",
	"pass": true,
	"post_delete_hits": {
		"node-2": 0,
		"node-3": 0,
		"node-4": 0
	},
	"post_delete_still_visible_peers": 0,
	"pre_delete_visible_peers": 3,
	"reasons": [],
	"scenario": "10",
	"skipped": false,
	"tls_mode": "mtls",
	"uuid": "d-7aa9da8b30bf4704a90c672f3b4f397b"
}

raw file

alice writes M1 content=d-7aa9da8b30bf4704a90c672f3b4f397b on node-1
  created memory id=b27b30e5-3410-4e16-a100-36dbdd3e4f4e
settle 8s for pre-delete fanout
pre-delete: verifying M1 is visible on all peers
  pre-delete node-2 sees 1
  pre-delete node-3 sees 1
  pre-delete node-4 sees 1
alice deletes M1 on node-1
  DELETE returned HTTP 200
settle 15s for tombstone propagation
post-delete: verifying M1 is GONE from all peers
  post-delete node-2 sees 0 (expected 0)
  post-delete node-3 sees 0 (expected 0)
  post-delete node-4 sees 0 (expected 0)

raw file

{
	"agent_group": "ironclaw",
	"charlie_sees_link": 1,
	"link_http_code": 201,
	"m1_id": "d480b8eb-3b72-4955-9b33-c7cb9b0f6268",
	"m2_id": "ca67e1f5-8f29-408b-b898-41e7fee2de82",
	"pass": true,
	"reasons": [],
	"relation": "related_to",
	"scenario": "11",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes M1 on node-1
bob writes M2 on node-2
  M1=d480b8eb-3b72-4955-9b33-c7cb9b0f6268 M2=ca67e1f5-8f29-408b-b898-41e7fee2de82
settle 5s for pre-link replication
alice links M1 -> M2 with relation=related_to
  link POST returned HTTP 201
settle 8s for link fanout
charlie queries links of M1 on node-3
  charlie sees M1->M2 link: 1 (expected >=1)

raw file

{
	"agent_group": "ironclaw",
	"pass": false,
	"peers_see": {
		"node_2": 0,
		"node_3": 0,
		"node_4": 0
	},
	"reason": "node-2 did not see registered agent ai:dave-probe-62956459; node-3 did not see registered agent ai:dave-probe-62956459; node-4 did not see registered agent ai:dave-probe-62956459",
	"reasons": [
		"node-2 did not see registered agent ai:dave-probe-62956459",
		"node-3 did not see registered agent ai:dave-probe-62956459",
		"node-4 did not see registered agent ai:dave-probe-62956459"
	],
	"register_http_code": 201,
	"registered_agent": "ai:dave-probe-62956459",
	"scenario": "12",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice registers new agent ai:dave-probe-62956459 on node-1
  POST /api/v1/agents returned HTTP 201
settle 10s for agent-list fanout
  node-2 sees ai:dave-probe-62956459: 0 (expected >=1)
  node-3 sees ai:dave-probe-62956459: 0 (expected >=1)
  node-4 sees ai:dave-probe-62956459: 0 (expected >=1)

raw file

{
	"agent_group": "ironclaw",
	"m1_id": "552037be-b324-4fa2-90c4-59ca59eba57b",
	"pass": true,
	"peer_view": {
		"node_1": "va-6e9167c844c3492ba6ebee1ea7c57bf2",
		"node_2": "va-6e9167c844c3492ba6ebee1ea7c57bf2",
		"node_3": "va-6e9167c844c3492ba6ebee1ea7c57bf2",
		"node_4": "va-6e9167c844c3492ba6ebee1ea7c57bf2"
	},
	"reasons": [],
	"scenario": "13",
	"skipped": false,
	"submitted": {
		"v0": "v0-d642cdb075c2432b860653dfa554bd40",
		"vA_alice": "va-6e9167c844c3492ba6ebee1ea7c57bf2",
		"vB_bob": "vb-1b370fecb12a43b290b55701ad420b94"
	},
	"tls_mode": "mtls"
}

raw file

alice writes M1 content=v0-d642cdb075c2432b860653dfa554bd40 on node-1
  M1 id=552037be-b324-4fa2-90c4-59ca59eba57b
settle 5s for initial replication
alice + bob issue concurrent PUTs (vA=va-6e9167c844c3492ba6ebee1ea7c57bf2 from alice, vB=vb-1b370fecb12a43b290b55701ad420b94 from bob)
  concurrent PUT results: [(0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'va-6e9167c844c3492ba6ebee1ea7c57bf2', 'created_at': '2026-04-22T23:36:10.738480327+00:00', 'expires_at': '2026-04-29T23:36:10.738480327+00:00', 'id': '552037be-b324-4fa2-90c4-59ca59eba57b', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-04-22T23:36:16.425722896+00:00'}, 'http_code': 200}), (0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'vb-1b370fecb12a43b290b55701ad420b94', 'created_at': '2026-04-22T23:36:10.738480327+00:00', 'expires_at': '2026-04-29T23:36:10.738480327+00:00', 'id': '552037be-b324-4fa2-90c4-59ca59eba57b', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-04-22T23:36:16.414154151+00:00'}, 'http_code': 200})]
settle 10s for quorum convergence
  node-1 sees content=va-6e9167c844c3492ba6ebee1ea7c57bf2
  node-2 sees content=va-6e9167c844c3492ba6ebee1ea7c57bf2
  node-3 sees content=va-6e9167c844c3492ba6ebee1ea7c57bf2
  node-4 sees content=va-6e9167c844c3492ba6ebee1ea7c57bf2

raw file

{
	"agent_group": "ironclaw",
	"expected_post_recovery": 20,
	"node3_saw": 20,
	"partition_target": "node-3",
	"pass": true,
	"reasons": [],
	"scenario": "14",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

suspending ai-memory on node-3 (SIGSTOP)
  !! ssh timeout (15s): root@165.227.104.184 pgrep -f 'ai-memory serve' | xargs -r kill -STOP
settle 2s for process-suspend observe
writing 10 memories each from alice + bob during node-3 outage
resuming ai-memory on node-3 (SIGCONT)
settle 20s for post-partition catchup
checking node-3 caught up
  node-3 sees 20 memories in scenario14-partition (expected 20)

raw file

{
	"agent_group": "ironclaw",
	"pass": true,
	"reasons": [],
	"scenario": "15",
	"skipped": false,
	"tls_mode": "mtls",
	"uuid": "ryw-2c5402f0e4cc430ca68fc4c756c208d0",
	"writer_sees_own_write": 1
}

raw file

alice writes + immediately reads M1 on node-1 (uuid=ryw-2c5402f0e4cc430ca68fc4c756c208d0)
  alice sees 1 (expected 1) immediately after write

raw file

{
	"agent_group": "ironclaw",
	"bob_sees_tier": "long",
	"m1_id": "7d4b480b-d794-4e78-ab6d-7defab887553",
	"pass": true,
	"promote_http_code": 200,
	"reasons": [],
	"scenario": "16",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes M1 tier=short on node-1
  M1 id=7d4b480b-d794-4e78-ab6d-7defab887553
settle 5s for pre-promote replication
alice promotes M1 to tier=long
  promote returned HTTP 200
settle 8s for promotion fanout
  bob sees tier=long (expected long)

raw file

{
	"agent_group": "ironclaw",
	"expected_count": 15,
	"pass": true,
	"per_peer": {
		"node_1": 15,
		"node_2": 15,
		"node_3": 15,
		"node_4": 15
	},
	"reasons": [],
	"scenario": "17",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

phase A: each of 3 agents writes 5 memories to scenario17-stats
  ai:alice on 165.227.185.137
  ai:bob on 165.227.112.16
  ai:charlie on 165.227.104.184
settle 15s for W=2 fanout
phase B: querying count on every peer
  node-1 count=15 (expected 15)
  node-2 count=15 (expected 15)
  node-3 count=15 (expected 15)
  node-4 count=15 (expected 15)

raw file

{
	"agent_group": "ironclaw",
	"pass": false,
	"query": "morning outdoor exercise routine",
	"reason": "semantic query did not surface alice's memory; semantic query did not surface bob's memory",
	"reasons": [
		"semantic query did not surface alice's memory",
		"semantic query did not surface bob's memory"
	],
	"scenario": "18",
	"skipped": false,
	"tls_mode": "mtls",
	"writers": [
		{
			"agent": "ai:alice",
			"marker": "alice-sunrise-238e781a",
			"seen_by_charlie": 0
		},
		{
			"agent": "ai:bob",
			"marker": "bob-daybreak-5a20a3b6",
			"seen_by_charlie": 0
		}
	]
}

raw file

alice writes A on node-1
bob writes B on node-2
settle 15s for fanout + index rebuild
charlie queries on node-3 with semantically-related prompt
  charlie sees alice's memory: 0 (expected >=1)
  charlie sees bob's memory: 0 (expected >=1)

raw file

{
	"agent_group": "ironclaw",
	"marker": "mtls-b948fdfc53b9426b96dc385b3d7d4941",
	"pass": true,
	"peers_see": {
		"node_2": 1,
		"node_3": 1
	},
	"reasons": [],
	"scenario": "20",
	"skipped": false,
	"tls_mode": "mtls",
	"write_http_code": 201
}

raw file

alice writes HTTPS + client cert on node-1
  write returned HTTP 201
settle 12s for W=2/N=4 quorum
  node-2 sees marker: 1
  node-3 sees marker: 1

raw file

{
	"agent_group": "ironclaw",
	"anonymous_probe": {
		"curl_message": "OpenSSL SSL_read: OpenSSL/3.0.13: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0",
		"http_code": "curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0\n000"
	},
	"namespace_count_after_attempt": 0,
	"pass": true,
	"reasons": [],
	"scenario": "21",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

attempting anonymous HTTPS POST to node-1 (must be rejected)
  anonymous probe result: code=curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
000 msg=OpenSSL SSL_read: OpenSSL/3.0.13: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
settle 3s for let any leak land before checking namespace
  post-probe count for namespace=scenario21: 0 (must be 0)

raw file

{
	"agent_group": "ironclaw",
	"pass": true,
	"reasons": [],
	"scenario": "22",
	"skipped": false,
	"tests": {
		"body_vs_header_conflict": {
			"acceptable": [
				"ai:body-wins",
				"ai:attacker"
			],
			"stored_agent_id": "ai:attacker"
		},
		"header_only": {
			"expected": "ai:alice",
			"stored_agent_id": "ai:alice"
		}
	},
	"tls_mode": "mtls"
}

raw file

test 1: header-only X-Agent-Id=ai:alice
settle 2s for read-settle
  stored metadata.agent_id for header-only write: ai:alice (expected ai:alice)
test 2: body.metadata.agent_id=ai:body-wins vs X-Agent-Id=ai:attacker
settle 2s for read-settle
  stored metadata.agent_id for body+header conflict: ai:attacker

raw file

raw file

payload sql: 61 bytes
payload html: 66 bytes
payload oversize: 1048576 bytes
Traceback (most recent call last):
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/scenarios/23_malicious_content_fuzz.py", line 106, in <module>
    main()
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/scenarios/23_malicious_content_fuzz.py", line 49, in main
    rc, write_doc = h.write_memory(
                    ^^^^^^^^^^^^^^^
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/a2a_harness.py", line 202, in write_memory
    return self.http_on(node_ip, "POST", "/api/v1/memories",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/a2a_harness.py", line 157, in http_on
    result = self.ssh_exec(node_ip, remote_cmd, timeout=timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/a2a_harness.py", line 103, in ssh_exec
    return self._run(cmd, timeout=timeout, stdin=stdin)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ai-memory-ai2ai-gate/ai-memory-ai2ai-gate/scripts/a2a_harness.py", line 87, in _run
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: 'ssh'

raw file

{
	"agent_group": "ironclaw",
	"byzantine_marker": "bz-f0c90f02ef4342ac8c6e6e23f8a9a14c",
	"pass": true,
	"reasons": [],
	"scenario": "24",
	"skipped": false,
	"stored_metadata_agent_id": "REJECTED_BY_SERVER",
	"sync_push_http_code": "422",
	"tls_mode": "mtls"
}

raw file

node-2 sends sync_push to node-3 claiming sender_agent_id=ai:alice
  sync_push returned HTTP 422
settle 5s for server-side sync apply
  node-3 stored metadata.agent_id=ABSENT (declared: ai:alice)
  sync_push rejected HTTP 422 — stricter-than-spec, acceptable

raw file

{
	"agent_group": "ironclaw",
	"clock_offset_seconds": 300,
	"marker": "ck-618aae99d3af4943bb09ec212520af43",
	"pass": true,
	"reasons": [],
	"scenario": "25",
	"seen_on": {
		"node_1": 1,
		"node_3": 1
	},
	"skipped": false,
	"target_node": "node-3",
	"tls_mode": "mtls"
}

raw file

shifting node-3 clock +300s (NTP disabled for the duration)
  node-3 now reports: Wed Apr 22 23:43:39 UTC 2026
alice writes on node-1 (normal clock); waiting for quorum fanout to skewed node-3
settle 15s for skewed-peer convergence
  node-3 (+300s clock) sees marker: 1 (expected >=1)
  node-1 sees marker: 1 (expected >=1)
reverting node-3 clock

raw file

{
	"agent_group": "ironclaw",
	"pass": true,
	"peer_hits": {
		"node_2": 1,
		"node_3": 1
	},
	"reasons": [],
	"scenario": "28",
	"skipped": false,
	"tls_mode": "mtls",
	"token": "kwsearch24d853d002"
}

raw file

alice writes a row containing unique token=kwsearch24d853d002
settle 8s for search index populate + fanout
bob + charlie call /api/v1/search with the exact token
  node-2 keyword search returned 1 hits
  node-3 keyword search returned 1 hits

raw file

{
	"agent_group": "ironclaw",
	"archive_http_code": 200,
	"bob_sees_archived": false,
	"m1_id": "a7734948-9c3a-49b7-91f4-4f2db949b524",
	"node4_active_rows": 0,
	"pass": false,
	"reason": "bob did not see M1 in /api/v1/archive; restore returned HTTP 404; node-4 did not see M1 restored into active set",
	"reasons": [
		"bob did not see M1 in /api/v1/archive",
		"restore returned HTTP 404",
		"node-4 did not see M1 restored into active set"
	],
	"restore_http_code": 404,
	"scenario": "29",
	"skipped": false,
	"stats_shape_ok": true,
	"tls_mode": "mtls"
}

raw file

alice writes M1 on node-1
  M1 id=a7734948-9c3a-49b7-91f4-4f2db949b524
settle 5s for pre-archive replication
alice archives M1 via DELETE /api/v1/memories/{id} (soft-delete → archive)
  archive (DELETE) returned HTTP 200
settle 5s for archive propagation
bob queries /api/v1/archive on node-2
  bob sees M1 in archive: False
charlie restores M1 via /api/v1/archive/{id}/restore on node-3
  restore returned HTTP 404
settle 5s for restore propagation
node-4 aggregator: M1 must be active again
  node-4 active rows matching marker: 0
fetch /api/v1/archive/stats on node-4

raw file

{
	"agent_group": "ironclaw",
	"pass": false,
	"peer_views": {
		"node_1": null,
		"node_2": null,
		"node_3": null,
		"node_4": null
	},
	"reason": "no peer returned a capabilities response — endpoint may not be exposed",
	"reasons": [
		"no peer returned a capabilities response — endpoint may not be exposed"
	],
	"scenario": "30",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

  node-1 capabilities: []
  node-2 capabilities: []
  node-3 capabilities: []
  node-4 capabilities: []

raw file

{
	"agent_group": "ironclaw",
	"expected_live": 2,
	"forget_http_code": 400,
	"gc_http_code": 200,
	"live_markers_per_peer": {
		"node_1": 2,
		"node_2": 2,
		"node_3": 2,
		"node_4": 2
	},
	"pass": true,
	"reasons": [],
	"scenario": "31",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes 4 memories
settle 6s for pre-gc replication
alice forgets 2 via /api/v1/forget
  forget returned HTTP 400
settle 5s for forget propagation
bob triggers /api/v1/gc on node-2
  gc returned HTTP 200
settle 8s for post-gc settle
verify remaining 2 markers are still readable on every peer
  node-1 sees 2/2 live markers
  node-2 sees 2/2 live markers
  node-3 sees 2/2 live markers
  node-4 sees 2/2 live markers

raw file

{
	"agent_group": "ironclaw",
	"bob_inbox_count": 0,
	"bob_sees_marker": false,
	"charlie_inbox_count": 0,
	"charlie_sees_marker": false,
	"marker": "inb-90aed0eb71cd4184920b08f2401f31a0",
	"notify_http_code": 404,
	"pass": false,
	"reason": "notify returned HTTP 404; bob's inbox did not deliver alice's notify",
	"reasons": [
		"notify returned HTTP 404",
		"bob's inbox did not deliver alice's notify"
	],
	"scenario": "32",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice calls /api/v1/notify → target=ai:bob
  notify returned HTTP 404
settle 6s for notification fanout
bob queries his inbox on node-2
  bob inbox has 0 messages; sees marker: False
charlie queries his inbox on node-3 (must NOT see it)
  charlie inbox has 0 messages; sees marker: False

raw file

{
	"agent_group": "ironclaw",
	"m1_delivered": 1,
	"namespace": "scenario33-pubsub-fdad6a",
	"ns_in_subs_after": false,
	"ns_in_subs_before": false,
	"pass": false,
	"reason": "subscribe returned HTTP 404; bob's subscription list did not include the subscribed namespace; unsubscribe returned HTTP 404",
	"reasons": [
		"subscribe returned HTTP 404",
		"bob's subscription list did not include the subscribed namespace",
		"unsubscribe returned HTTP 404"
	],
	"scenario": "33",
	"skipped": false,
	"subscribe_http_code": 404,
	"subscriptions_after_count": 0,
	"subscriptions_before_count": 0,
	"tls_mode": "mtls",
	"unsubscribe_http_code": 404
}

raw file

bob subscribes to namespace scenario33-pubsub-fdad6a on node-2
  subscribe returned HTTP 404
settle 2s for subscription settle
  bob subscriptions: 0 entries; contains ns: False
alice writes M1 into the subscribed namespace
settle 6s for write fanout to subscribers
  bob sees M1 in subscribed namespace: 1
bob unsubscribes from scenario33-pubsub-fdad6a
  unsubscribe returned HTTP 404
settle 2s for unsubscribe settle
  bob subscriptions after unsubscribe: ns still present = False
alice writes M2 post-unsubscribe (may still replicate via federation but subscription list excludes ns)
settle 5s for post-unsubscribe settle

raw file

{
	"agent_group": "ironclaw",
	"approve_http_code": 403,
	"charlie_sees": {
		"approved": 1,
		"rejected": 1
	},
	"namespace": "scenario34-pending-2cb992",
	"pass": false,
	"pending_queue_count": 0,
	"reason": "set-standard returned HTTP 405; approve returned HTTP 403; reject returned HTTP 404; charlie saw rejected row — reject didn't prevent publication",
	"reasons": [
		"set-standard returned HTTP 405",
		"approve returned HTTP 403",
		"reject returned HTTP 404",
		"charlie saw rejected row — reject didn't prevent publication"
	],
	"reject_http_code": 404,
	"scenario": "34",
	"set_standard_http_code": 405,
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice sets namespace standard on scenario34-pending-2cb992: write=approve, approver=ai:bob
  set-standard returned HTTP 405
settle 2s for standard settle
alice writes two memories into the governed namespace (should land in pending)
  p1=894c3a20-d4c8-44d4-b4f8-a7525f658ad9 p2=3d69ba99-c577-41e5-961e-e5d7b8998997
settle 4s for pending queue settle
bob lists pending on node-2
  pending queue has 0 entries
bob approves p1, rejects p2
  approve HTTP 403; reject HTTP 404
settle 5s for decision fanout
charlie reads the namespace — expects ONLY approved marker
  charlie sees approved=1 rejected=1

raw file

{
	"agent_group": "ironclaw",
	"child_ns": "scenario35-parent-3f00f6/child",
	"clear_http_code": 405,
	"get_standard_http_code": 200,
	"parent_ns": "scenario35-parent-3f00f6",
	"pass": false,
	"post_clear_has_child_rule": false,
	"reason": "set-parent returned HTTP 405; set-child returned HTTP 405; clear-standard returned HTTP 405; parent rule not layered into child's standard view; child rule missing from standard view",
	"reasons": [
		"set-parent returned HTTP 405",
		"set-child returned HTTP 405",
		"clear-standard returned HTTP 405",
		"parent rule not layered into child's standard view",
		"child rule missing from standard view"
	],
	"scenario": "35",
	"sees_child_rule": false,
	"sees_parent_rule": false,
	"set_child_http_code": 405,
	"set_parent_http_code": 405,
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes parent-standard-memory on node-1
alice sets namespace standard on scenario35-parent-3f00f6
  set-parent returned HTTP 405
alice writes child-standard-memory on node-1
alice sets namespace standard on scenario35-parent-3f00f6/child with parent=scenario35-parent-3f00f6
  set-child returned HTTP 405
settle 4s for standard fanout
bob gets standard for scenario35-parent-3f00f6/child on node-2 (expects layered parent+child)
  get-standard returned HTTP 200
  parent-rule visible=False; child-rule visible=False
alice clears standard on scenario35-parent-3f00f6/child
  clear returned HTTP 405
settle 3s for clear settle

raw file

{
	"agent_group": "ironclaw",
	"pass": false,
	"reason": "session_start returned HTTP 404",
	"reasons": [
		"session_start returned HTTP 404"
	],
	"scenario": "36",
	"skipped": false,
	"start_http_code": 404,
	"tls_mode": "mtls"
}

raw file

alice starts a session on node-1
  session_start returned HTTP 404, session_id=

raw file

{
	"agent_group": "ironclaw",
	"forward_has_target": true,
	"m1": "a3a11bea-c9f0-4690-80df-f5f1f76db3c4",
	"m2": "d54e0f0f-5c1d-4cc8-8d81-e5889c899c9f",
	"pass": true,
	"reasons": [],
	"reverse_has_source": true,
	"scenario": "37",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes M1 + M2 + links M1→M2
  M1=a3a11bea-c9f0-4690-80df-f5f1f76db3c4 M2=d54e0f0f-5c1d-4cc8-8d81-e5889c899c9f
settle 6s for link fanout
charlie queries /api/v1/links/M1 (forward)
charlie queries /api/v1/links/M2 (reverse)

raw file

{
	"agent_group": "ironclaw",
	"dst_ns": "scenario38-dst-fcecc5",
	"expected_rows": 5,
	"export_http_code": 200,
	"import_http_code": 200,
	"markers_preserved": 5,
	"pass": true,
	"reasons": [],
	"rows_exported": 5,
	"rows_in_destination": 5,
	"scenario": "38",
	"skipped": false,
	"src_ns": "scenario38-src-fcecc5",
	"tls_mode": "mtls"
}

raw file

alice writes 5 rows into scenario38-src-fcecc5
settle 4s for pre-export replication
alice exports on node-1 (endpoint has no namespace filter; filter client-side)
  export returned HTTP 200, total_rows=196
  rewrote 5 memories from scenario38-src-fcecc5 -> scenario38-dst-fcecc5
bob imports the payload into scenario38-dst-fcecc5 on node-2
  import returned HTTP 200
settle 6s for import + fanout
verify row counts match on destination
  scenario38-dst-fcecc5 has 5 rows (expected 5)
  markers preserved in destination: 5/5

raw file

{
	"agent_group": "ironclaw",
	"checkpoint": "2026-04-22T23:41:35+00:00",
	"expected_markers": 6,
	"markers_present": 0,
	"namespace": "scenario39-delta-c8d105",
	"pass": false,
	"reason": "delta returned 0/6 expected markers — delta-sync incomplete",
	"reasons": [
		"delta returned 0/6 expected markers — delta-sync incomplete"
	],
	"rows_returned": 0,
	"scenario": "39",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

checkpoint = 2026-04-22T23:41:35+00:00
suspending ai-memory on node-3
  !! ssh timeout (30s): root@165.227.104.184 pgrep -f 'ai-memory serve' | xargs -r kill -STOP
alice + bob write 6 rows while node-3 is out
resuming ai-memory on node-3
settle 4s for process resume
node-3 asks node-1 /api/v1/sync/since?since=2026-04-22T23:41:35+00:00
  !! ssh timeout (30s): root@165.227.104.184 curl -sS --cacert /etc/ai-memory-a2a/tls/ca.pem --cert /etc/ai-memory-a2a/tls/client.pem --key /etc/ai-memory-a2a/tls/client.key 'https://165.227.185.137:9077/api/v1/sync/since?since=2026-04-22T23%3A41%3A35%2B00%3A00&limit=500'
  /sync/since returned 0 rows in ns=scenario39-delta-c8d105; 0/6 match our markers

raw file

{
	"agent_group": "ironclaw",
	"bulk_http_code": "200",
	"bulk_size": 500,
	"namespace": "scenario40-bulk-34cd98",
	"pass": false,
	"per_peer_count": {
		"node_2": 0,
		"node_3": 0,
		"node_4": 0
	},
	"reason": "node-2 saw 0/500 bulk rows after fanout; node-3 saw 0/500 bulk rows after fanout; node-4 saw 0/500 bulk rows after fanout",
	"reasons": [
		"node-2 saw 0/500 bulk rows after fanout",
		"node-3 saw 0/500 bulk rows after fanout",
		"node-4 saw 0/500 bulk rows after fanout"
	],
	"scenario": "40",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

constructing 500-row bulk payload
staging bulk payload on node-1 /tmp, then POST /api/v1/memories/bulk
  bulk POST returned HTTP 200
settle 20s for bulk fanout across 3 peers + aggregator
  node-2 count=0 (expected 500)
  node-3 count=0 (expected 500)
  node-4 count=0 (expected 500)

raw file

{
	"activity_namespace": "scenario41-activity-af0e2a",
	"agent_group": "ironclaw",
	"pass": true,
	"per_peer": {
		"node_1": {
			"counters_t0": 8,
			"counters_t1": 8,
			"regressed_keys": 0
		},
		"node_2": {
			"counters_t0": 8,
			"counters_t1": 8,
			"regressed_keys": 0
		},
		"node_3": {
			"counters_t0": 7,
			"counters_t1": 7,
			"regressed_keys": 0
		}
	},
	"reasons": [],
	"scenario": "41",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

scrape T0
  node-1 T0 parsed 8 memory counters
  node-2 T0 parsed 8 memory counters
  node-3 T0 parsed 7 memory counters
settle 5s for counter update
scrape T1
  node-1 T1 parsed 8 memory counters
  node-2 T1 parsed 8 memory counters
  node-3 T1 parsed 7 memory counters

raw file

{
	"agent_group": "ironclaw",
	"namespaces": [
		"scenario42-689d06-0",
		"scenario42-689d06-1",
		"scenario42-689d06-2"
	],
	"pass": true,
	"per_peer": {
		"node_1": {
			"scenario42-689d06-0": 2,
			"scenario42-689d06-1": 2,
			"scenario42-689d06-2": 2
		},
		"node_2": {
			"scenario42-689d06-0": 2,
			"scenario42-689d06-1": 2,
			"scenario42-689d06-2": 2
		},
		"node_3": {
			"scenario42-689d06-0": 2,
			"scenario42-689d06-1": 2,
			"scenario42-689d06-2": 2
		},
		"node_4": {
			"scenario42-689d06-0": 2,
			"scenario42-689d06-1": 2,
			"scenario42-689d06-2": 2
		}
	},
	"reasons": [],
	"scenario": "42",
	"skipped": false,
	"tls_mode": "mtls"
}

raw file

alice writes into 3 distinct namespaces: ['scenario42-689d06-0', 'scenario42-689d06-1', 'scenario42-689d06-2']
settle 10s for namespace index fanout
  node-1 sees 3/3 target namespaces, counts: {'scenario42-689d06-0': 2, 'scenario42-689d06-1': 2, 'scenario42-689d06-2': 2}
  node-2 sees 3/3 target namespaces, counts: {'scenario42-689d06-0': 2, 'scenario42-689d06-1': 2, 'scenario42-689d06-2': 2}
  node-3 sees 3/3 target namespaces, counts: {'scenario42-689d06-0': 2, 'scenario42-689d06-1': 2, 'scenario42-689d06-2': 2}
  node-4 sees 3/3 target namespaces, counts: {'scenario42-689d06-0': 2, 'scenario42-689d06-1': 2, 'scenario42-689d06-2': 2}

raw file