Campaign a2a-hermes-v0.6.3.1-r12 FAIL

Agent group: hermes (homogeneous)
ai-memory ref: v0.6.3.1
Completed at: 2026-05-03T16:24:29Z
Overall pass: false
Skipped reports: 0

Infrastructure

Provider: digitalocean
Region: nyc3
Droplet size
Topology: 4-node federation mesh (W=2/N=4)
Scenarios started: 2026-05-03T16:09:36Z
Scenarios ended: 2026-05-03T16:21:31Z
Dispatched by: alphaonedev
Harness SHA: bf280d4e095d
Workflow run: https://github.com/alphaonedev/ai-memory-a2a-v0.6.3.1/actions/runs/25283818770

Node roster

#	Role	Agent ID	Public IP	Private IP
1	agent	`ai:alice`	`138.197.28.73`	`10.11.0.2`
2	agent	`ai:bob`	`45.55.164.30`	`10.11.0.3`
3	agent	`ai:charlie`	`159.89.187.128`	`10.11.0.4`
4	memory-only	`—`	`167.99.60.62`	`10.11.0.5`

Baseline attestation BASELINE OK

Per the authoritative baseline spec, every agent node must emit a self-attestation before any scenario is permitted to run. This run's attestation:

Spec version: 1.4.0 — see authoritative baseline.

Node	Agent	Framework	Authentic	MCP ai-memory	xAI cfg	xAI default	Agent ID	Federation	UFW off	iptables	dead-man	F1 xAI	F2a substrate	F2b agent (non-gating)	Config SHA	Pass
node-1	`ai:alice`	`hermes Hermes Agent v0.12.0 (2026.4.30)`	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	—	`12f99ec56116`	PASS
node-2	`ai:bob`	`hermes Hermes Agent v0.12.0 (2026.4.30)`	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	—	`889c475f0926`	PASS
node-3	`ai:charlie`	`hermes Hermes Agent v0.12.0 (2026.4.30)`	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	—	`00622f57d71b`	PASS

a2a-baseline.json

{
	"baseline_pass": true,
	"per_node": [
		{
			"spec_version": "1.4.0",
			"agent_type": "hermes",
			"agent_id": "ai:alice",
			"node_index": "1",
			"framework_version": "Hermes Agent v0.12.0 (2026.4.30)",
			"ai_memory_version": "0.6.3.1",
			"peer_urls": "http://10.11.0.3:9077,http://10.11.0.4:9077,http://10.11.0.5:9077",
			"config_file_sha256": "12f99ec56116dbd03748777fabc1697dbcc89bd41e0a1470c0dae152987998de",
			"config_attestation": {
				"framework_is_authentic": true,
				"mcp_server_ai_memory_registered": true,
				"llm_backend_is_xai_grok": true,
				"llm_is_default_provider": true,
				"mcp_command_is_ai_memory": true,
				"agent_id_stamped": true,
				"federation_live": true,
				"ufw_disabled": true,
				"iptables_flushed": true,
				"dead_man_switch_scheduled": true
			},
			"negative_invariants": {
				"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory. Any true here = thesis-preserving.",
				"a2a_protocol_off": true,
				"sub_agent_or_sessions_spawn_off": true,
				"alternative_channels_off": true,
				"tool_allowlist_is_memory_only": true,
				"a2a_gate_profile_locked": true
			},
			"functional_probes": {
				"xai_grok_chat_reachable": true,
				"xai_grok_sample_reply": "READY",
				"substrate_http_canary_f2a": true,
				"substrate_http_canary_uuid": "2f319312-8828-4097-8731-d53efe8b07b7",
				"agent_mcp_canary_f2b": false,
				"agent_mcp_canary_uuid": "8ccc92b9-3996-497d-ab05-9f02584ce46f",
				"agent_canary_response_head": "DONE ",
				"_f2b_note": "F2b is LLM-dependent and non-blocking. F2a (deterministic HTTP substrate) gates baseline_pass.",
				"hermes_peer_a2a_repro_f3b": false,
				"hermes_peer_a2a_repro_uuid": "",
				"_f3b_note": "F3b is hermes-only and observed (non-blocking). Asserts the agent-driven mcp_memory_memory_store path actually lands a row through the hermes_cli tool dispatcher, distinct from the workflow-level F3 which probes substrate-only HTTP federation. False here while F2b is true => allowlist filter regression in hermes_cli/mcp_tools.py.",
				"mesh_connectivity_f4": true,
				"mesh_edges_ok": 3,
				"mesh_edges_total": 3,
				"mesh_edges_detail": "10.11.0.3:9077:OK,10.11.0.4:9077:OK,10.11.0.5:9077:OK",
				"_f4_note": "F4 verifies this local nodes N-1 OUTBOUND mesh edges to every peer via both GET health and POST sync_push dry_run. Aggregator ANDs across N nodes to confirm full N*(N-1) bidirectional reachability. Gates baseline_pass.",
				"ai_memory_mcp_stdio_f5": true,
				"ai_memory_mcp_stdio_init_ok": true,
				"ai_memory_mcp_stdio_tools_ok": true,
				"ai_memory_mcp_stdio_tools_found": "memory_agent_list,memory_agent_register,memory_archive_list,memory_archive_purge,memory_archive_restore,memory_archive_stats,memory_auto_tag,memory_capabilities,memory_check_duplicate,memory_consolidate,memory_delete,memory_detect_contradiction,memory_entity_get_by_alias,memory_entity_register,memory_expand_query,memory_forget,memory_gc,memory_get,memory_get_links,memory_get_taxonomy,memory_inbox,memory_kg_invalidate,memory_kg_query,memory_kg_timeline,memory_link,memory_list,memory_list_subscriptions,memory_namespace_clear_standard,memory_namespace_get_standard,memory_namespace_set_standard,memory_notify,memory_pending_approve,memory_pending_list,memory_pending_reject,memory_promote,memory_recall,memory_search,memory_session_start,memory_stats,memory_store,memory_subscribe,memory_unsubscribe,memory_update",
				"_f5_note": "F5 spawns the ai-memory stdio MCP subprocess using the framework-configured invocation and verifies initialize + tools/list return memory_store, memory_recall, memory_list. Deterministic (no LLM). Gates baseline_pass.",
				"tls_mode": "off",
				"tls_handshake_f6": true,
				"tls_handshake_f6_reason": "",
				"mtls_enforcement_f7": true,
				"mtls_enforcement_f7_reason": "",
				"_f6_f7_note": "F6 verifies the TLS 1.3 handshake against the local serve + CA chain. F7 verifies mTLS enforcement — anonymous client rejected, whitelisted client accepted. Both gate baseline_pass when tls_mode != off / mtls respectively.",
				"embedder_loaded_f8": true,
				"embedder_loaded_f8_reason": "",
				"_f8_note": "F8 verifies /api/v1/capabilities reports features.embedder_loaded=true — i.e. the MiniLM embedder initialised at serve startup. Gates baseline_pass unconditionally. Without this, scenario-18 silently black-holes (semantic recall returns 0 rows).",
				"agent_mcp_ai_memory_canary": true,
				"canary_uuid": "2f319312-8828-4097-8731-d53efe8b07b7",
				"canary_namespace": "_baseline_canary_f2a"
			},
			"baseline_pass": true
		},
		{
			"spec_version": "1.4.0",
			"agent_type": "hermes",
			"agent_id": "ai:bob",
			"node_index": "2",
			"framework_version": "Hermes Agent v0.12.0 (2026.4.30)",
			"ai_memory_version": "0.6.3.1",
			"peer_urls": "http://10.11.0.2:9077,http://10.11.0.4:9077,http://10.11.0.5:9077",
			"config_file_sha256": "889c475f0926686ffdc3827accbd19cba3d820a90c5ae908eb1ac33f71eb0098",
			"config_attestation": {
				"framework_is_authentic": true,
				"mcp_server_ai_memory_registered": true,
				"llm_backend_is_xai_grok": true,
				"llm_is_default_provider": true,
				"mcp_command_is_ai_memory": true,
				"agent_id_stamped": true,
				"federation_live": true,
				"ufw_disabled": true,
				"iptables_flushed": true,
				"dead_man_switch_scheduled": true
			},
			"negative_invariants": {
				"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory. Any true here = thesis-preserving.",
				"a2a_protocol_off": true,
				"sub_agent_or_sessions_spawn_off": true,
				"alternative_channels_off": true,
				"tool_allowlist_is_memory_only": true,
				"a2a_gate_profile_locked": true
			},
			"functional_probes": {
				"xai_grok_chat_reachable": true,
				"xai_grok_sample_reply": "READY",
				"substrate_http_canary_f2a": true,
				"substrate_http_canary_uuid": "c1a94759-84a7-4b94-92ab-78bd1888e92a",
				"agent_mcp_canary_f2b": false,
				"agent_mcp_canary_uuid": "d2a2cc18-17c9-4052-8d04-26a2f2e0f27e",
				"agent_canary_response_head": "I'm sorry, but the \"ai-memory MCP mcp_memory_memory_store\" tool is not among my available tools, and I can't use or simulate tools that aren't provided in my configuration. If this is related to saving information, I can use the built-in `memory` tool instead—let me know how you'd like to proceed. ",
				"_f2b_note": "F2b is LLM-dependent and non-blocking. F2a (deterministic HTTP substrate) gates baseline_pass.",
				"hermes_peer_a2a_repro_f3b": false,
				"hermes_peer_a2a_repro_uuid": "",
				"_f3b_note": "F3b is hermes-only and observed (non-blocking). Asserts the agent-driven mcp_memory_memory_store path actually lands a row through the hermes_cli tool dispatcher, distinct from the workflow-level F3 which probes substrate-only HTTP federation. False here while F2b is true => allowlist filter regression in hermes_cli/mcp_tools.py.",
				"mesh_connectivity_f4": true,
				"mesh_edges_ok": 3,
				"mesh_edges_total": 3,
				"mesh_edges_detail": "10.11.0.2:9077:OK,10.11.0.4:9077:OK,10.11.0.5:9077:OK",
				"_f4_note": "F4 verifies this local nodes N-1 OUTBOUND mesh edges to every peer via both GET health and POST sync_push dry_run. Aggregator ANDs across N nodes to confirm full N*(N-1) bidirectional reachability. Gates baseline_pass.",
				"ai_memory_mcp_stdio_f5": true,
				"ai_memory_mcp_stdio_init_ok": true,
				"ai_memory_mcp_stdio_tools_ok": true,
				"ai_memory_mcp_stdio_tools_found": "memory_agent_list,memory_agent_register,memory_archive_list,memory_archive_purge,memory_archive_restore,memory_archive_stats,memory_auto_tag,memory_capabilities,memory_check_duplicate,memory_consolidate,memory_delete,memory_detect_contradiction,memory_entity_get_by_alias,memory_entity_register,memory_expand_query,memory_forget,memory_gc,memory_get,memory_get_links,memory_get_taxonomy,memory_inbox,memory_kg_invalidate,memory_kg_query,memory_kg_timeline,memory_link,memory_list,memory_list_subscriptions,memory_namespace_clear_standard,memory_namespace_get_standard,memory_namespace_set_standard,memory_notify,memory_pending_approve,memory_pending_list,memory_pending_reject,memory_promote,memory_recall,memory_search,memory_session_start,memory_stats,memory_store,memory_subscribe,memory_unsubscribe,memory_update",
				"_f5_note": "F5 spawns the ai-memory stdio MCP subprocess using the framework-configured invocation and verifies initialize + tools/list return memory_store, memory_recall, memory_list. Deterministic (no LLM). Gates baseline_pass.",
				"tls_mode": "off",
				"tls_handshake_f6": true,
				"tls_handshake_f6_reason": "",
				"mtls_enforcement_f7": true,
				"mtls_enforcement_f7_reason": "",
				"_f6_f7_note": "F6 verifies the TLS 1.3 handshake against the local serve + CA chain. F7 verifies mTLS enforcement — anonymous client rejected, whitelisted client accepted. Both gate baseline_pass when tls_mode != off / mtls respectively.",
				"embedder_loaded_f8": true,
				"embedder_loaded_f8_reason": "",
				"_f8_note": "F8 verifies /api/v1/capabilities reports features.embedder_loaded=true — i.e. the MiniLM embedder initialised at serve startup. Gates baseline_pass unconditionally. Without this, scenario-18 silently black-holes (semantic recall returns 0 rows).",
				"agent_mcp_ai_memory_canary": true,
				"canary_uuid": "c1a94759-84a7-4b94-92ab-78bd1888e92a",
				"canary_namespace": "_baseline_canary_f2a"
			},
			"baseline_pass": true
		},
		{
			"spec_version": "1.4.0",
			"agent_type": "hermes",
			"agent_id": "ai:charlie",
			"node_index": "3",
			"framework_version": "Hermes Agent v0.12.0 (2026.4.30)",
			"ai_memory_version": "0.6.3.1",
			"peer_urls": "http://10.11.0.2:9077,http://10.11.0.3:9077,http://10.11.0.5:9077",
			"config_file_sha256": "00622f57d71b6fd1a9018469eff1efa71f7ec334eda63f58017c4ce34744c35f",
			"config_attestation": {
				"framework_is_authentic": true,
				"mcp_server_ai_memory_registered": true,
				"llm_backend_is_xai_grok": true,
				"llm_is_default_provider": true,
				"mcp_command_is_ai_memory": true,
				"agent_id_stamped": true,
				"federation_live": true,
				"ufw_disabled": true,
				"iptables_flushed": true,
				"dead_man_switch_scheduled": true
			},
			"negative_invariants": {
				"_description": "Alternative A2A channels must be OFF so a passing scenario is only passing via ai-memory shared memory. Any true here = thesis-preserving.",
				"a2a_protocol_off": true,
				"sub_agent_or_sessions_spawn_off": true,
				"alternative_channels_off": true,
				"tool_allowlist_is_memory_only": true,
				"a2a_gate_profile_locked": true
			},
			"functional_probes": {
				"xai_grok_chat_reachable": true,
				"xai_grok_sample_reply": "READY",
				"substrate_http_canary_f2a": true,
				"substrate_http_canary_uuid": "feb649ad-c6ec-469a-a4d5-7cf8bb770158",
				"agent_mcp_canary_f2b": false,
				"agent_mcp_canary_uuid": "23cf7b66-e18d-4251-8eb0-d7c6cde3bd2f",
				"agent_canary_response_head": "DONE ",
				"_f2b_note": "F2b is LLM-dependent and non-blocking. F2a (deterministic HTTP substrate) gates baseline_pass.",
				"hermes_peer_a2a_repro_f3b": false,
				"hermes_peer_a2a_repro_uuid": "",
				"_f3b_note": "F3b is hermes-only and observed (non-blocking). Asserts the agent-driven mcp_memory_memory_store path actually lands a row through the hermes_cli tool dispatcher, distinct from the workflow-level F3 which probes substrate-only HTTP federation. False here while F2b is true => allowlist filter regression in hermes_cli/mcp_tools.py.",
				"mesh_connectivity_f4": true,
				"mesh_edges_ok": 3,
				"mesh_edges_total": 3,
				"mesh_edges_detail": "10.11.0.2:9077:OK,10.11.0.3:9077:OK,10.11.0.5:9077:OK",
				"_f4_note": "F4 verifies this local nodes N-1 OUTBOUND mesh edges to every peer via both GET health and POST sync_push dry_run. Aggregator ANDs across N nodes to confirm full N*(N-1) bidirectional reachability. Gates baseline_pass.",
				"ai_memory_mcp_stdio_f5": true,
				"ai_memory_mcp_stdio_init_ok": true,
				"ai_memory_mcp_stdio_tools_ok": true,
				"ai_memory_mcp_stdio_tools_found": "memory_agent_list,memory_agent_register,memory_archive_list,memory_archive_purge,memory_archive_restore,memory_archive_stats,memory_auto_tag,memory_capabilities,memory_check_duplicate,memory_consolidate,memory_delete,memory_detect_contradiction,memory_entity_get_by_alias,memory_entity_register,memory_expand_query,memory_forget,memory_gc,memory_get,memory_get_links,memory_get_taxonomy,memory_inbox,memory_kg_invalidate,memory_kg_query,memory_kg_timeline,memory_link,memory_list,memory_list_subscriptions,memory_namespace_clear_standard,memory_namespace_get_standard,memory_namespace_set_standard,memory_notify,memory_pending_approve,memory_pending_list,memory_pending_reject,memory_promote,memory_recall,memory_search,memory_session_start,memory_stats,memory_store,memory_subscribe,memory_unsubscribe,memory_update",
				"_f5_note": "F5 spawns the ai-memory stdio MCP subprocess using the framework-configured invocation and verifies initialize + tools/list return memory_store, memory_recall, memory_list. Deterministic (no LLM). Gates baseline_pass.",
				"tls_mode": "off",
				"tls_handshake_f6": true,
				"tls_handshake_f6_reason": "",
				"mtls_enforcement_f7": true,
				"mtls_enforcement_f7_reason": "",
				"_f6_f7_note": "F6 verifies the TLS 1.3 handshake against the local serve + CA chain. F7 verifies mTLS enforcement — anonymous client rejected, whitelisted client accepted. Both gate baseline_pass when tls_mode != off / mtls respectively.",
				"embedder_loaded_f8": true,
				"embedder_loaded_f8_reason": "",
				"_f8_note": "F8 verifies /api/v1/capabilities reports features.embedder_loaded=true — i.e. the MiniLM embedder initialised at serve startup. Gates baseline_pass unconditionally. Without this, scenario-18 silently black-holes (semantic recall returns 0 rows).",
				"agent_mcp_ai_memory_canary": true,
				"canary_uuid": "feb649ad-c6ec-469a-a4d5-7cf8bb770158",
				"canary_namespace": "_baseline_canary_f2a"
			},
			"baseline_pass": true
		}
	]
}

raw file

F3 — peer A2A via shared memory F3 OK

Workflow-level probe answering "can agents communicate through ai-memory?". Writer ai:alice posted canary UUID 48886a77-7b8d-4f2f-9745-b0b610753a6e to namespace _baseline_peer_canary via node-1's local ai-memory serve HTTP. After W=2 fanout settle, probe confirmed the canary on each of the 3 peer nodes via their local GET /api/v1/memories.

f3-peer-a2a.json

{
	"probe": "F3",
	"name": "peer-a2a-via-shared-memory",
	"description": "Writer agent posts a canary via local ai-memory HTTP on node-1; verifies the row propagates to the 3 peer nodes (W=2/N=4 quorum) before scenarios run.",
	"canary_uuid": "48886a77-7b8d-4f2f-9745-b0b610753a6e",
	"canary_namespace": "_baseline_peer_canary",
	"writer_agent": "ai:alice",
	"pass": true
}

raw file

Run focus

35/35 scenarios passed; 0 failed, 0 skipped

What this campaign tested: Campaign a2a-hermes-v0.6.3.1-r12 ran 35 testbook scenarios across transport, primitives, and cross-cutting axes.

What it demonstrated: Direct counts: pass=35, fail=0, skip=0.

AI NHI analysis · Claude Opus 4.7

35/35 scenarios passed; 0 failed, 0 skipped

PASS — auto-generated (LLM unavailable: xAI HTTP 429: {"code":"Some resource has been exhausted","error":"Your team 406ee526-41d2-4b73-bf58-d35c12860ed4 has eit)

For three audiences

Non-technical end users

This run exercised 35 tests of AI-agent-to-AI-agent communication through ai-memory. 35 worked correctly, 0 did not, and 0 were intentionally skipped because prerequisites weren't met.

C-level decision makers

Run verdict pass. Detailed narrative synthesis unavailable (LLM call failed: xAI HTTP 429: {"code":"Some resource has been exhausted","error":"Your team 406ee526-41d2-4b73-bf58-d35c12860ed4 has eit). Counts are reliable; consult the per-scenario PASS/FAIL and the testbook for primitive-level mapping.

Engineers & architects

Scenario outcomes: pass=35 fail=0 skip=0 of 35 total. First failure reasons are persisted on each scenario-N.json. LLM narrative unavailable: xAI HTTP 429: {"code":"Some resource has been exhausted","error":"Your team 406ee526-41d2-4b73-bf58-d35c12860ed4 has eit.

What changes going into the next campaign

Investigate the LLM-call failure or re-run with XAI_API_KEY verified; counts are unaffected.

Tests performed in this run

Every scenario that produced a JSON report in this campaign, in testbook order. Click a row's scenario id to jump to its full report below. See the Every test performed page for the authoritative catalog.

ID	Title	Result
S1	Per-agent write + read (MCP stdio)	PASS
S1b	Per-agent write + read (HTTP)	PASS
S2	Shared-context handoff	PASS
S4	Federation-aware concurrent writes	PASS
S5	Consolidation + curation	PASS
S6	Contradiction detection	PASS
S9	Mutation round-trip	PASS
S10	Deletion propagation	PASS
S11	Link integrity	PASS
S12	Agent registration	PASS
S13	Concurrent write contention	PASS
S14	Partition tolerance	PASS
S15	Read-your-writes	PASS
S16	Tier promotion	PASS
S17	Stats consistency	PASS
S18	Semantic query expansion	PASS
S22	Identity spoofing resistance	PASS
S23	Malicious content fuzz	PASS
S24	Byzantine peer	PASS
S25	Clock skew tolerance	PASS
S28	memory_search keyword	PASS
S29	memory_archive lifecycle	PASS
S30	memory_capabilities handshake	PASS
S31	memory_gc quiescence	PASS
S32	memory_inbox + notify	PASS
S33	memory_subscribe pub/sub	PASS
S34	memory_pending governance	PASS
S35	memory_namespace standards	PASS
S36	memory_session_start	PASS
S37	memory_get_links bidirectional	PASS
S38	/export + /import	PASS
S39	/sync/since delta	PASS
S40	/memories/bulk	PASS
S41	/metrics Prometheus	PASS
S42	/namespaces enumeration	PASS

Scenario 1 — Per-agent write + read (MCP stdio) PASS

scenario-1.json (report)

{
	"agent_group": "hermes",
	"expected_per_reader": 20,
	"pass": true,
	"per_agent": {
		"ai:alice": {
			"recall": 20
		},
		"ai:bob": {
			"recall": 20
		},
		"ai:charlie": {
			"recall": 20
		}
	},
	"per_namespace_node4": {
		"scenario1-ai:alice": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1-ai:bob": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1-ai:charlie": {
			"count": 10,
			"wrong_agent_id": 0
		}
	},
	"reasons": [],
	"scenario": "1",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-1.log (console trace)

phase A: each agent writes 10 memories via MCP
  ai:alice on 138.197.28.73
  ai:bob on 45.55.164.30
  ai:charlie on 159.89.187.128
settle 15s for W=2/N=4 convergence
phase B: each agent counts rows in the OTHER two namespaces
  ai:alice recalled 20 rows from the other two namespaces
  ai:bob recalled 20 rows from the other two namespaces
  ai:charlie recalled 20 rows from the other two namespaces
phase C: cross-cluster identity check on node-4
  ns=scenario1-ai:alice count=10 wrong_agent_id=0
  ns=scenario1-ai:bob count=10 wrong_agent_id=0
  ns=scenario1-ai:charlie count=10 wrong_agent_id=0

raw file

Scenario 1b — Per-agent write + read (HTTP) PASS

scenario-1b.json (report)

{
	"agent_group": "hermes",
	"expected_per_reader": 20,
	"pass": true,
	"path": "serve-http",
	"per_agent": {
		"ai:alice": {
			"recall": 20
		},
		"ai:bob": {
			"recall": 20
		},
		"ai:charlie": {
			"recall": 20
		}
	},
	"per_namespace_node4": {
		"scenario1b-ai:alice": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1b-ai:bob": {
			"count": 10,
			"wrong_agent_id": 0
		},
		"scenario1b-ai:charlie": {
			"count": 10,
			"wrong_agent_id": 0
		}
	},
	"reasons": [],
	"scenario": "1b",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-1b.log (console trace)

phase A: each agent POSTs 10 memories to local serve
  ai:alice on 138.197.28.73
  ai:bob on 45.55.164.30
  ai:charlie on 159.89.187.128
settle 15s for W=2/N=4 convergence
phase B: count rows in other two namespaces via local serve HTTP
  ai:alice sees 20 rows from the other two namespaces
  ai:bob sees 20 rows from the other two namespaces
  ai:charlie sees 20 rows from the other two namespaces
phase C: cross-cluster identity check on node-4
  ns=scenario1b-ai:alice count=10 wrong_agent_id=0
  ns=scenario1b-ai:bob count=10 wrong_agent_id=0
  ns=scenario1b-ai:charlie count=10 wrong_agent_id=0

raw file

Scenario 2 — Shared-context handoff PASS

scenario-2.json (report)

{
	"ack_uuid": "a-24118ee746734ddb9553957d12f4c28d",
	"agent_group": "hermes",
	"handoff_uuid": "h-0b3b5e1016464f578aacae0fce3de77c",
	"pass": true,
	"path": "serve-http",
	"per_agent": {
		"ai:alice": {
			"sees_ack": 1
		},
		"ai:bob": {
			"sees_handoff": 1
		}
	},
	"reasons": [],
	"scenario": "2",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-2.log (console trace)

phase A: ai:alice writes handoff to ai:bob (uuid=h-0b3b5e1016464f578aacae0fce3de77c)
settle 8s for quorum fanout
phase B: ai:bob reads handoff on node-2
  ai:bob sees 1 handoff memories from ai:alice
phase C: ai:bob writes acknowledgement (uuid=a-24118ee746734ddb9553957d12f4c28d)
settle 8s for reverse-direction fanout
phase D: ai:alice reads ack on node-1
  ai:alice sees 1 ack memories from ai:bob

raw file

Scenario 4 — Federation-aware concurrent writes PASS

scenario-4.json (report)

{
	"agent_group": "hermes",
	"expected_per_agent": 30,
	"pass": true,
	"per_agent": {
		"ai:alice": {
			"count": 30,
			"wrong_agent_id": 0
		},
		"ai:bob": {
			"count": 30,
			"wrong_agent_id": 0
		},
		"ai:charlie": {
			"count": 30,
			"wrong_agent_id": 0
		}
	},
	"reasons": [],
	"scenario": "4",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-4.log (console trace)

phase A: launching concurrent 30-row bursts from 3 agents
  ai:alice burst ok=30/30
  ai:bob burst ok=30/30
  ai:charlie burst ok=30/30
settle 20s for W=2 fanout convergence
phase B: querying node-4 aggregator for per-agent counts
  ai:alice: count=30 (expected 30) wrong_agent_id=0
  ai:bob: count=30 (expected 30) wrong_agent_id=0
  ai:charlie: count=30 (expected 30) wrong_agent_id=0

raw file

Scenario 5 — Consolidation + curation PASS

scenario-5.json (report)

{
	"agent_group": "hermes",
	"consolidate_http_code": 201,
	"consolidated_from_agents": [
		"ai:charlie",
		"ai:bob",
		"ai:alice"
	],
	"consolidated_id": "558486bd-676a-4501-a268-66739e396e94",
	"pass": true,
	"reasons": [],
	"scenario": "5",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-5.log (console trace)

phase A: each agent writes 3 related memories
  ai:alice on 138.197.28.73
  ai:bob on 45.55.164.30
  ai:charlie on 159.89.187.128
settle 8s for quorum fanout
phase B: collect source ids on node-1, then trigger consolidate
  source ids (count=9): ['1fa75f99-2bf9-40fb-8123-82fa3297ae6c', 'f6f64c93-c220-4b44-b785-d122a37a964a', '1e9c6630-4e7b-4d4a-b50f-86c846ef84f9', '87a819cb-c8b2-493c-a229-3b51fef20a3a', '8ea2fdc7-53c5-41df-b680-0add80800896']...
  consolidate HTTP 201, consolidated_id=558486bd-676a-4501-a268-66739e396e94
settle 10s for consolidation fanout
phase C: verifying consolidated_from_agents on node-4
  consolidated_from_agents=['ai:charlie', 'ai:bob', 'ai:alice']

raw file

Scenario 6 — Contradiction detection PASS

scenario-6.json (report)

{
	"agent_group": "hermes",
	"alice_id": "2e0a2a13-3f6b-45f0-9059-f721e6612cd3",
	"bob_id": "7480c700-b1b5-41ba-84d3-61ac2c05db28",
	"charlie_sees_both_memories": true,
	"charlie_sees_contradicts_link": true,
	"detect_http_code": 200,
	"pass": true,
	"reasons": [],
	"scenario": "6",
	"skipped": false,
	"tls_mode": "off",
	"topic": "sky-color-05bb2d95"
}

raw file

scenario-6.log (console trace)

alice writes claim: "sky-color-05bb2d95 is blue" on node-1
bob writes contradicting claim: "sky-color-05bb2d95 is red" on node-2
  alice.id=2e0a2a13-3f6b-45f0-9059-f721e6612cd3 bob.id=7480c700-b1b5-41ba-84d3-61ac2c05db28
settle 10s for quorum fanout + contradiction indexing
charlie queries /api/v1/contradictions on node-3
  HTTP 200
  sees both memories: True; sees contradicts link: True

raw file

Scenario 9 — Mutation round-trip PASS

scenario-9.json (report)

{
	"agent_group": "hermes",
	"charlie_view": {
		"agent_id": "ai:alice",
		"content": "v2-a2e4f78b04a144659c5e495860dbc0f4"
	},
	"m1_id": "247af775-b33a-4d64-b56b-d4171f650c74",
	"pass": true,
	"put_http_code": 200,
	"reasons": [],
	"scenario": "9",
	"skipped": false,
	"tls_mode": "off",
	"v1_uuid": "v1-48a4f58125b542cdbb1d392d72dbb502",
	"v2_uuid": "v2-a2e4f78b04a144659c5e495860dbc0f4"
}

raw file

scenario-9.log (console trace)

alice writes M1 content=v1-48a4f58125b542cdbb1d392d72dbb502 on node-1
  M1 id=247af775-b33a-4d64-b56b-d4171f650c74
settle 5s for initial replication
bob updates M1 content=v2-a2e4f78b04a144659c5e495860dbc0f4 on node-2 via PUT
  PUT returned HTTP 200
settle 8s for update fanout
charlie reads M1 on node-3 and checks content + provenance
  charlie sees content="v2-a2e4f78b04a144659c5e495860dbc0f4" agent_id="ai:alice"

raw file

Scenario 10 — Deletion propagation PASS

scenario-10.json (report)

{
	"agent_group": "hermes",
	"delete_http_code": 200,
	"m1_id": "7302477e-ff8c-4a75-8105-e2eec5658935",
	"pass": true,
	"post_delete_hits": {
		"node-2": 0,
		"node-3": 0,
		"node-4": 0
	},
	"post_delete_still_visible_peers": 0,
	"pre_delete_visible_peers": 3,
	"reasons": [],
	"scenario": "10",
	"skipped": false,
	"tls_mode": "off",
	"uuid": "d-955a667983434914bf0d48bf6b07e548"
}

raw file

scenario-10.log (console trace)

alice writes M1 content=d-955a667983434914bf0d48bf6b07e548 on node-1
  created memory id=7302477e-ff8c-4a75-8105-e2eec5658935
settle 8s for pre-delete fanout
pre-delete: verifying M1 is visible on all peers
  pre-delete node-2 sees 1
  pre-delete node-3 sees 1
  pre-delete node-4 sees 1
alice deletes M1 on node-1
  DELETE returned HTTP 200
settle 15s for tombstone propagation
post-delete: verifying M1 is GONE from all peers
  post-delete node-2 sees 0 (expected 0)
  post-delete node-3 sees 0 (expected 0)
  post-delete node-4 sees 0 (expected 0)

raw file

Scenario 11 — Link integrity PASS

scenario-11.json (report)

{
	"agent_group": "hermes",
	"charlie_sees_link": 1,
	"link_http_code": 201,
	"m1_id": "044f8088-0b5e-48b2-b762-e9d40e9b4f30",
	"m2_id": "3b70af07-366f-4bbf-9ea7-16b4f1531da9",
	"pass": true,
	"reasons": [],
	"relation": "related_to",
	"scenario": "11",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-11.log (console trace)

alice writes M1 on node-1
bob writes M2 on node-2
  M1=044f8088-0b5e-48b2-b762-e9d40e9b4f30 M2=3b70af07-366f-4bbf-9ea7-16b4f1531da9
settle 5s for pre-link replication
alice links M1 -> M2 with relation=related_to
  link POST returned HTTP 201
settle 8s for link fanout
charlie queries links of M1 on node-3
  charlie sees M1->M2 link: 1 (expected >=1)

raw file

Scenario 12 — Agent registration PASS

scenario-12.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"peers_see": {
		"node_2": 1,
		"node_3": 1,
		"node_4": 1
	},
	"reasons": [],
	"register_http_code": 201,
	"registered_agent": "ai:dave-probe-cf01b887",
	"scenario": "12",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-12.log (console trace)

alice registers new agent ai:dave-probe-cf01b887 on node-1
  POST /api/v1/agents returned HTTP 201
settle 10s for agent-list fanout
  node-2 sees ai:dave-probe-cf01b887: 1 (expected >=1)
  node-3 sees ai:dave-probe-cf01b887: 1 (expected >=1)
  node-4 sees ai:dave-probe-cf01b887: 1 (expected >=1)

raw file

Scenario 13 — Concurrent write contention PASS

scenario-13.json (report)

{
	"agent_group": "hermes",
	"m1_id": "e5181520-d4d7-41ef-b2b1-df8647acd4e0",
	"pass": true,
	"peer_view": {
		"node_1": "vb-53251afb5f7f4163a47ff71dc5118706",
		"node_2": "vb-53251afb5f7f4163a47ff71dc5118706",
		"node_3": "vb-53251afb5f7f4163a47ff71dc5118706",
		"node_4": "vb-53251afb5f7f4163a47ff71dc5118706"
	},
	"reasons": [],
	"scenario": "13",
	"skipped": false,
	"submitted": {
		"v0": "v0-01c8d75afae04286b194db0ed2dc16ee",
		"vA_alice": "va-ff2143966ab4438398190122fd8cc310",
		"vB_bob": "vb-53251afb5f7f4163a47ff71dc5118706"
	},
	"tls_mode": "off"
}

raw file

scenario-13.log (console trace)

alice writes M1 content=v0-01c8d75afae04286b194db0ed2dc16ee on node-1
  M1 id=e5181520-d4d7-41ef-b2b1-df8647acd4e0
settle 5s for initial replication
alice + bob issue concurrent PUTs (vA=va-ff2143966ab4438398190122fd8cc310 from alice, vB=vb-53251afb5f7f4163a47ff71dc5118706 from bob)
  concurrent PUT results: [(0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'va-ff2143966ab4438398190122fd8cc310', 'created_at': '2026-05-03T16:14:00.091341680+00:00', 'expires_at': '2026-05-10T16:14:00.091341680+00:00', 'id': 'e5181520-d4d7-41ef-b2b1-df8647acd4e0', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-05-03T16:14:05.711040829+00:00'}, 'http_code': 200}), (0, {'body': {'access_count': 0, 'confidence': 1.0, 'content': 'vb-53251afb5f7f4163a47ff71dc5118706', 'created_at': '2026-05-03T16:14:00.091341680+00:00', 'expires_at': '2026-05-10T16:14:00.091341680+00:00', 'id': 'e5181520-d4d7-41ef-b2b1-df8647acd4e0', 'metadata': {'agent_id': 'ai:alice', 'scenario': '13'}, 'namespace': 'scenario13-contention', 'priority': 5, 'source': 'api', 'tags': [], 'tier': 'mid', 'title': 'm1', 'updated_at': '2026-05-03T16:14:05.736239941+00:00'}, 'http_code': 200})]
settle 10s for quorum convergence
  node-1 sees content=vb-53251afb5f7f4163a47ff71dc5118706
  node-2 sees content=vb-53251afb5f7f4163a47ff71dc5118706
  node-3 sees content=vb-53251afb5f7f4163a47ff71dc5118706
  node-4 sees content=vb-53251afb5f7f4163a47ff71dc5118706

raw file

Scenario 14 — Partition tolerance PASS

scenario-14.json (report)

{
	"agent_group": "hermes",
	"expected_post_recovery": 20,
	"node3_saw": 20,
	"partition_target": "node-3",
	"pass": true,
	"reasons": [],
	"scenario": "14",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-14.log (console trace)

suspending ai-memory on node-3 (SIGSTOP)
  !! ssh timeout (15s): root@159.89.187.128 pgrep -f 'ai-memory serve' | xargs -r kill -STOP
settle 2s for process-suspend observe
writing 10 memories each from alice + bob during node-3 outage
resuming ai-memory on node-3 (SIGCONT)
settle 20s for post-partition catchup
checking node-3 caught up
  node-3 sees 20 memories in scenario14-partition (expected 20)

raw file

Scenario 15 — Read-your-writes PASS

scenario-15.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"reasons": [],
	"scenario": "15",
	"skipped": false,
	"tls_mode": "off",
	"uuid": "ryw-f257586a367b4ebb9ba2426b1820264e",
	"writer_sees_own_write": 1
}

raw file

scenario-15.log (console trace)

alice writes + immediately reads M1 on node-1 (uuid=ryw-f257586a367b4ebb9ba2426b1820264e)
  alice sees 1 (expected 1) immediately after write

raw file

Scenario 16 — Tier promotion PASS

scenario-16.json (report)

{
	"agent_group": "hermes",
	"bob_sees_tier": "long",
	"m1_id": "f473089e-ce0b-4787-b142-9708ad8c3e95",
	"pass": true,
	"promote_http_code": 200,
	"reasons": [],
	"scenario": "16",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-16.log (console trace)

alice writes M1 tier=short on node-1
  M1 id=f473089e-ce0b-4787-b142-9708ad8c3e95
settle 5s for pre-promote replication
alice promotes M1 to tier=long
  promote returned HTTP 200
settle 8s for promotion fanout
  bob sees tier=long (expected long)

raw file

Scenario 17 — Stats consistency PASS

scenario-17.json (report)

{
	"agent_group": "hermes",
	"expected_count": 15,
	"pass": true,
	"per_peer": {
		"node_1": 15,
		"node_2": 15,
		"node_3": 15,
		"node_4": 15
	},
	"reasons": [],
	"scenario": "17",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-17.log (console trace)

phase A: each of 3 agents writes 5 memories to scenario17-stats
  ai:alice on 138.197.28.73
  ai:bob on 45.55.164.30
  ai:charlie on 159.89.187.128
settle 15s for W=2 fanout
phase B: querying count on every peer
  node-1 count=15 (expected 15)
  node-2 count=15 (expected 15)
  node-3 count=15 (expected 15)
  node-4 count=15 (expected 15)

raw file

Scenario 18 — Semantic query expansion PASS

scenario-18.json (report)

{
	"agent_group": "hermes",
	"diag_list_alice_present": 1,
	"diag_list_bob_present": 1,
	"diag_node3_embedding_probe": "dawn-walk|1537|BYTES | ridge-strides|1537|BYTES",
	"pass": true,
	"query": "morning outdoor exercise routine",
	"reasons": [],
	"recall_mode": "hybrid",
	"rows_in_recall": 2,
	"scenario": "18",
	"skipped": false,
	"tls_mode": "off",
	"writers": [
		{
			"agent": "ai:alice",
			"marker": "alice-sunrise-3e446bac",
			"seen_by_charlie": 1
		},
		{
			"agent": "ai:bob",
			"marker": "bob-daybreak-63e4c12a",
			"seen_by_charlie": 1
		}
	]
}

raw file

scenario-18.log (console trace)

alice writes A on node-1
bob writes B on node-2
polling node-3 for both writes to propagate (max 30 s)
  both writes visible after 1 s
settle 3s for embedder + HNSW catch-up
  node-3 DB embedding probe: 'dawn-walk|1537|BYTES | ridge-strides|1537|BYTES'
charlie queries on node-3 with semantically-related prompt
  recall mode=hybrid returned 2 rows
  charlie sees alice's memory: 1 (expected >=1)
  charlie sees bob's memory: 1 (expected >=1)

raw file

Scenario 22 — Identity spoofing resistance PASS

scenario-22.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"reasons": [],
	"scenario": "22",
	"skipped": false,
	"tests": {
		"body_vs_header_conflict": {
			"acceptable": [
				"ai:body-wins",
				"ai:attacker"
			],
			"stored_agent_id": "ai:attacker"
		},
		"header_only": {
			"expected": "ai:alice",
			"stored_agent_id": "ai:alice"
		}
	},
	"tls_mode": "off"
}

raw file

scenario-22.log (console trace)

test 1: header-only X-Agent-Id=ai:alice
settle 2s for read-settle
  stored metadata.agent_id for header-only write: ai:alice (expected ai:alice)
test 2: body.metadata.agent_id=ai:body-wins vs X-Agent-Id=ai:attacker
settle 2s for read-settle
  stored metadata.agent_id for body+header conflict: ai:attacker

raw file

Scenario 23 — Malicious content fuzz PASS

scenario-23.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"payloads": {
		"html": {
			"input_bytes": 66,
			"roundtrip_bytes": 66,
			"write_http": 201
		},
		"oversize": {
			"input_bytes": 1048576,
			"roundtrip_bytes": 0,
			"write_http": 400
		},
		"sql": {
			"input_bytes": 61,
			"roundtrip_bytes": 61,
			"write_http": 201
		},
		"unicode": {
			"input_bytes": 19,
			"roundtrip_bytes": 19,
			"write_http": 201
		}
	},
	"payloads_note": "accept+faithful OR 4xx reject both acceptable for oversize",
	"reasons": [],
	"scenario": "23",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-23.log (console trace)

payload sql: 61 bytes
payload html: 66 bytes
payload oversize: 1048576 bytes
  oversize: server rejected oversize with HTTP 400 (acceptable)
payload unicode: 19 bytes

raw file

Scenario 24 — Byzantine peer PASS

scenario-24.json (report)

{
	"agent_group": "hermes",
	"byzantine_marker": "bz-4ce5e9864b12485183a1a4a66e4f275f",
	"pass": true,
	"reasons": [],
	"scenario": "24",
	"skipped": false,
	"stored_metadata_agent_id": "REJECTED_BY_SERVER",
	"sync_push_http_code": "422",
	"tls_mode": "off"
}

raw file

scenario-24.log (console trace)

node-2 sends sync_push to node-3 claiming sender_agent_id=ai:alice
  sync_push returned HTTP 422
settle 5s for server-side sync apply
  node-3 stored metadata.agent_id=ABSENT (declared: ai:alice)
  sync_push rejected HTTP 422 — stricter-than-spec, acceptable

raw file

Scenario 25 — Clock skew tolerance PASS

scenario-25.json (report)

{
	"agent_group": "hermes",
	"clock_offset_seconds": 300,
	"marker": "ck-c34089e9991a43c5870abcab27dd299d",
	"pass": true,
	"reasons": [],
	"scenario": "25",
	"seen_on": {
		"node_1": 1,
		"node_3": 1
	},
	"skipped": false,
	"target_node": "node-3",
	"tls_mode": "off"
}

raw file

scenario-25.log (console trace)

shifting node-3 clock +300s (NTP disabled for the duration)
  node-3 now reports: Sun May  3 16:21:18 UTC 2026
alice writes on node-1 (normal clock); waiting for quorum fanout to skewed node-3
settle 15s for skewed-peer convergence
  node-3 (+300s clock) sees marker: 1 (expected >=1)
  node-1 sees marker: 1 (expected >=1)
reverting node-3 clock

raw file

Scenario 28 — memory_search keyword PASS

scenario-28.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"peer_hits": {
		"node_2": 1,
		"node_3": 1
	},
	"reasons": [],
	"scenario": "28",
	"skipped": false,
	"tls_mode": "off",
	"token": "kwsearch4ba5e46b2d"
}

raw file

scenario-28.log (console trace)

alice writes a row containing unique token=kwsearch4ba5e46b2d
settle 8s for search index populate + fanout
bob + charlie call /api/v1/search with the exact token
  node-2 keyword search returned 1 hits
  node-3 keyword search returned 1 hits

raw file

Scenario 29 — memory_archive lifecycle PASS

scenario-29.json (report)

{
	"agent_group": "hermes",
	"archive_http_code": 200,
	"bob_sees_archived": true,
	"m1_id": "af22b8f7-fa02-402b-a8ef-32cc8b626a0a",
	"node4_active_rows": 1,
	"pass": true,
	"reasons": [],
	"restore_http_code": 200,
	"scenario": "29",
	"skipped": false,
	"stats_shape_ok": true,
	"tls_mode": "off"
}

raw file

scenario-29.log (console trace)

alice writes M1 on node-1
  M1 id=af22b8f7-fa02-402b-a8ef-32cc8b626a0a
settle 5s for pre-archive replication
alice archives M1 via POST /api/v1/archive (ai-memory-mcp PR #361)
  archive (POST) returned HTTP 200
settle 5s for archive propagation
bob queries /api/v1/archive on node-2
  bob sees M1 in archive: True
charlie restores M1 via /api/v1/archive/{id}/restore on node-3
  restore returned HTTP 200
settle 5s for restore propagation
node-4 aggregator: M1 must be active again
  node-4 active rows matching marker: 1
fetch /api/v1/archive/stats on node-4

raw file

Scenario 30 — memory_capabilities handshake PASS

scenario-30.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"peer_views": {
		"node_1": {
			"_path": "/api/v1/capabilities",
			"approval": {
				"pending_requests": 0
			},
			"compaction": {
				"enabled": false,
				"planned": true,
				"version": "v0.8+"
			},
			"features": {
				"auto_consolidation": false,
				"auto_tagging": false,
				"contradiction_analysis": false,
				"cross_encoder_reranking": false,
				"embedder_loaded": true,
				"hybrid_recall": true,
				"keyword_search": true,
				"memory_reflection": {
					"enabled": false,
					"planned": true,
					"version": "v0.7+"
				},
				"query_expansion": false,
				"recall_mode_active": "hybrid",
				"reranker_active": "off",
				"semantic_search": true
			},
			"hnsw": {
				"evicted_recently": false,
				"evictions_total": 0
			},
			"hooks": {
				"registered_count": 0,
				"webhook_events": [
					"memory_store",
					"memory_promote",
					"memory_delete",
					"memory_link_created",
					"memory_consolidated"
				]
			},
			"models": {
				"cross_encoder": "none",
				"embedding": "sentence-transformers/all-MiniLM-L6-v2",
				"embedding_dim": 384,
				"llm": "none"
			},
			"permissions": {
				"active_rules": 0,
				"inheritance": "enforced",
				"mode": "advisory"
			},
			"schema_version": "2",
			"tier": "semantic",
			"transcripts": {
				"enabled": false,
				"planned": true,
				"version": "v0.7+"
			},
			"version": "0.6.3+patch.1"
		},
		"node_2": {
			"_path": "/api/v1/capabilities",
			"approval": {
				"pending_requests": 0
			},
			"compaction": {
				"enabled": false,
				"planned": true,
				"version": "v0.8+"
			},
			"features": {
				"auto_consolidation": false,
				"auto_tagging": false,
				"contradiction_analysis": false,
				"cross_encoder_reranking": false,
				"embedder_loaded": true,
				"hybrid_recall": true,
				"keyword_search": true,
				"memory_reflection": {
					"enabled": false,
					"planned": true,
					"version": "v0.7+"
				},
				"query_expansion": false,
				"recall_mode_active": "hybrid",
				"reranker_active": "off",
				"semantic_search": true
			},
			"hnsw": {
				"evicted_recently": false,
				"evictions_total": 0
			},
			"hooks": {
				"registered_count": 0,
				"webhook_events": [
					"memory_store",
					"memory_promote",
					"memory_delete",
					"memory_link_created",
					"memory_consolidated"
				]
			},
			"models": {
				"cross_encoder": "none",
				"embedding": "sentence-transformers/all-MiniLM-L6-v2",
				"embedding_dim": 384,
				"llm": "none"
			},
			"permissions": {
				"active_rules": 0,
				"inheritance": "enforced",
				"mode": "advisory"
			},
			"schema_version": "2",
			"tier": "semantic",
			"transcripts": {
				"enabled": false,
				"planned": true,
				"version": "v0.7+"
			},
			"version": "0.6.3+patch.1"
		},
		"node_3": {
			"_path": "/api/v1/capabilities",
			"approval": {
				"pending_requests": 0
			},
			"compaction": {
				"enabled": false,
				"planned": true,
				"version": "v0.8+"
			},
			"features": {
				"auto_consolidation": false,
				"auto_tagging": false,
				"contradiction_analysis": false,
				"cross_encoder_reranking": false,
				"embedder_loaded": true,
				"hybrid_recall": true,
				"keyword_search": true,
				"memory_reflection": {
					"enabled": false,
					"planned": true,
					"version": "v0.7+"
				},
				"query_expansion": false,
				"recall_mode_active": "hybrid",
				"reranker_active": "off",
				"semantic_search": true
			},
			"hnsw": {
				"evicted_recently": false,
				"evictions_total": 0
			},
			"hooks": {
				"registered_count": 0,
				"webhook_events": [
					"memory_store",
					"memory_promote",
					"memory_delete",
					"memory_link_created",
					"memory_consolidated"
				]
			},
			"models": {
				"cross_encoder": "none",
				"embedding": "sentence-transformers/all-MiniLM-L6-v2",
				"embedding_dim": 384,
				"llm": "none"
			},
			"permissions": {
				"active_rules": 0,
				"inheritance": "enforced",
				"mode": "advisory"
			},
			"schema_version": "2",
			"tier": "semantic",
			"transcripts": {
				"enabled": false,
				"planned": true,
				"version": "v0.7+"
			},
			"version": "0.6.3+patch.1"
		},
		"node_4": {
			"_path": "/api/v1/capabilities",
			"approval": {
				"pending_requests": 0
			},
			"compaction": {
				"enabled": false,
				"planned": true,
				"version": "v0.8+"
			},
			"features": {
				"auto_consolidation": false,
				"auto_tagging": false,
				"contradiction_analysis": false,
				"cross_encoder_reranking": false,
				"embedder_loaded": true,
				"hybrid_recall": true,
				"keyword_search": true,
				"memory_reflection": {
					"enabled": false,
					"planned": true,
					"version": "v0.7+"
				},
				"query_expansion": false,
				"recall_mode_active": "hybrid",
				"reranker_active": "off",
				"semantic_search": true
			},
			"hnsw": {
				"evicted_recently": false,
				"evictions_total": 0
			},
			"hooks": {
				"registered_count": 0,
				"webhook_events": [
					"memory_store",
					"memory_promote",
					"memory_delete",
					"memory_link_created",
					"memory_consolidated"
				]
			},
			"models": {
				"cross_encoder": "none",
				"embedding": "sentence-transformers/all-MiniLM-L6-v2",
				"embedding_dim": 384,
				"llm": "none"
			},
			"permissions": {
				"active_rules": 0,
				"inheritance": "enforced",
				"mode": "advisory"
			},
			"schema_version": "2",
			"tier": "semantic",
			"transcripts": {
				"enabled": false,
				"planned": true,
				"version": "v0.7+"
			},
			"version": "0.6.3+patch.1"
		}
	},
	"reasons": [],
	"scenario": "30",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-30.log (console trace)

  node-1 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path']
  node-2 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path']
  node-3 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path']
  node-4 capabilities: ['approval', 'compaction', 'features', 'hnsw', 'hooks', 'models', 'permissions', 'schema_version', 'tier', 'transcripts', 'version', '_path']

raw file

Scenario 31 — memory_gc quiescence PASS

scenario-31.json (report)

{
	"agent_group": "hermes",
	"expected_live": 2,
	"forget_http_code": 400,
	"gc_http_code": 200,
	"live_markers_per_peer": {
		"node_1": 2,
		"node_2": 2,
		"node_3": 2,
		"node_4": 2
	},
	"pass": true,
	"reasons": [],
	"scenario": "31",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-31.log (console trace)

alice writes 4 memories
settle 6s for pre-gc replication
alice forgets 2 via /api/v1/forget
  forget returned HTTP 400
settle 5s for forget propagation
bob triggers /api/v1/gc on node-2
  gc returned HTTP 200
settle 8s for post-gc settle
verify remaining 2 markers are still readable on every peer
  node-1 sees 2/2 live markers
  node-2 sees 2/2 live markers
  node-3 sees 2/2 live markers
  node-4 sees 2/2 live markers

raw file

Scenario 32 — memory_inbox + notify PASS

scenario-32.json (report)

{
	"agent_group": "hermes",
	"bob_inbox_count": 1,
	"bob_sees_marker": true,
	"charlie_inbox_count": 0,
	"charlie_sees_marker": false,
	"marker": "inb-94cd9af65d0a482a935c9ae0103385e3",
	"notify_http_code": 201,
	"pass": true,
	"reasons": [],
	"scenario": "32",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-32.log (console trace)

alice calls /api/v1/notify → target=ai:bob
  notify returned HTTP 201
settle 6s for notification fanout
bob queries his inbox on node-2
  bob inbox has 1 messages; sees marker: True
charlie queries his inbox on node-3 (must NOT see it)
  charlie inbox has 0 messages; sees marker: False

raw file

Scenario 33 — memory_subscribe pub/sub PASS

scenario-33.json (report)

{
	"agent_group": "hermes",
	"m1_delivered": 1,
	"namespace": "scenario33-pubsub-f74718",
	"ns_in_subs_after": false,
	"ns_in_subs_before": true,
	"pass": true,
	"reasons": [],
	"scenario": "33",
	"skipped": false,
	"subscribe_http_code": 201,
	"subscriptions_after_count": 0,
	"subscriptions_before_count": 1,
	"tls_mode": "off",
	"unsubscribe_http_code": 200
}

raw file

scenario-33.log (console trace)

bob subscribes to namespace scenario33-pubsub-f74718 on node-2
  subscribe returned HTTP 201
settle 2s for subscription settle
  bob subscriptions: 1 entries; contains ns: True
alice writes M1 into the subscribed namespace
settle 6s for write fanout to subscribers
  bob sees M1 in subscribed namespace: 1
bob unsubscribes from scenario33-pubsub-f74718
  unsubscribe returned HTTP 200
settle 2s for unsubscribe settle
  bob subscriptions after unsubscribe: ns still present = False
alice writes M2 post-unsubscribe (may still replicate via federation but subscription list excludes ns)
settle 5s for post-unsubscribe settle

raw file

Scenario 34 — memory_pending governance PASS

scenario-34.json (report)

{
	"agent_group": "hermes",
	"approve_http_code": 200,
	"charlie_sees": {
		"approved": 1,
		"rejected": 0
	},
	"namespace": "scenario34-pending-e98707",
	"pass": true,
	"pending_queue_count": 2,
	"reasons": [],
	"reject_http_code": 200,
	"scenario": "34",
	"set_standard_http_code": 201,
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-34.log (console trace)

alice sets namespace standard on scenario34-pending-e98707: write=approve, approver=ai:bob
  set-standard returned HTTP 201
settle 2s for standard settle
alice writes two memories into the governed namespace (should land in pending)
  p1=82ceb459-4f52-4743-9bda-424191969035 p2=72f1e660-5bc0-422b-a701-5b1c3e81edd8
settle 4s for pending queue settle
bob lists pending on node-2
  pending queue has 2 entries
bob approves p1, rejects p2
  approve HTTP 200; reject HTTP 200
settle 5s for decision fanout
charlie reads the namespace — expects ONLY approved marker
  charlie sees approved=1 rejected=0

raw file

Scenario 35 — memory_namespace standards PASS

scenario-35.json (report)

{
	"agent_group": "hermes",
	"child_ns": "scenario35-parent-7d8c1c/child",
	"clear_http_code": 200,
	"get_standard_http_code": 200,
	"parent_ns": "scenario35-parent-7d8c1c",
	"pass": true,
	"post_clear_has_child_rule": false,
	"reasons": [],
	"scenario": "35",
	"sees_child_rule": true,
	"sees_parent_rule": true,
	"set_child_http_code": 201,
	"set_parent_http_code": 201,
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-35.log (console trace)

alice writes parent-standard-memory on node-1
alice sets namespace standard on scenario35-parent-7d8c1c
  set-parent returned HTTP 201
alice writes child-standard-memory on node-1
alice sets namespace standard on scenario35-parent-7d8c1c/child with parent=scenario35-parent-7d8c1c
  set-child returned HTTP 201
settle 4s for standard fanout
bob gets standard for scenario35-parent-7d8c1c/child on node-2 (expects layered parent+child)
  get-standard returned HTTP 200
  parent-rule visible=True; child-rule visible=True
alice clears standard on scenario35-parent-7d8c1c/child
  clear returned HTTP 200
settle 3s for clear settle

raw file

Scenario 36 — memory_session_start PASS

scenario-36.json (report)

{
	"agent_group": "hermes",
	"pass": true,
	"reasons": [],
	"scenario": "36",
	"session_id": "87113951-33dd-4c10-a0ce-029f9a5fd654",
	"session_tagged_rows_on_bob": 2,
	"skipped": false,
	"start_http_code": 200,
	"tls_mode": "off"
}

raw file

scenario-36.log (console trace)

alice starts a session on node-1
  session_start returned HTTP 200, session_id=87113951-33dd-4c10-a0ce-029f9a5fd654
alice writes 2 memories tagged with session_id
settle 6s for session-tagged fanout
bob lists on node-2 filtered by session_id=87113951-33dd-4c10-a0ce-029f9a5fd654
  bob sees 2 rows tagged session_id=87113951-33dd-4c10-a0ce-029f9a5fd654 (expected 2)

raw file

Scenario 37 — memory_get_links bidirectional PASS

scenario-37.json (report)

{
	"agent_group": "hermes",
	"forward_has_target": true,
	"m1": "b4a848a7-8269-4047-86a1-bfbc7e937999",
	"m2": "00f9cf56-de43-4766-a76c-38688fc5db0d",
	"pass": true,
	"reasons": [],
	"reverse_has_source": true,
	"scenario": "37",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-37.log (console trace)

alice writes M1 + M2 + links M1→M2
  M1=b4a848a7-8269-4047-86a1-bfbc7e937999 M2=00f9cf56-de43-4766-a76c-38688fc5db0d
settle 6s for link fanout
charlie queries /api/v1/links/M1 (forward)
charlie queries /api/v1/links/M2 (reverse)

raw file

Scenario 38 — /export + /import PASS

scenario-38.json (report)

{
	"agent_group": "hermes",
	"dst_ns": "scenario38-dst-4ba8b8",
	"expected_rows": 5,
	"export_http_code": 200,
	"import_http_code": 200,
	"markers_preserved": 5,
	"pass": true,
	"reasons": [],
	"rows_exported": 5,
	"rows_in_destination": 5,
	"scenario": "38",
	"skipped": false,
	"src_ns": "scenario38-src-4ba8b8",
	"tls_mode": "off"
}

raw file

scenario-38.log (console trace)

alice writes 5 rows into scenario38-src-4ba8b8
settle 4s for pre-export replication
alice exports on node-1 (endpoint has no namespace filter; filter client-side)
  export returned HTTP 200, total_rows=232
  rewrote 5 memories from scenario38-src-4ba8b8 -> scenario38-dst-4ba8b8
bob imports the payload into scenario38-dst-4ba8b8 on node-2
  import returned HTTP 200
settle 6s for import + fanout
verify row counts match on destination
  scenario38-dst-4ba8b8 has 5 rows (expected 5)
  markers preserved in destination: 5/5

raw file

Scenario 39 — /sync/since delta PASS

scenario-39.json (report)

{
	"agent_group": "hermes",
	"checkpoint": "2026-05-03T16:19:10+00:00",
	"diag_curl_body_head": "{\"count\":6,\"earliest_updated_at\":\"2026-05-03T16:19:43.231690728+00:00\",\"latest_updated_at\":\"2026-05-03T16:19:46.632196116+00:00\",\"limit\":500,\"memories\":[{\"access_count\":0,\"confidence\":1.0,\"content\":\"marker=delta-0-dfe90f971bc7448cba5604b504075ec4\",\"created_at\":\"2026-05-03T16:19:43.231690728+00:00\",\"",
	"diag_curl_exit": 0,
	"diag_curl_http_code": 200,
	"diag_curl_stderr": "",
	"diag_earliest_updated_at": "2026-05-03T16:19:43.231690728+00:00",
	"diag_latest_updated_at": "2026-05-03T16:19:46.632196116+00:00",
	"diag_node3_health_reachable": true,
	"diag_updated_since": "2026-05-03T16:19:10+00:00",
	"expected_markers": 6,
	"markers_present": 6,
	"namespace": "scenario39-delta-809204",
	"pass": true,
	"reasons": [],
	"rows_returned": 6,
	"rows_returned_raw": 6,
	"scenario": "39",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-39.log (console trace)

checkpoint = 2026-05-03T16:19:10+00:00
suspending ai-memory on node-3
  !! ssh timeout (30s): root@159.89.187.128 pgrep -f 'ai-memory serve' | xargs -r kill -STOP
alice + bob write 6 rows while node-3 is out
resuming ai-memory on node-3
settle 15s for process resume + federation catchup
  node-3 → node-1 health reachable: True (after 1 probes)
node-3 asks node-1 /api/v1/sync/since?since=2026-05-03T16:19:10+00:00
  curl exit=0 http_code=200 body_len=2898 stderr=''
  /sync/since raw=6 ns-filtered=6; 6/6 match our markers
  diag: updated_since=2026-05-03T16:19:10+00:00 earliest=2026-05-03T16:19:43.231690728+00:00 latest=2026-05-03T16:19:46.632196116+00:00

raw file

Scenario 40 — /memories/bulk PASS

scenario-40.json (report)

{
	"agent_group": "hermes",
	"bulk_http_code": "200",
	"bulk_size": 500,
	"namespace": "scenario40-bulk-7ff364",
	"pass": true,
	"per_peer_count": {
		"node_2": 500,
		"node_3": 500,
		"node_4": 500
	},
	"reasons": [],
	"scenario": "40",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-40.log (console trace)

constructing 500-row bulk payload
staging bulk payload on node-1 /tmp, then POST /api/v1/memories/bulk
  bulk POST returned HTTP 200
settle 20s for bulk fanout across 3 peers + aggregator
  node-2 count=500 (expected 500)
  node-3 count=500 (expected 500)
  node-4 count=500 (expected 500)

raw file

Scenario 41 — /metrics Prometheus PASS

scenario-41.json (report)

{
	"activity_namespace": "scenario41-activity-afc252",
	"agent_group": "hermes",
	"pass": true,
	"per_peer": {
		"node_1": {
			"counters_t0": 10,
			"counters_t1": 10,
			"regressed_keys": 0
		},
		"node_2": {
			"counters_t0": 10,
			"counters_t1": 10,
			"regressed_keys": 0
		},
		"node_3": {
			"counters_t0": 7,
			"counters_t1": 7,
			"regressed_keys": 0
		}
	},
	"reasons": [],
	"scenario": "41",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-41.log (console trace)

scrape T0
  node-1 T0 parsed 10 memory counters
  node-2 T0 parsed 10 memory counters
  node-3 T0 parsed 7 memory counters
settle 5s for counter update
scrape T1
  node-1 T1 parsed 10 memory counters
  node-2 T1 parsed 10 memory counters
  node-3 T1 parsed 7 memory counters

raw file

Scenario 42 — /namespaces enumeration PASS

scenario-42.json (report)

{
	"agent_group": "hermes",
	"namespaces": [
		"scenario42-2df017-0",
		"scenario42-2df017-1",
		"scenario42-2df017-2"
	],
	"pass": true,
	"per_peer": {
		"node_1": {
			"scenario42-2df017-0": 2,
			"scenario42-2df017-1": 2,
			"scenario42-2df017-2": 2
		},
		"node_2": {
			"scenario42-2df017-0": 2,
			"scenario42-2df017-1": 2,
			"scenario42-2df017-2": 2
		},
		"node_3": {
			"scenario42-2df017-0": 2,
			"scenario42-2df017-1": 2,
			"scenario42-2df017-2": 2
		},
		"node_4": {
			"scenario42-2df017-0": 2,
			"scenario42-2df017-1": 2,
			"scenario42-2df017-2": 2
		}
	},
	"reasons": [],
	"scenario": "42",
	"skipped": false,
	"tls_mode": "off"
}

raw file

scenario-42.log (console trace)

alice writes into 3 distinct namespaces: ['scenario42-2df017-0', 'scenario42-2df017-1', 'scenario42-2df017-2']
settle 10s for namespace index fanout
  node-1 sees 3/3 target namespaces, counts: {'scenario42-2df017-0': 2, 'scenario42-2df017-1': 2, 'scenario42-2df017-2': 2}
  node-2 sees 3/3 target namespaces, counts: {'scenario42-2df017-0': 2, 'scenario42-2df017-1': 2, 'scenario42-2df017-2': 2}
  node-3 sees 3/3 target namespaces, counts: {'scenario42-2df017-0': 2, 'scenario42-2df017-1': 2, 'scenario42-2df017-2': 2}
  node-4 sees 3/3 target namespaces, counts: {'scenario42-2df017-0': 2, 'scenario42-2df017-1': 2, 'scenario42-2df017-2': 2}

raw file

All artifacts

Generated by scripts/generate_run_html.sh. Methodology: alphaonedev.github.io/ai-memory-ai2ai-gate/methodology. Analysis source: analysis/run-insights.json.