../ runs index · rendered on Pages
Run focus
What this campaign set out to test: Phase 4 with three local chaos processes on distinct ports, all sharing the chaos-client droplet as designed. 50 cycles × 2 fault classes.
What it demonstrated: Proved the harness infrastructure (processes, ports, iptables, signals) is structurally sound — writes reach the handler. Disproved that `source="chaos"` is an acceptable caller label for production ai-memory instances with default validation. Reached the same validation boundary that blocked Phase 2 three campaigns earlier.
Detailed tri-audience analysis is below, followed by per-phase test results for all four phases of the protocol — including any phase that did not run in this campaign.
AI NHI analysis · Claude Opus 4.7
Chaos writes now reach the server but all return 400 because `"chaos"` isn't in VALID_SOURCES. Same class of bug as Phase 2 at r12, different location.
Phase 4 with three local chaos processes on distinct ports, all sharing the chaos-client droplet as designed. 50 cycles × 2 fault classes.
Proved the harness infrastructure (processes, ports, iptables, signals) is structurally sound — writes reach the handler. Disproved that `source="chaos"` is an acceptable caller label for production ai-memory instances with default validation. Reached the same validation boundary that blocked Phase 2 three campaigns earlier.
Two test-rig labelling problems of the same shape, in two different phases, three weeks apart. The first time we adapted the test (used a different label). This time we decided the label was legitimate and belonged in the product's allowlist — chaos-originated writes are a real thing, distinct from API traffic, and ai-memory operators should be able to filter by them in logs. Product change, small, principled.
The product's input-validation allowlist has now caught TWO test-harness label mismatches in a single campaign series. That's not a defect; it's the system working. We extended the allowlist for chaos specifically because chaos traffic is a legitimate caller class worth distinguishing in production audit logs (you want to know which writes came from fault-injection tests vs real usage). This is the kind of decision that gets documented once and then stays documented — PR #310 pinned it in validate.rs + MCP schema + TS SDK in lockstep to prevent drift.
Same validate_source rejection as r12 (source="ship-gate") but different correct fix. `"ship-gate"` is a test-harness marker and was pragmatically replaced by `"api"` in the harness. `"chaos"` is a legitimate caller class — operational audits benefit from distinguishing chaos-campaign writes from real API traffic in mixed-source databases. Fix: add `"chaos"` to `VALID_SOURCES` in `src/validate.rs` + the `enum` in `src/mcp.rs` tool schema + the `Source` union in `sdk/typescript/src/types.ts` (three-file triple-sync invariant; breaking any one creates runtime-vs-compile-time drift). PR #310 merged.
source="chaos" rejected by validate_source (all 5000 writes → 400)
Impact: Phase 4 convergence_bound stuck at 0; total_ok 0 / total_fail 5000. No product behaviour exercised.
Root cause: `"chaos"` was not in VALID_SOURCES. validate_source returns 400 before federation fanout. Harness payload was otherwise well-formed.
Fixed in:
PR #310 extends VALID_SOURCES + MCP schema + TS SDK in one commit. r17 should see chaos writes returning 201 pre-fault and exercising the fault-tolerance logic.
What this phase proves: Single-node CRUD, backup, curator dry-run, and MCP handshake on each of the three peer droplets. Establishes that ai-memory starts and is functional at the one-node level before federation is exercised.
{
"phase": 1,
"host": "aim-v0-6-0-0-final-r16-node-a",
"version": "ai-memory 0.6.0",
"pass": true,
"reasons": [
""
],
"stats": {
"total": 1,
"by_tier": [
{
"tier": "mid",
"count": 1
}
],
"by_namespace": [
{
"namespace": "ship-gate-phase1",
"count": 1
}
],
"expiring_soon": 0,
"links_count": 0,
"db_size_bytes": 139264
},
"curator": {
"started_at": "2026-04-20T12:10:01.782719619+00:00",
"completed_at": "2026-04-20T12:10:01.783259384+00:00",
"cycle_duration_ms": 0,
"memories_scanned": 1,
"memories_eligible": 1,
"auto_tagged": 0,
"contradictions_found": 0,
"operations_attempted": 0,
"operations_skipped_cap": 0,
"autonomy": {
"clusters_formed": 0,
"memories_consolidated": 0,
"memories_forgotten": 0,
"priority_adjustments": 0,
"rollback_entries_written": 0,
"errors": []
},
"errors": [
"no LLM client configured"
],
"dry_run": true
},
"mcp_tool_count": 36,
"recall_count": 1,
"snapshot_count": 1,
"manifest_count": 1
}
{
"phase": 1,
"host": "aim-v0-6-0-0-final-r16-node-b",
"version": "ai-memory 0.6.0",
"pass": true,
"reasons": [
""
],
"stats": {
"total": 1,
"by_tier": [
{
"tier": "mid",
"count": 1
}
],
"by_namespace": [
{
"namespace": "ship-gate-phase1",
"count": 1
}
],
"expiring_soon": 0,
"links_count": 0,
"db_size_bytes": 139264
},
"curator": {
"started_at": "2026-04-20T12:10:01.305932295+00:00",
"completed_at": "2026-04-20T12:10:01.306527273+00:00",
"cycle_duration_ms": 0,
"memories_scanned": 1,
"memories_eligible": 1,
"auto_tagged": 0,
"contradictions_found": 0,
"operations_attempted": 0,
"operations_skipped_cap": 0,
"autonomy": {
"clusters_formed": 0,
"memories_consolidated": 0,
"memories_forgotten": 0,
"priority_adjustments": 0,
"rollback_entries_written": 0,
"errors": []
},
"errors": [
"no LLM client configured"
],
"dry_run": true
},
"mcp_tool_count": 36,
"recall_count": 1,
"snapshot_count": 1,
"manifest_count": 1
}
{
"phase": 1,
"host": "aim-v0-6-0-0-final-r16-node-c",
"version": "ai-memory 0.6.0",
"pass": true,
"reasons": [
""
],
"stats": {
"total": 1,
"by_tier": [
{
"tier": "mid",
"count": 1
}
],
"by_namespace": [
{
"namespace": "ship-gate-phase1",
"count": 1
}
],
"expiring_soon": 0,
"links_count": 0,
"db_size_bytes": 139264
},
"curator": {
"started_at": "2026-04-20T12:10:01.636234669+00:00",
"completed_at": "2026-04-20T12:10:01.636731017+00:00",
"cycle_duration_ms": 0,
"memories_scanned": 1,
"memories_eligible": 1,
"auto_tagged": 0,
"contradictions_found": 0,
"operations_attempted": 0,
"operations_skipped_cap": 0,
"autonomy": {
"clusters_formed": 0,
"memories_consolidated": 0,
"memories_forgotten": 0,
"priority_adjustments": 0,
"rollback_entries_written": 0,
"errors": []
},
"errors": [
"no LLM client configured"
],
"dry_run": true
},
"mcp_tool_count": 36,
"recall_count": 1,
"snapshot_count": 1,
"manifest_count": 1
}
What this phase proves: 4 agents × 50 writes against the 3-node federation with W=2 quorum, then 90s settle and convergence count on every peer. Plus two quorum probes (one-peer-down must 201, both-peers-down must 503). Catches silent-data-loss and quorum-misclassification regressions.
{
"phase": 2,
"pass": true,
"total_writes": 200,
"ok": 200,
"quorum_not_met": 0,
"fail": 0,
"counts": {
"a": 200,
"b": 200,
"c": 200
},
"probe1_single_peer_down": "201",
"probe2_both_peers_down": "503",
"reasons": [
""
]
}
What this phase proves: 1000-memory round-trip: SQLite → Postgres, re-run for idempotency, Postgres → SQLite. Asserts zero errors and counts match. Catches migration-correctness regressions in either direction of a production upgrade path.
{
"phase": 3,
"pass": true,
"report_forward": {
"batches": 1,
"dry_run": false,
"errors": [],
"from_url": "sqlite:///tmp/phase3-source.db",
"memories_read": 1000,
"memories_written": 1000,
"to_url": "postgres://ai_memory:ai_memory_test@127.0.0.1:5433/ai_memory_test"
},
"report_idempotent": {
"batches": 1,
"dry_run": false,
"errors": [],
"from_url": "sqlite:///tmp/phase3-source.db",
"memories_read": 1000,
"memories_written": 1000,
"to_url": "postgres://ai_memory:ai_memory_test@127.0.0.1:5433/ai_memory_test"
},
"report_reverse": {
"batches": 1,
"dry_run": false,
"errors": [],
"from_url": "postgres://ai_memory:ai_memory_test@127.0.0.1:5433/ai_memory_test",
"memories_read": 1000,
"memories_written": 1000,
"to_url": "sqlite:///tmp/phase3-roundtrip.db"
},
"src_count": 1000,
"dst_count": 1000,
"reasons": [
""
]
}
What this phase proves: packaging/chaos/run-chaos.sh on the chaos-client droplet with 50 cycles × 100 writes per fault class. Measures convergence_bound = min(count_node1, count_node2) / total_ok. Catches fault-tolerance regressions under SIGKILL of the primary, brief network partition, and related fault models.
[chaos] chaos campaign: fault=kill_primary_mid_write cycles=50 writes/cycle=100
[chaos] workdir: /tmp/phase4-kill_primary_mid_write
[chaos] binary: /usr/local/bin/ai-memory
[chaos] cycle 1: nodes ready (pids 4426 4428 4430)
[chaos] cycle 2: nodes ready (pids 4791 4793 4795)
[chaos] cycle 3: nodes ready (pids 5132 5134 5136)
[chaos] cycle 4: nodes ready (pids 5471 5473 5475)
[chaos] cycle 5: nodes ready (pids 5810 5812 5814)
[chaos] cycle 6: nodes ready (pids 6149 6151 6153)
[chaos] cycle 7: nodes ready (pids 6488 6490 6492)
[chaos] cycle 8: nodes ready (pids 6827 6829 6831)
[chaos] cycle 9: nodes ready (pids 7166 7168 7170)
[chaos] cycle 10: nodes ready (pids 7507 7509 7511)
[chaos] cycle 11: nodes ready (pids 7848 7850 7852)
[chaos] cycle 12: nodes ready (pids 8187 8189 8191)
[chaos] cycle 13: nodes ready (pids 8526 8528 8530)
[chaos] cycle 14: nodes ready (pids 8865 8867 8869)
[chaos] cycle 15: nodes ready (pids 9204 9206 9208)
[chaos] cycle 16: nodes ready (pids 9544 9546 9548)
[chaos] cycle 17: nodes ready (pids 9883 9885 9887)
[chaos] cycle 18: nodes ready (pids 10222 10224 10226)
[chaos] cycle 19: nodes ready (pids 10561 10563 10565)
[chaos] cycle 20: nodes ready (pids 10900 10902 10904)
[chaos] cycle 21: nodes ready (pids 11241 11243 11245)
[chaos] cycle 22: nodes ready (pids 11580 11582 11584)
[chaos] cycle 23: nodes ready (pids 11919 11921 11923)
[chaos] cycle 24: nodes ready (pids 12258 12260 12262)
[chaos] cycle 25: nodes ready (pids 12597 12599 12601)
[chaos] cycle 26: nodes ready (pids 12936 12938 12940)
[chaos] cycle 27: nodes ready (pids 13275 13277 13279)
[chaos] cycle 28: nodes ready (pids 13614 13616 13618)
[chaos] cycle 29: nodes ready (pids 13953 13955 13957)
[chaos] cycle 30: nodes ready (pids 14293 14295 14297)
[chaos] cycle 31: nodes ready (pids 14632 14634 14636)
[chaos] cycle 32: nodes ready (pids 14971 14973 14975)
[chaos] cycle 33: nodes ready (pids 15315 15317 15319)
[chaos] cycle 34: nodes ready (pids 15656 15658 15660)
[chaos] cycle 35: nodes ready (pids 15995 15997 15999)
[chaos] cycle 36: nodes ready (pids 16334 16336 16338)
[chaos] cycle 37: nodes ready (pids 16673 16675 16677)
[chaos] cycle 38: nodes ready (pids 17012 17014 17016)
[chaos] cycle 39: nodes ready (pids 17353 17355 17357)
[chaos] cycle 40: nodes ready (pids 17692 17694 17696)
[chaos] cycle 41: nodes ready (pids 18031 18033 18035)
[chaos] cycle 42: nodes ready (pids 18372 18374 18376)
[chaos] cycle 43: nodes ready (pids 18711 18713 18715)
[chaos] cycle 44: nodes ready (pids 19050 19052 19054)
[chaos] cycle 45: nodes ready (pids 19389 19391 19393)
[chaos] cycle 46: nodes ready (pids 19728 19730 19732)
[chaos] cycle 47: nodes ready (pids 20067 20069 20071)
[chaos] cycle 48: nodes ready (pids 20406 20408 20410)
[chaos] cycle 49: nodes ready (pids 20745 20747 20749)
[chaos] cycle 50: nodes ready (pids 21084 21086 21088)
[chaos] ---- summary ----
{
"total_cycles": 50,
"total_writes": 5000,
"total_ok": 0,
"total_quorum_not_met": 0,
"total_fail": 5000,
"convergence_bound": 0
}
[chaos] per-cycle JSONL: /tmp/phase4-kill_primary_mid_write/chaos-report.jsonl
Every JSON committed to this campaign directory. Raw, machine-readable, and stable.