v0.7.0 — DigitalOcean scaled-down T4 swarm campaign (2026-06-02)

Operator directive (2026-06-02): “do something similar to the [T4] reference architecture but scaled down significantly — simulating swarm architecture and swarm testing.” AI NHI autonomous, $75 hard DO budget cap, native droplets (no Docker), swarm (not hive). This is the audit-trail evidence per the prime directive (discovery → tracker → fix → retest → close) for the live multi-node federation campaign behind EPIC #1461.

Reference contract: docs/architectures-t4.html (data-center swarm — peer mesh, W-of-N quorum writes, mTLS fingerprint allowlist, federated governance plane, optional Postgres backbone).

1. Baseline test environment

Field	Value
Cloud	DigitalOcean, region `nyc3`, VPC `10.108.0.0/24`
OS (all nodes)	Ubuntu 24.04 LTS x64 (`ubuntu-24-04-x64`)
Toolchain	`rustc 1.96.0 (ac68faa20 2026-05-25)` (native build on the 8 GB node)
Binary	`ai-memory v0.7.0`, branch `release/v0.7.0` @ `ff8caac8`, built `--release --features sal-postgres` (amd64), scp’d to peers
Schema version	v54 (`CURRENT_SCHEMA_VERSION`)
Quorum	N=3, W=2, `--quorum-timeout-ms 2000`, `--catchup-interval-secs 15`
mTLS	ed25519 private CA; per-peer ed25519 leaf certs (SAN = private+public IP); allowlist = SHA-256(leaf DER), one per line
Postgres backbone	PostgreSQL 16 + Apache AGE 1.5.0 + pgvector 0.8.2 (native apt/source install, no Docker)
Budget	`$75` cap; actual burn tracked below

Node inventory

Role	Name	Size	vCPU/RAM	Public IP	Private IP
PG/AGE + build host	`amx-pg`	s-4vcpu-8gb	4 / 8 GB	143.198.14.248	10.108.0.2
Quorum peer	`amx-peer-1`	s-2vcpu-4gb	2 / 4 GB	104.131.186.185	10.108.0.3
Quorum peer	`amx-peer-2`	s-2vcpu-4gb	2 / 4 GB	159.203.82.155	10.108.0.5
Quorum peer	`amx-peer-3`	s-2vcpu-4gb	2 / 4 GB	45.55.35.52	10.108.0.4

Cost: 1× s-4vcpu-8gb ($0.0714/hr) + 3× s-2vcpu-4gb ($0.0357/hr) ≈ $0.18/hr — a multi-hour campaign is a few dollars, far under the $75 cap. All droplets tagged ai-memory-swarm; torn down at campaign end.

2. Scenarios (T4 contract)

#	Scenario	Expectation	Result
S1	Quorum-write success (W=2, all peers up)	`201`, local commit + ≥1 peer ack	PASS
S2	Quorum not met (partition majority)	`503 quorum_not_met` when acks < W−1	PASS
S3	mTLS allowlist rejection	non-allowlisted client cert refused at handshake	PASS
S4	Post-partition convergence	write made while a peer is down converges on rejoin	PASS
S5	Postgres + Apache AGE backend	`--store-url postgres://…` boots, migrates to v54, CRUD + AGE Cypher	PASS

Evidence per scenario

S1 — quorum write. POST /api/v1/memories to amx-peer-1 (W=2) → HTTP 201, body quorum_acks: 2. GET /memories/{id} on amx-peer-2 and amx-peer-3 both return the row with matching content. The returned expires_at is populated (created + 7 d for the mid tier) — a live cross-check that the #1466 backfill chokepoint fires on the federated write path. (Methodology note: cross-node reads require a non-private scope + stable agent_id; the first probe used the default private scope under a per-request anonymous: id and so was correctly invisible to other agents — not a defect.)
S2 — quorum not met. With amx-peer-2 + amx-peer-3 stopped, a write to amx-peer-1 returns HTTP 503, {"error":"quorum_not_met","got":1,"needed":2,"reason":"timeout"}, Retry-After: 2 — exactly the T4 shortfall contract.
S3 — mTLS allowlist. A rogue ed25519 cert (own CA, fingerprint not on the allowlist) and a no-client-cert request are both refused at the TLS handshake (curl exit 56, HTTP 000); the allowlisted amx-peer-1 cert control returns 200.
S4 — convergence. A write committed on amx-peer-1+amx-peer-2 while amx-peer-3 was down (201, acks: 2) converged on amx-peer-3 after it restarted with an empty DB — via the federation push DLQ-replay worker + the 15 s catchup loop. Daemon log confirms /api/v1/sync/* is mTLS-gated (api-key check bypassed when an allowlist is configured — the client cert is the peer identity).
S5 — Postgres + Apache AGE. Daemon launched with --store-url postgres://aimem@localhost/aimem migrated the schema v50 → v54 live, including v54 (#1466) backfilled tier-default expiry on the postgres ladder. create_memory → 201, GET → 200 (expiry backfilled), and the row is confirmed in PG via psql. Apache AGE 1.5.0 Cypher CREATE (:Agent)-[:WROTE]->(:Memory) + MATCH returns swarm-tester → pg-s5. (The daemon logs a documented Wave-3 notice that some non-CRUD handlers return 501 on the postgres SAL surface — expected v0.7.0 scope per docs/postgres-age-guide.md, not a defect.)

3. Findings

Zero defects. All five scenarios passed against the live release/v0.7.0 binary. No banned-phrase deferrals; nothing to file. Two items were initially mistaken for failures and resolved as test methodology / documented scope (S1 scope-visibility, S5 Wave-3 501), not product bugs. The campaign also served as a live regression check on #1466 (TTL backfill confirmed on both the federated SQLite write path and the postgres CRUD path).

4. Verdict

The scaled-down T4 swarm is GREEN — peer-mesh quorum writes, quorum shortfall semantics, mTLS fingerprint-allowlist identity gating, post-partition convergence, and the Postgres + Apache AGE backbone all behave to the T4 contract on native DigitalOcean droplets (no Docker). Total spend ≈ $0.18 (≈1 droplet-hour × 4 nodes), far under the $75 cap. Droplets torn down at campaign close (see teardown record).