An audit you cannot reproduce is an anecdote. This page is the single, GitHub-Pages-sourceable procedure for standing up the exact substrate the v1.0.0 enterprise-federation audit ran on — a Docker PostgreSQL 18.6 + Apache AGE 1.8.0 + pgvector 0.8.6 data tier over TLS 1.3 (hostssl-only, cleartext refused) with mutual-TLS to the store, and ai-memory wired into ONLY that tier, served on the client↔daemon mTLS API under the certified posture — and re-running the 3×7 audit scheme. Any AI NHI (or human peer reviewer) can operate the substrate exactly as this audit did.
The committed kit lives at deploy/enterprise-federation-repro/. It EXTENDS the canonical deploy/docker-1461/ provisioning lane rather than forking it: the PostgreSQL / AGE / pgvector version pins are sourced from deploy/docker-1461/provision/lib.sh (the one provisioning source of truth), the image is built from deploy/docker-1461/Dockerfile.pg-age-vector, and the fail-closed hostssl proof reuses deploy/docker-1461/pg-hostssl-healthcheck.sh. The kit adds only the audit-shaped pieces: the single-daemon standup, the two encrypted legs' certificate sets, the deterministic seed-corpus generator, and the 3×7 verify harness.
Idempotent one-command standup: stack → certs → keys → schema-init → seed → migrate → certified daemon. Every scratch artifact lands under .local-runs/ef-repro/ (gitignored; never /tmp).
Proves every invariant with real output and fails loud on any deviation: server version, extension versions, TLS on, cleartext refused, TLS 1.3 negotiated, the certified posture readout, the signed audit trail, semantic recall, and the fail-closed write gate.
Mints — into the gitignored run dir, never committed — the CA plus both encrypted legs: the PG server + client (store mTLS) and the daemon server + an allowed API client with a SHA-256 fingerprint allowlist.
# Prereqs: docker (or podman), openssl, curl, cargo (only if no prebuilt binary). deploy/enterprise-federation-repro/repro.sh # stand it up deploy/enterprise-federation-repro/verify.sh # prove the invariants
The tier image is built from the canonical in-repo Dockerfile. Apache AGE 1.8.0 installs from a pinned pgdg apt package (postgresql-18-age=1.8.0~rc0-2.pgdg13+1) overlaid on the apache/age:release_PG18_1.7.0 base — NOT a source build; CREATE EXTENSION age then reports extversion 1.8.0. Every version literal lives once, in deploy/docker-1461/provision/lib.sh, and arrives at the build as a --build-arg. repro.sh reads those same pins.
# repro.sh builds this automatically; pins come from provision/lib.sh.
docker build \
--build-arg AGE_BASE_IMAGE=apache/age:release_PG18_1.7.0 \
--build-arg PG_MAJOR=18 \
--build-arg PG_APT_VERSION=18.6-1.pgdg13+2 \
--build-arg AGE_APT_VERSION=1.8.0~rc0-2.pgdg13+1 \
--build-arg PGVECTOR_APT_VERSION=0.8.6-1.pgdg13+1 \
-f deploy/docker-1461/Dockerfile.pg-age-vector \
-t ai-memory-cert-pg:pg18.6-age1.8.0-pgv0.8.6 \
deploy/docker-1461
The first-boot hook creates only the two extension binaries the sal-postgres adapter needs. The AGE graph itself (create_graph('memory_graph')) is created later by ai-memory schema-init, so this stays purely about the binaries.
# initdb/01-extensions.sql
CREATE EXTENSION IF NOT EXISTS age;
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE lands in whichever schema the connection's search_path resolves first. If ag_catalog precedes the app schema, ai-memory's own tables would be created inside ag_catalog. The store DSN this kit builds pins public,ag_catalog (public first), so app tables live in public and AGE's catalog stays reachable. This is the honest, load-bearing ordering — stated, not hidden.There are NO plain host lines. A plaintext (sslmode=disable) TCP connection is refused PRE-AUTH with "no pg_hba.conf entry … no encryption"; clientcert=verify-full additionally demands a client certificate whose CN matches the connecting role. The local unix socket stays trust so the container's own bootstrap + the hostssl healthcheck work without a password.
# initdb/pg_hba.conf # TYPE DATABASE USER ADDRESS METHOD local all all trust hostssl all all 0.0.0.0/0 scram-sha-256 clientcert=verify-full hostssl all all ::/0 scram-sha-256 clientcert=verify-full
postgres \ -c shared_preload_libraries=age \ -c ssl=on \ -c ssl_cert_file=/etc/postgresql/tls/server.pem \ -c ssl_key_file=/etc/postgresql/tls/server.key \ -c ssl_ca_file=/certs/ca.pem \ -c ssl_min_protocol_version=TLSv1.2 \ -c hba_file=/etc/postgresql/pg_hba.conf
A bind-mounted host key keeps its host uid and PostgreSQL refuses an ssl_key_file it does not own, so the entrypoint wrapper installs the server cert/key to a postgres-owned 0600 path before handing off. Readiness is gated on the fail-closed pg-hostssl-healthcheck.sh — "healthy" means hostssl is PROVABLY armed, not merely that the postmaster is up.
Generation is part of the reproduction. gen-certs.sh mints all material into the gitignored run dir, idempotently (existing material is reused so the trust topology is stable across re-runs). Two legs, one CA.
| Leg | Material | Purpose |
|---|---|---|
| Store (daemon → PG) | ca.crt · pg-server.{crt,key} (SAN IP:127.0.0.1,DNS:localhost) · pg-client.{crt,key} (CN = the pg role ai_memory) | The daemon dials the store under sslmode=verify-full — chain + hostname validated — and presents its client cert for mTLS. |
| API (AI NHI → daemon) | daemon-server.{crt,key} (SAN IP:127.0.0.1,DNS:localhost) · client-good.{crt,key} · allowlist.txt (SHA-256(DER) fingerprint pin) | The client↔daemon mTLS API on :9077: the daemon demands a client cert whose fingerprint is on the allowlist. |
migrateThe audit ran over a 7,855-row Atlas corpus (embedder = local all-MiniLM-L6-v2, 384-d, under loopback-only egress). That corpus is proprietary; a portable reproduction ships a generator, not the data. seed-corpus.sh writes a small, byte-deterministic corpus into a local sqlite DB — every run produces identical rows — then repro.sh migrates it into the pg tier verbatim (memories + links + embeddings; #3054/#3060).
ai-memory schema-init --store-url "postgres://…?sslmode=verify-full&sslrootcert=ca.crt&sslcert=pg-client.crt&sslkey=pg-client.key&options=-csearch_path%3Dpublic%2Cag_catalog" ai-memory migrate --from "sqlite:///…/seed.db" --to "<same TLS DSN>" --batch 1000
Embeddings are derived, not durable. The seed rows carry only their durable TEXT — the source of truth. The certified daemon RE-DERIVES the embedding vectors from that text at serve-boot (its backfill sweep, under the local MiniLM embedder / loopback-only egress). Embedding vectors are disposable, regenerable artifacts; the memory text is never at risk. Semantic recall over the seed exercises the result.
The posture is not a prose checklist — it is the checked-in profile docs/deploy/enterprise-federation.env, validated by src/enterprise_federation_posture.rs (the single source of truth both the boot gate and ai-memory doctor --posture enterprise-federation share). repro.sh renders the effective posture by EXTENDING that template (its KEY=VALUE lines) with the per-deployment values, then arms the boot-refusing gate — the daemon boots ONLY when every one of the 19 controls passes (#2954 added check #19, the append-only-audit-spine-armed pairing).
| Control group | What the posture pins |
|---|---|
| Hardened base | AI_MEMORY_SECURITY_PROFILE=asi-hard engages the no-disable posture (its pinned knobs at their hard floor, including AI_MEMORY_DB_SYNCHRONOUS=FULL). |
| Federation receive | Peer enrollment required, per-message Ed25519 signatures + nonce freshness, inbound-write namespace confinement, stale-policy refusal; the two trust-bypass knobs must stay UNSET. |
| Governance | AI_MEMORY_PERMISSIONS_MODE=enforce, governance fail-CLOSED on rule-consultation error. |
| Trust material | A set trust domain, a readable peer server-cert pin file, valid peer-attestation JSON with no bare ** allow-all glob. |
| Transport + at-rest | https-only peers (plaintext-peer refusal in force); at-rest encryption on a sqlcipher build. |
| Self-attestation | The boot-refusing gate itself armed, so later posture drift cannot silently boot. |
# repro.sh sources the posture env, then: ai-memory doctor --posture enterprise-federation # ALL PASS, or repro refuses to serve ai-memory serve \ --host 127.0.0.1 --port 9077 \ --tls-cert daemon-server.crt --tls-key daemon-server.key \ --mtls-allowlist allowlist.txt # the daemon signs as AI_MEMORY_AGENT_ID=ai:cert-fed-proxy and reads the # postgres DSN from AI_MEMORY_STORE_URL_FILE (0600, non-argv secure channel).
doctor --posture machine-checks the 18 boot-time controls. It does NOT check the four audit-custody require-modes (witness / role-separation / cause-binding / rollback), which are consumed by the SEPARATE ai-memory verify-audit-trail. A node can boot clean and pass doctor --posture while verify-audit-trail still reports a lane needing key enrollment. This kit runs BOTH.The AI NHI connects through the client↔daemon mTLS API — presenting the allowed client cert + the CA + its X-Agent-Id principal — the same way the audit operated the substrate.
RUN=.local-runs/ef-repro
# stats / recall / a signed store, all over mTLS
curl --cacert "$RUN/tls/ca.crt" --cert "$RUN/tls/client-good.crt" --key "$RUN/tls/client-good.key" \
-H 'X-Agent-Id: ai:cert-fed-proxy' \
https://127.0.0.1:9077/api/v1/stats
curl --cacert "$RUN/tls/ca.crt" --cert "$RUN/tls/client-good.crt" --key "$RUN/tls/client-good.key" \
-H 'X-Agent-Id: ai:cert-fed-proxy' -H 'Content-Type: application/json' \
-X POST --data '{"context":"append-only signed audit chain","namespace":"ef-repro/audit","limit":5}' \
https://127.0.0.1:9077/api/v1/recall
An MCP-over-mTLS shim points the same client-cert material at the HTTP API, so an MCP host drives memory_store / memory_recall against the certified daemon.
verify.sh — real output, fail-loud on deviationEach check prints what it observed AND what it required, so a reviewer reads the evidence directly rather than trusting a green tick.
| # | Invariant | How it is proven |
|---|---|---|
| 1 | SELECT version() = PostgreSQL 18.6 | psql over the container's local trust socket. |
| 2 | age / vector extversion = 1.8.0 / 0.8.6 | SELECT extversion FROM pg_extension. |
| 3 | SHOW ssl = on | direct GUC read. |
| 4 | cleartext refused | a sslmode=disable TCP connection is rejected pre-auth (hostssl healthcheck + host-side probe). |
| 5 | TLS 1.3 negotiated | openssl s_client -starttls postgres reports Protocol: TLSv1.3. |
| 6 | certified posture | ai-memory doctor --posture enterprise-federation exits 0 (all controls PASS). |
| 7 | signed audit trail | ai-memory verify-audit-trail --store-url <DSN> — the append-only signed_events hash chain + witness + role-separation lanes. |
| 8 | semantic recall | a recall over the seed returns rows over the mTLS API (keyword is an honest DEGRADE when the local embedder is not pre-staged). |
| 9 | write governance | an unsigned POST /api/v1/memories is refused 403 (agent-attestation enforced on the HTTP-direct surface). |
The audit method is a matrix: each of seven feature domains is examined through three adversarial lenses. The lenses are deliberately independent so they do not converge by groupthink — one asks whether the feature works on this postgres tier at all, one whether its control is actually enforced (not merely present), and one whether the published claim matches the code. A domain passes a cell only when its probe returns the evidence the lens demands.
| Feature domain | L1 · works-on-pg | L2 · control-enforced | L3 · claims-match-code |
|---|---|---|---|
| Recall / search | a recall over the mTLS API returns seed rows; the reported mode is honest (semantic or degraded keyword). | recall is PURE — it mutates no memories row; the query-embed budget degrades loudly rather than hanging. | the recall pipeline in src/handlers/recall.rs matches the documented blend + fail-closed degrade. |
| Store / write-governance | a signed write lands and reads back over the pg store. | an unsigned HTTP-direct write is refused 403; secret-screen refuses credential-shaped content. | AI_MEMORY_REQUIRE_AGENT_ATTESTATION surface scoping in src/identity/attest.rs matches the documented tri-state. |
| Federation | /api/v1/sync/* answers over mTLS. | an unenrolled peer / out-of-scope namespace push is refused; the two trust-bypass knobs are UNSET. | the receive gates in src/federation/receive_auth.rs match the documented fail-closed defaults. |
| Audit chain | verify-audit-trail walks the chain over postgres. | a tail-truncation / mid-suffix rewrite is DETECTED; witness + role-separation lanes verify. | the tamper-evidence scope in src/signed_events.rs matches what the docs claim is detectable. |
| Identity / capability / governance | agent identity + capability tokens resolve over mTLS. | governance enforce mode denies a policy-violating action; capability caveats are non-escalating. | the macaroon caveat chain in src/governance/capability.rs matches the attenuation-only claim. |
| Knowledge-graph (AGE) | create_graph('memory_graph') exists; a Cypher traversal returns. | the relational recursive-CTE path stays read-your-own-write correct under deferred projection. | the AGE + recursive-CTE dual path in src/kg/ matches the documented traversal semantics. |
| Postgres coverage | every advertised operation the tier supports answers. | an operation with no postgres trait coverage returns an HONEST 501 (never a silent success). | the postgres-gate surface in src/handlers/postgres_gate.rs matches the documented supported set. |
Drive any API surface with the allowed client cert + CA + X-Agent-Id (see §5). Negative tests point the cert at a rogue / absent identity and assert the refusal.
psql over the local trust socket for substrate facts (pg_extension, pg_stat_ssl, SHOW ssl); a verify-full TLS connection for the mTLS + TLS-1.3 proof.
For the L3 claims-match-code lens: query the codegraph knowledge graph (or read the cited src/… symbol) to confirm the enforcing code matches the published claim, following real call edges rather than text matches.
A cell is PASS only when its probe returns the evidence the lens demands with real output; DEGRADE when the substrate correctly returns fewer / weaker results without producing a wrong one (e.g. keyword recall when the local embedder is absent) — a pass under the data-integrity North Star; FAIL on any wrong result, silent success where a refusal was required, or a claim the code does not support. A domain is certified only when all three lenses are PASS or a documented DEGRADE; a single FAIL blocks the domain and is filed, tracked, and fixed.
CREATE TABLE lands in ag_catalog if it precedes the app schema on the search path. The store DSN pins public first; this is stated, not hidden.501 (unsupported_on_postgres) — never a silent success. The 3×7 postgres-coverage domain probes exactly this boundary.AI_MEMORY_ENCRYPT_AT_REST=1 (which governs local sqlite). Encrypting the durable memory ROWS in the postgres tier is the PG tier's responsibility — disk / tablespace / pgcrypto encryption — tracked as #3061.repro.sh enrolls. Under the certified asi-hard posture the identity-lineage require-mode additionally needs a genesis succession record enrolled; that enrollment is a tracked follow-up, and verify.sh reports the exact lane honestly rather than claiming a false clean.