Signed events V-4 closeout — cross-row hash chain

v0.7.0 flips the V-4 validation from YELLOW to GREEN by adding a cross-row hash chain to the signed_events audit table. Each row carries prev_hash + sequence so the chain itself is tamper-evident, not just each individual signed payload. Verify end-to-end with ai-memory verify-signed-events-chain.

• Issue trail: #698.
• Code paths: src/signed_events.rs, the migrate_v34_backfill_chain function in src/storage/migrations.rs.
• Schema:
  • migrations/sqlite/0020_v07_signed_events.sql — base signed_events table (schema v25 origin; the v34 chain columns landed in the same file as inline comments).
  • migrations/sqlite/0028_v07_signed_events_chain.sql — V-4 cross-row chain columns (schema v34).
  • migrations/postgres/0015_v07_signed_events_chain.sql — postgres mirror (postgres ladder ran one step behind).
• CLI verb: src/cli/verify_signed_events.rs — ai-memory verify-signed-events-chain [--since N] [--format text|json].

Row shape (v34)

CREATE TABLE signed_events (
  id            TEXT PRIMARY KEY,             -- UUIDv4 minted by writer
  agent_id      TEXT NOT NULL,                -- writer identity
  event_type    TEXT NOT NULL,                -- e.g. "memory_link.created"
  payload_hash  BLOB NOT NULL,                -- SHA-256 over canonical-CBOR payload
  signature     BLOB,                         -- Ed25519 over the source row (NULL when unsigned)
  attest_level  TEXT NOT NULL,                -- "unsigned" | "signed" | …
  timestamp     TEXT NOT NULL,                -- RFC3339 UTC
  prev_hash     BLOB NOT NULL,                -- 32 bytes; zero for first row (v34)
  sequence      INTEGER NOT NULL              -- monotonic per-DB; UNIQUE (v34)
);

CREATE UNIQUE INDEX signed_events_sequence_uniq ON signed_events(sequence);

The canonical bytes that the chain hash covers are (src/signed_events.rs::canonical_chain_bytes):

id || 0x1F || agent_id || 0x1F || event_type || 0x1F ||
payload_hash || 0x1F || signature_or_empty || 0x1F ||
attest_level || 0x1F || timestamp || 0x1F || sequence_be_8_bytes

The separator 0x1F (ASCII Unit Separator) is present in neither RFC3339 timestamps nor UUIDs nor the hex/base64 payloads — so concatenation is unambiguous without escaping.

prev_hash for row N is SHA-256(canonical_chain_bytes(row N-1)). For row 1, prev_hash is 32 zero bytes (ZERO_HASH in src/signed_events.rs). Tampering with any prior row’s canonical fields invalidates every subsequent row’s prev_hash.

prev_hash and sequence are populated by append_signed_event (src/signed_events.rs::append_signed_event) at insert time. Callers MUST NOT pre-populate them — any value set by the caller is ignored. Use ..SignedEvent::default() at the struct-literal tail to leave them empty.

Unsigned posture. Per-row Ed25519 sig population is gated on the resolved daemon agent_id having a *.priv keypair on disk under the key directory. When no signing key loads, the daemon boots with a stderr “continuing unsigned” line (src/main.rs:118) and writes rows with an empty signature + attest_level = "unsigned". The cross-row hash chain remains tamper-evident in either posture — only the per-row signature property is degraded.

Backfill (v33 → v34)

A fresh v0.7.0 install starts at v34 and writes the chain from row 1. A v0.7-alpha install at v33 has rows without prev_hash / sequence; the migrate_v34_backfill_chain function in src/storage/migrations.rs walks the existing rows in rowid order, assigns sequential sequence numbers, and computes prev_hash from the prior row’s canonical bytes. The backfill is idempotent — re-running on an already-backfilled table is a no-op.

A partially-backfilled state (some rows have sequence IS NULL, others don’t) is load-bearing-bad. read_chain_head (src/signed_events.rs::read_chain_head) hard-fails with a clear diagnostic in this case (the COR-9 fix; cluster-C issue #767) and refuses to append further rows until the operator re-runs the schema migration (re-open the DB with the v0.7.0 binary — migrations apply automatically on open and the backfill is idempotent). Silently treating NULL as 0 would collide with the legitimately-backfilled first row on the UNIQUE index, masking a real migration-needed state behind a misleading SQLITE_CONSTRAINT_UNIQUE.

Pinned by tests/signed_events_chain_v34.rs:

• First-row prev_hash is zero.
• Multi-row chaining (each row’s prev_hash = SHA-256 of prior canonical bytes).
• Payload tamper (the signature still verifies in isolation, but the chain walk catches it on the next row).
• Sequence tamper (gaps fail the chain check).
• Concurrent drainer inserts via PE-3 (the wire-point pre-write hook guarantees the chain stays monotonic under concurrent writers).
• Backfill idempotency.
• Backfill correctness (re-walked chain matches a fresh-write chain).

CLI verifier

# Text output (operator readable)
ai-memory verify-signed-events-chain

# JSON output (operator scriptable)
ai-memory verify-signed-events-chain --format json | jq

# Skip the first N rows (incremental verification)
ai-memory verify-signed-events-chain --since 100000

Exit codes:

• 0 — chain valid from start (or --since) to current tail.
• 1 — chain invalid; the JSON output identifies the first offending row’s sequence and the report’s signature_failures list contains any rows whose Ed25519 signature did not verify against the supplied key set.

JSON shape (src/cli/verify_signed_events.rs:57):

{
  "rows_checked": 142387,
  "chain_break": null,           // i64 sequence of first break, or null
  "signature_failures": [],      // list of sequences that failed sig verify
  "chain_holds": true
}

A chain break is structurally worse than a signature failure — the chain itself is the load-bearing tamper-evidence property, and a broken chain implicates every row past the break. Per-row signature failures are surfaced separately because they are a disjoint property: the chain may still be intact even if individual signatures fail.

Pinned by tests/cli_verify_chain.rs.

Three complementary verifiers

The substrate ships three verifier surfaces, each pinning a distinct property (src/cli/verify_signed_events.rs:11-19):

Audit defense-in-depth: a successful attack must tamper with all three without leaving evidence. Cross-host portability lives in the JSONL log; daemon-local tamper-evidence lives in the SQL chain; reflection ancestry attestation lives in the per-edge signatures.

Concurrent-writer guarantee (PE-3)

signed_events writes happen on the substrate storage::insert path under a transactional wrap that the PE-3 wire-point hook elevates so the chain stays monotonic under concurrent writers. The tests/deferred_audit_soak.rs soak fires 5,000 concurrent inserts and asserts the chain walk passes afterwards. append_signed_event (src/signed_events.rs::append_signed_event) wraps in a transaction; append_signed_event_no_tx (src/signed_events.rs::append_signed_event_no_tx) is the in-an-existing-transaction variant for callers that compose with a larger write.

Append-only invariant

The application layer exposes ONE writer (append_signed_event) and ZERO mutators — there are no UPDATE signed_events or DELETE FROM signed_events statements anywhere in the production code path. Operators that need to prune (compliance retention, disk pressure) MUST do so via direct SQL with explicit awareness that they are breaking the audit chain.

The substrate intentionally does NOT add SQLite triggers enforcing append-only — trigger-based enforcement would also fire against operator-driven pruning, defeating the escape hatch. The contract is enforced at the Rust API surface; the H5 test suite asserts no UPDATE signed_events / DELETE FROM signed_events strings appear in src/ outside doc comments.

Operator workflow

1. Generate an operator keypair (ai-memory identity generate — omit --agent-id to use the NHI-hardened default id, or pass --agent-id <id> explicitly).
2. Restart the daemon. The v34 schema migration runs automatically on first start and backfills the chain from existing signed_events rows. The backfill is idempotent; no manual step required for a healthy v0.6.x → v0.7.0 upgrade.
3. Run the verifier daily (cron / systemd timer):

   ai-memory verify-signed-events-chain --format json | \
     jq -e '.chain_holds' >/dev/null || \
     pager "signed-events chain broken on $(hostname)"
   

   A non-zero exit indicates either tampering or a v0.7.0-alpha row that didn’t make it through the backfill — investigate immediately.

4. Pair with the forensic bundle (L2-5, docs/forensic-export.md) — the signed events table ships inside the bundle by default. Offline reviewers can re-verify the chain without DB access.

Chain verification recipe (end-to-end)

For a routine integrity check (post-migration, post-incident, or scheduled audit):

# 1. Read current chain tail
TAIL=$(sqlite3 "$AI_MEMORY_DB" "SELECT MAX(sequence) FROM signed_events;")
echo "Current chain tail: $TAIL"

# 2. Full walk (cold start; expensive for >1M rows)
ai-memory verify-signed-events-chain --format json > ./verify-report.json
jq '{rows_checked, chain_holds, chain_break, signature_failures}' ./verify-report.json

# 3. Incremental walk (verify only what's new since last cron tick)
LAST_VERIFIED=$(cat /var/lib/ai-memory/last-verified-seq 2>/dev/null || echo 0)
ai-memory verify-signed-events-chain --since "$LAST_VERIFIED" --format json \
  | jq -r 'if .chain_holds then "OK at \(.rows_checked) rows" else "BROKEN at \(.chain_break)" end'

# 4. On success, advance the watermark
if jq -e '.chain_holds' ./verify-report.json > /dev/null; then
  echo "$TAIL" > /var/lib/ai-memory/last-verified-seq
fi

For a forensic walk after suspected tamper (the chain returns chain_break: <N> and the operator needs to know what changed):

# 1. Identify the broken row
SUSPECT=$(jq -r .chain_break ./verify-report.json)

# 2. Dump that row and its neighbours
sqlite3 -header "$AI_MEMORY_DB" "
  SELECT sequence, id, agent_id, event_type, hex(payload_hash) AS payload_hex,
         hex(prev_hash) AS prev_hex, timestamp
  FROM signed_events
  WHERE sequence BETWEEN $SUSPECT - 2 AND $SUSPECT + 2
  ORDER BY sequence;
"

# 3. Cross-check with the JSONL audit log
grep -F "\"sequence\":${SUSPECT}" /var/log/ai-memory/audit.log

The JSONL log carries an independent prev_hash chain (per F2), so a tamper that hits both must have happened at write-time, not post-hoc — a strong signal for incident response.

Key rotation procedure

signed_events.signature is Ed25519 over the source row’s canonical-CBOR bytes; the key material is the agent’s identity keypair at <key_dir>/<agent-id>.{pub,priv} (key dir: AI_MEMORY_KEY_DIR, default platform config dir + /ai-memory/keys — src/identity/keypair.rs). Rotation:

1. Back up the old key material first (copy <key_dir>/<id>.pub + <id>.priv aside — the generator does NOT preserve them), then generate the new keypair:

   ai-memory identity generate --agent-id "<id>" --force
   

   --force is required because generate refuses to overwrite an existing keypair by default (the safe default that prevents a typo from silently destroying a daemon key). The old public key remains needed for re-verification of historical rows.

2. Restart the daemon so the new key is the active signer.
3. Verify historical chain still holds under the rotated key set:

   ai-memory verify-signed-events-chain --format json | jq .signature_failures
   

   An empty signature_failures array means the verifier knows about both keys and every row’s signature checks out.

4. Audit the rotation transition by querying the row range that straddles the rotation timestamp; both signatures should be present (old for pre-rotation rows, new for post-).
5. Destroy the old key material ONLY after the rotation window has soaked and the chain walk has confirmed chain_holds == true on multiple post-rotation runs. The old key is needed to verify pre-rotation signatures.

Important: rotation does NOT change prev_hash or sequence values. The chain integrity is independent of the signing key — a key rotation is invisible to the chain walk except via the per-row signature verification.

Audit-trail forensic analysis

When the auditor needs to reconstruct “what happened in the window 2026-05-15T08:00Z to 2026-05-15T09:00Z”:

-- 1. Pull every signed event in the window
SELECT sequence, id, agent_id, event_type, attest_level, timestamp,
       LENGTH(signature) > 0 AS signed,
       hex(substr(prev_hash, 1, 8)) AS prev_hash_prefix
FROM signed_events
WHERE timestamp BETWEEN '2026-05-15T08:00:00Z' AND '2026-05-15T09:00:00Z'
ORDER BY sequence;

# 2. Verify just that window via --since
START_SEQ=$(sqlite3 "$AI_MEMORY_DB" \
  "SELECT MIN(sequence) FROM signed_events WHERE timestamp >= '2026-05-15T08:00:00Z';")
ai-memory verify-signed-events-chain --since "$((START_SEQ - 1))" --format json

# 3. Cross-reference with the on-disk audit log for the same window
jq -c 'select(.timestamp >= "2026-05-15T08:00:00Z" and .timestamp < "2026-05-15T09:00:00Z")' \
  /var/log/ai-memory/audit.log

# 4. Diff the SQL count against the JSONL count — they MUST match
SQL_COUNT=$(sqlite3 "$AI_MEMORY_DB" \
  "SELECT COUNT(*) FROM signed_events WHERE timestamp BETWEEN '2026-05-15T08:00:00Z' AND '2026-05-15T09:00:00Z';")
JSONL_COUNT=$(jq -c 'select(.timestamp >= "2026-05-15T08:00:00Z" and .timestamp < "2026-05-15T09:00:00Z")' \
  /var/log/ai-memory/audit.log | wc -l)
echo "SQL: $SQL_COUNT, JSONL: $JSONL_COUNT"

A divergence between the two counts is high-signal: either the substrate dropped a JSONL flush (rare; F2 hardened this) or post-write tampering removed an entry from one of the two surfaces.

Tuning guidance

The signed-events substrate has very few operator knobs by design — the chain integrity property is binary and should not be tunable. The operationally-relevant choices:

Troubleshooting

Operator runbook (3am procedures)

Chain break detected at runtime. First, stop appending — suspected tamper is worse than a brief audit-write outage. Set the daemon to read-only via the runtime flag if available, otherwise stop accepting writes at the load balancer. Then:

1. Pull the broken row + neighbours via the SQL recipe.
2. Cross-reference with the JSONL log for the same sequence. If the JSONL still has the row but SQL is broken, the tamper happened in SQL after the JSONL flush — strong signal.
3. Pull the forensic bundle (docs/forensic-export.md) for the incident window. The bundle is independently re-verifiable.
4. Decision: roll back to the most recent verified chain tail (lose N rows of audit history), or fork the chain into a “post-tamper” substrate and reconcile manually. Both are operator-policy calls.
5. After remediation, run a full --since 0 walk to confirm chain_holds == true end-to-end before re-opening writes.

Key rotation went sideways — chain still walks but signatures fail on old rows. The pre-rotation key was destroyed before verification. Recover from the off-host key backup (every operator runbook should have one). If no backup exists, the affected rows are not signature-verifiable but the chain itself is still tamper- evident — the operator notes the rotation-debt in the incident log and moves on.

Partial backfill stuck in a v0.7-alpha → v0.7.0 upgrade. The COR-9 diagnostic fires and the daemon refuses appends. Re-run the schema migration by re-opening the DB with the v0.7.0 binary — any CLI command that opens the DB (e.g. ai-memory stats) triggers the automatic current_version → apply_migrations ladder. The migrate_v34_backfill_chain function is idempotent — re-running is safe. After it completes, verify-signed-events-chain --format json should report chain_holds: true over the full row count. (Note: the ai-memory migrate subcommand is the sal-gated store-to-store migrator — --from/--to URLs — not a schema-version tool.)

Audit log + SQL chain divergence. Treat as high-signal. Pull both logs for the divergence window, file an incident, and roll out the forensic-export bundle to an offline reviewer. The substrate’s defense-in-depth design assumes any single surface can be tampered — it’s the agreement between SQL chain + JSONL log + per-edge signatures that pins the audit story.

Mapping to the V-4 audit

• V-4 (cross-row hash chain) PASS — pinned by tests/signed_events_chain_v34.rs.
• V-4 (per-row signature) PASS — unchanged from v0.6.3.
• V-4 (append-only enforcement) PASS — no UPDATE / DELETE path through the application layer; the invariant is pinned by the append_only_invariant_no_mutators_in_src test in src/signed_events.rs, which fails the build if any production UPDATE/DELETE call site against signed_events appears.

See also: docs/MIGRATION_v0.7.md §”Ed25519 attestation”, docs/v0.7.0/release-notes.md §”Signed events V-4 closeout”, the canonical inventory in docs/internal/v070-feature-inventory.md §”Feature: Signed events V-4 closeout”, the federation hardening layer that produces peer-write events on this chain at docs/federation.md, the K10 approvals path whose decisions are recorded as signed_events rows at docs/k10-sse-approvals.md, the forensic-bundle exporter at docs/forensic-export.md, the hook pipeline whose gated writes generate signed-event rows at docs/hook-pipeline.md, the K8 quotas substrate whose refusals are also audit events at docs/k8-quotas.md, and the sidechain transcripts whose store-events appear in the chain at docs/sidechain-transcripts.md.