ai-memory v0.8.0

Recursive learning (v0.7.0)

Substrate-vs-application boundary (Item B, issue #973)

Ortega & de Freitas (2026) “delusion amplification” is a training-layer result: an LLM under supervised fine-tuning on its own past outputs accumulates miscalibrated beliefs because SFT doesn’t carry the intervention/observation distinction from the do-calculus. ai-memory operates one layer up, at the storage substrate. The honest claim follows directly:

ai-memory’s substrate-level Form-4 fact-provenance + 7-level Provenance Gap framework + Form-7 agent-EXTERNAL governance gate stops cross-session delusion amplification by preserving the intervention/observation distinction in the storage layer (citations vs agent self-claim; MemoryKind::Observation vs Reflection vs Claim; AgentAction-gated writes typed at the policy engine). It does not stop intra-session hallucination — that’s the consumer LLM’s training-layer + decoding-layer problem, outside the substrate’s contract.

A second axis: the substrate’s evidence claim depends on federation reliability as much as on cryptographic attestation. v48 schema added the federation_push_dlq table (#933) so federation broadcast failures land in a per-peer DLQ with a retry-replay worker + Prometheus federation_push_dlq_depth gauge. A signed event chain that drops messages silently across the mesh is not the same as one that delivers reliably with DLQ-tracked failures. DLQ is observable + replayable, not hidden. A consumer that needs to claim “this agent’s reflection chain is forensically defensible across the federation” must inspect DLQ depth + replay history alongside the cryptographic V-4 hash chain (#698) — both are wire-format authoritative.

In short: the substrate gives a clean intervention/observation cut at write-time, a recall-consumption ledger at read-time (Gap 3), DLQ-tracked federation reliability at multi-node-time, and a signed hash chain at audit-time. What an LLM agent does between calls to the substrate (intra-session reasoning) is its own responsibility. See docs/rationale/academic-context.md for the procurement- audience version.

The procurement-grade requirements for these layers in high-assurance MCP deployments are enumerated in the National Security Agency’s Cybersecurity Information document on Model Context Protocol security (U/OO/6030316-26, May 2026). Substrate compliance with those requirements is documented at docs/compliance/nsa-csi-mcp-security-mapping.md (10 of 10 NSA concerns + 7 of 7 NSA recommendations structurally addressed at v0.7.0); the honest-limitations framing for what the substrate does NOT defend against is at docs/compliance/honest-limitations.md.


Status (2026-05-14): The recursive-learning grand-slam ships in v0.7.0. Tasks 1-8 of the original recursive-learning add-on (issue #655) plus the L1 substrate stack (#666-680) and the L2 wave (#666-#673) all land on feat/v0.7.0-grand-slam and roll up into the v0.7.0 tag. The L2 wave extends the substrate primitive into a curator mode, federation-aware coordination, invalidation propagation, transcript replay union, forensic bundles, reflection-as-skill promotion, skill composition, and a reflection-aware reranker boost — every claim on this page maps to shipped code at commit c359e89.

ai-memory v0.7.0 ships a substrate-native primitive for recursive refinement: an agent reads one or more memories, synthesises a higher-order reflection (a lesson, pattern, contradiction-resolution, etc.), and persists it with cryptographic-grade provenance back to each source it reflects on. Reflection depth is bounded by a substrate-enforced cap. No autonomous goal modification, no model fine-tuning loops, no unbounded recursion — the substrate refuses runaway recursion before any write opens.

This page is the engineering-precise primer. The procurement-level introduction lives in v0.7.0/release-notes.md. The CHANGELOG block sits under v0.7.0 in ../CHANGELOG.md. The category-level comparison against other agent-memory projects (mem0, Letta, Tencent TencentDB Agent Memory, etc.) lives in positioning.md.

Conceptual model

A reflection is a higher-order memory whose reflection_depth is strictly greater than zero and whose reflects_on provenance links point back to one or more lower-depth source memories. The reflection row is just another memory — the same recall, search, governance, federation, attestation, and audit primitives apply to it. What’s new is the recursion contract:

A reflection is provenance-pinned, not just provenance-claimed. The reflects_on edges are the cryptographic-grade link — when the v0.7.0 H-track Ed25519 keypair is configured, the link is signed and verifiable via memory_verify. A downstream auditor can walk the reflection chain and re-verify every step.

The depth cap

Reflection depth is substrate-enforced, not application-enforced. db::reflect (and the postgres parity PostgresStore::reflect) consult GovernancePolicy.max_reflection_depth for the resolved namespace before opening the write transaction. If the proposed depth exceeds the cap, the substrate refuses with a structured error — no partial write, no autonomous escape hatch.

Why 3?

The compiled default is 3. It bounds reflection recursion without strangling the legitimate reflection-on-reflection chains the v0.8.0 Pillar 2.5 curator mode is being designed against. Operators who want a different global default change the constant at the effective_max_reflection_depth accessor; per-namespace overrides ride the same JSON governance blob resolve_governance_policy already walks leaf-first.

Per-namespace override

max_reflection_depth: Option<u32> (a CorePolicy field, surfaced through GovernancePolicy) is a pure JSON metadata field on the standard-memory governance object. No schema bump — it rides alongside the existing write, promote, delete, approver, and inherit fields. The accessor is flat:

pub fn effective_max_reflection_depth(&self) -> u32 {
    self.core.max_reflection_depth.unwrap_or(3)
}

Resolution is leaf-first via resolve_governance_policy (same path the rest of the K1/G1 governance enforcement walks), so a child namespace’s max_reflection_depth = None falls through to the nearest ancestor that does set it, and finally to the compiled default 3.

Some(0) is the kill-switch

The substrate-side check is attempted > cap, not attempted >= cap. That means cap = 0 refuses every reflection — depth 1 already exceeds 0, depth 2 exceeds it, and so on. Some(0) is the documented kill-switch for a namespace (or subtree) that should never accept reflection writes. Set it on a namespace whose contents must remain caller-minted and you have a per-namespace opt-out from the entire primitive.

API surfaces

Surface Where Notes
memories.reflection_depth INTEGER NOT NULL DEFAULT 0 SQLite schema v29 (src/storage/migrations.rs; pre-#961 was src/db.rs); Postgres schema v31 (src/store/postgres_schema.sql, migrations/postgres/0013_v0700_reflection_depth.sql) Caller-minted rows are 0; reflections are max(source_depths) + 1. UPSERT clauses take MAX(old, new) so federation merges preserve the higher-depth signal.
Memory::reflection_depth: i32 src/models/memory.rs #[serde(default)] keeps wire-compat with pre-v0.7.0 federation peers. impl Default for Memory ships in the same commit so future struct-field adds stop fanning out to ~50 test fixtures.
CorePolicy::max_reflection_depth: Option<u32> (read via GovernancePolicy) src/models/namespace.rs Per-namespace cap. None → compiled default 3. Some(0) → kill-switch.
GovernancePolicy::effective_max_reflection_depth(&self) -> u32 src/models/namespace.rs Flat accessor. Does NOT walk ancestors — call resolve_governance_policy first, then this accessor on the result.
reflects_on relation src/validate.rs (VALID_RELATIONS); MCP relation handling in src/mcp/tools/link.rs (post-#1066 split; was src/mcp.rs pre-split — there is no separate unlink tool/module); claude_help prompt pipe-list No schema migration required. memory_links.relation has no CHECK (relation IN ...) clause on either adapter — adding a label is a pure validator + documentation change.
memory_reflect MCP tool src/mcp/tools/reflect.rs (post-#1066 + #987 D1.6 split); substrate impl reflect in src/storage/reflect.rs (post-#961 SAL boundary cleanup; db::reflect is the back-compat alias); postgres parity PostgresStore::reflect in src/store/postgres.rs Family::Power. Tool count 51 → 52. Atomic insert + N reflects_on link writes inside a single BEGIN IMMEDIATE / COMMIT block (SQLite) or sqlx::Transaction (Postgres).
MemoryError::ReflectionDepthExceeded { attempted: u32, cap: u32, namespace: String } src/errors.rs HTTP 409 CONFLICT, code REFLECTION_DEPTH_EXCEEDED. The structured triple is what downstream auditors and hook emitters need without parsing error strings.

Directionality contract for reflects_on

The reflection memory is the link’s source_id; the original being reflected on is the link’s target_id. This matches the existing derived_from convention:

reflection_memory  --reflects_on-->  original_source
(reflection_depth = N)               (reflection_depth = N-1 or less)
   ^^ link.source_id                    ^^ link.target_id

The arrow points from the newer/derived row to the original. A consolidated memory derived_from its sources is the same shape — the derived row is on the left, the source on the right. Operators tracing reflection provenance walk edges outward from the reflection to find its sources, exactly as they walk edges outward from a consolidated memory to find the inputs that produced it.

find_paths chain-walk behaviour

db::find_paths’s recursive CTE projects every edge in memory_links without filtering by relation label. That means reflects_on edges auto-participate in chain walks alongside the other relations — operators tracing reflection provenance see chains surface naturally without further work. The Task 3 regression test (tests/recursive_learning_task3_reflects_on.rs:: sqlite_find_paths_walks_reflects_on_edges) pins this behaviour against a 3-hop chain.

When walking a reflection chain, expect the path to alternate between memories that were caller-minted (reflection_depth = 0) and their reflections (reflection_depth > 0). A reflection of a reflection of a reflection is a 3-edge reflects_on chain whose terminal nodes carry depths 0, 1, 2, 3 from leaf to root.

Reproducibility

The end-to-end demo script is scripts/reproduce-recursive-learning.sh. It builds the release binary, creates a fresh sqlite DB under .local-runs/repro-recursive-learning-<timestamp>/, inserts three sample memories, calls memory_reflect to produce a reflection at depth=1, recursively reflects up to depth=3 (the default cap), and demonstrates the refusal at depth=4 with a clearly-formatted REFLECTION_DEPTH_EXCEEDED verdict block. Idempotent on re-run (each invocation uses a fresh timestamped subdirectory).

The script honors the project no-/tmp HARD RULE — all scratch lives under .local-runs/, which is gitignored.

Audit record on depth-cap refusal

Landed in v0.7.0 (Task 5/8, commit c61a05b).

Every db::reflect call that would exceed the namespace’s resolved max_reflection_depth appends a row to the append-only signed_events audit table before the cap refusal propagates back to the caller. The row carries:

PII guarantee. The reflection’s content body is deliberately omitted from the audit payload. The cap-refusal audit captures only the enumerable provenance the refusal needed to make its decision — the proposed title is human-readable but the body is not. A caller that placed PII in content and tripped the cap therefore does not leak that body into the audit chain.

Best-effort write semantics. Audit-row insertion is best-effort: on insertion failure (disk full, lock contention, table corruption), the substrate logs at WARN via tracing::warn!(target: "signed_events", ...) but the cap refusal still propagates to the caller with the same ReflectError::DepthExceeded shape. The wire contract is unchanged by audit-write success/failure — operators verify the audit chain via ai-memory verify-signed-events-chain CLI (there is no HTTP endpoint for the signed_events surface at v0.7.0) to reconcile gaps against the daemon’s signed_events warn-log target rather than the caller observing a different error.

Hook integration

Landed in v0.7.0 (Task 6/8, commit fbf093c).

The Track-G hook pipeline grows from 21 to 23 events with two new HookEvent variants for the reflection primitive:

The hook decision surface is the narrow ReflectHookDecision enum (post-#961 SAL boundary cleanup relocated the type from src/db.rs to src/storage/reflect.rs; db::ReflectHookDecision remains a back-compat alias):

pub enum ReflectHookDecision {
    /// Continue with the reflection unchanged. Default decision.
    Allow,
    /// Reject the reflection. Propagates as
    /// `ReflectError::HookVeto { reason, code }` distinct from the
    /// Task 5 substrate cap refusal so callers can disambiguate
    /// caller-policy refusals from substrate-policy refusals.
    Deny { reason: String, code: u16 },
}

Returning Deny from a PreReflect handler short-circuits the reflection and propagates as ReflectError::HookVeto, which surfaces on the wire as "REFLECTION_HOOK_VETO (code=<N>): <reason>". Notify-class PostReflect handlers cannot veto — their return value is ignored beyond logging.

Explicit non-interaction with the Task 5 audit. A PreReflect hook veto does not emit a Task 5 reflection.depth_exceeded audit row. The Task 5 row is the substrate’s tamper-evident record that the substrate refused the reflection on cap grounds. Caller-policy refusals (hook vetoes) carry their own provenance via the hook’s own audit channel — conflating them with substrate-cap refusals would dilute the cap-refusal audit signal and mis-attribute the refusal source.

Curator mode — Pattern 4 (L2-1)

Landed in v0.7.0 (L2-1, commit c3f6e82, issue #666).

The substrate primitive (memory_reflect) is one synchronous write per caller-driven reflection. The reflection-pass curator is the asynchronous orchestrator that walks the namespace, clusters Observation-kind memories by namespace + temporal proximity + recall co-occurrence proxy, asks the configured LLM to summarise the pattern, and persists each summary as a typed MemoryKind::Reflection through the same substrate path (storage::reflect_with_hooks). One level of reflection per pass; multi-level chains form naturally over repeated passes when the namespace governance max_reflection_depth permits.

Key contracts (src/curator/reflection_pass.rs):

Operator-facing surface lives in ai-memory curator --reflect (src/cli/curator.rs). Operational runbook sits at docs/RUNBOOK-curator-soak.md.

Federation behavior (L2-2)

Landed in v0.7.0 (L2-2, commit 0b1c9cc, issue #667).

Reflection-row federation is governed by local territorial sovereignty over depth. A peer cannot launder depth across hosts by syncing a depth-N reflection into a stricter receiver.

src/federation/reflection_bookkeeping.rs guarantees three behaviours on the receive path:

  1. Origin stamping. Every inbound reflection memory gets metadata.reflection_origin = { peer_origin, original_depth, local_depth_at_arrival } stamped on import. peer_origin is the substrate identity of the peer that pushed us the row; signing_agent is the original author (preserved across hops via metadata.agent_id); original_depth is the wire-truth depth as delivered; local_depth_at_arrival is the receiver’s effective cap at the moment of arrival (so an after-the-fact tightening of the cap is visible on every imported row).
  2. Derived-write enforcement. A NEW reflection derived locally from one or more imported rows is checked against the LOCAL cap regardless of the source peers’ caps. Cross-peer chain extension cannot launder depth.
  3. Inspection surface. The MCP tool memory_reflection_origin (tool count bump: 60 → 61 in this wave) answers “where did this reflection come from?” for any memory id, returning the structured {memory_id, peer_origin, signing_agent, original_depth, local_depth_at_arrival, is_reflection} envelope.

Depth on the column is preserved across federation — we never silently rewrite incoming depth values. Enforcement happens on write-time decisions about derived rows, not on import.

Invalidation propagation (L2-3)

Landed in v0.7.0 (L2-3, commit 3f419be, issue #668).

When a Reflection-kind memory is superseded by another reflection (i.e. a Reflection → Reflection supersedes edge lands via memory_link), the substrate fires propagate_reflection_invalidation. For every memory whose reflects_on edge points at the now-superseded reflection, the substrate writes one notification memory under <dependent.namespace>/_invalidations carrying:

The wave is notification, NOT cascade. Dependents are flagged for operator/curator review, never auto-superseded. The substrate refuses to mutate caller-visible rows under any invalidation pathway — operator review remains the only path for promoting an invalidation into an actual supersession.

Read-only inspection lives at the MCP memory_dependents_of_invalidated tool (tool count bump 61 → 62 in this wave). The tool returns {memory_id, count, dependents: [{id, namespace}]} without firing the walker; the walker only fires from the memory_link handler on the Reflection→Reflection supersedes path.

Reranker boost (L2-8)

Landed in v0.7.0 (L2-8, commit 90291c0, issue #673).

Reflections are higher-information rows than the observations they generalise over. The recall pipeline acknowledges that with a reflection-aware reranker boost applied AFTER the cross-encoder blend (see src/reranker.rs):

per_depth_factor = 1.0 + per_depth_increment * min(reflection_depth, max_depth_cap)
final_score      = base_score * (kind == Reflection ? boost * per_depth_factor : 1.0)

The defaults are pinned in ReflectionBoostConfig:

Field Default Behaviour
boost 1.2 Multiplicative boost for Reflection-kind rows. 1.0 is the documented kill-switch (reproduces pre-L2-8 ranking exactly).
per_depth_increment 0.05 Additional multiplier per depth level.
max_depth_cap 3 Mirrors effective_max_reflection_depth. Deeper rows clamp to this cap; the multiplier is bounded.

The boost is opt-in at the daemon level — set via reflection_boost = { boost, per_depth_increment, max_depth_cap } in config.toml or the equivalent capabilities-fed runtime config. A boost = 1.0 config is reported honestly in capabilities as “no ranking change” so operators can verify the kill-switch took effect.

Reflection-as-skill (L2-6, closing the loop)

Landed in v0.7.0 (L2-6, commit 505c538, issue #671).

The closing-loop primitive of the grand slam: a Reflection-kind memory at depth ≥ 1 can be promoted to a reusable Agent Skill via the MCP tool memory_skill_promote_from_reflection. The substrate constructs an agentskills.io-compliant SKILL.md whose frontmatter carries:

Promotion refuses depth-0 reflections (no synthesised insight to promote) and refuses depths below namespace.governance.skill_promotion_min_depth (default 1). The round-trip digest guarantee holds: promote → export → re-register produces the IDENTICAL SHA-256 digest as the in-DB row — the lineage is preserved cryptographically across promotion and re-registration.

Full surface is documented in docs/agent-skills.md; this section pins the substrate-side contract for the reflection ↔ skill bridge.

Forensic export

The forensic-bundle and verify-reflection-chain surfaces are documented in docs/forensic-export.md. Both are the procurement-grade audit path for reflection chains: a single tar an external auditor can re-verify with no daemon state, just the public keys of the signing agents.

File-backed export

v0.7.0 QW-1 ships ai-memory export-reflections — a CLI subcommand that walks every reflection memory under a namespace and writes a YAML-frontmatter markdown file per row to ~/.ai-memory/reflections/<namespace>/<id>.md. Operators can cat or grep the directory to inspect the reflection chain without opening a sqlite3 shell. The on-disk artefact is derived — the SQL row stays canonical; the directory is safe to delete and regenerate.

# Bulk export, operator-driven.
ai-memory export-reflections \
    --namespace team/alpha \
    --out-dir ~/.ai-memory/reflections \
    --format md \
    --since 2026-05-01T00:00:00Z

cat ~/.ai-memory/reflections/team/alpha/<id>.md
grep -c reflects_on: ~/.ai-memory/reflections/team/alpha/*.md

Frontmatter fields (in this order): memory_id, namespace, title, reflection_depth, attest_level (highest attestation level across the row’s outbound reflects_on edges), created_at, agent_id, then a sequence-shaped reflects_on block listing target ids + per-edge attest levels. Body is the reflection’s content field, untouched.

The companion MCP tool memory_export_reflection returns the rendered content + a suggested filename without touching the filesystem — the agent harness owns disk I/O so the substrate stays under the operator’s capability gate. Symmetric with memory_skill_export (L1-5).

Per-namespace auto-export: setting governance.auto_export_reflections_to_filesystem: true on a namespace standard installs a post_reflect substrate hook that deferred-spawns the disk write inside the substrate process the moment a reflection commits. The hook is non-blocking (worker thread + own connection) and notify-class (failure logs, never propagates to the caller). Default is false — the substrate is SQL-canonical out of the box; opt-in per-namespace via the same G1 inheritance walk every other policy field uses.

See cookbook/file-backed-export/01-export-and-inspect.sh for a runnable demo (5 reflections, export, cat, grep — reproducible in under 3 minutes).

Substrate authority claim — v0.7.0 Option B foundation

v0.7.0 ships Option B of the substrate-authority programme: the L1-6 substrate rules-enforcement engine (src/governance/rules_store.rs, src/governance/agent_action.rs, issue #693) ships un-wired to the live write path by default — it is the operator-keypair-signed rule store, the bypass-impossibility test fleet, and the check_agent_action enforcement helper, and it runs on storage::insert as a governance pre-write hook (L1-6 Deliverable E, #691). The end-to-end “100% of write paths go through the substrate” coverage is a separate v0.8.0 epic (#697) that wires the rule engine into every adapter write path with the full bypass-impossibility surface.

What the substrate DOES today (v0.7.0):

What the substrate does NOT yet do (v0.7.0 → v0.8.0 #697):

The audit-honest framing: substrate authority is a foundation in v0.7.0, a complete cover in v0.8.0. Operators evaluating the authority claim today should read this section, the v0.7.0 release notes, and #697 together — and treat any “100% substrate authority” marketing that elides the wiring gap as inaccurate.

Forward roadmap

Cross-references