A first-of-its-kind multi-evaluator, model-heterogeneous assessment of ai-memory v0.7.0 by three frontier AI NHI agents — Anthropic Claude Opus 4.7, OpenAI GPT 5.5, xAI Grok 4.3 — running the same prompt in isolation, then synthesised. The decorrelated-errors property across three model families with different training distributions, RLHF lineage, and architectural priors is the entire methodological point. Same-model reflection re-introduces the echo-chamber problem this assessment exists to surface.
ai-memory v0.7.0 makes the reflection boundary LLM-agnostic by design — same row shape, same signature primitives, same audit chain whether the curator is Opus, Gemma, GPT, or Grok. This assessment lifts that property up to the assessment layer itself: same prompt, three model families, no cross-talk during Phase 1, then orchestrator synthesis in Phase 2.
This is bias-detection-by-architecture, not redundant tooling, for four reasons:
“Opus reflects on Opus” is NOT substrate-equivalent to “Grok reflects on Opus.” The substrate is the same; the cognitive operation is fundamentally different. Any leakage between evaluators during Phase 1 collapses three decorrelated samples into one correlated sample.
Each evaluator runs the full 22-probe matrix against its own isolated workspace under .local-runs/v070-nhi-assessment-<ts>-<evaluator>/ with no cross-talk to the other two reports during Phase 1.
Model: claude-opus-4-7[1m] (1M-context variant). Probe execution: 2026-05-24, refined 2026-05-25.
Model: gpt-5 (canonical alias openai backend). Awaiting independent Phase-1 execution.
Model: grok-4.3 (canonical alias xai backend, https://api.x.ai/v1). Awaiting independent Phase-1 execution.
After all three Phase-1 reports land, an orchestrator pass (Claude Opus 4.7 in synthesizer-role, against all three reports as input) produces the cross-evaluator synthesis at synthesis.md with three load-bearing outputs:
| Phase | Who | What | Cross-talk |
|---|---|---|---|
| 1 — Isolated execution | Each evaluator independently | Run the full 22-probe matrix against the evaluator's own .local-runs/ workspace. Write report-<evaluator>.md. 90-120 min per evaluator. | NONE. Reading another evaluator's report during Phase 1 collapses the decorrelated-errors property. |
| 2 — Orchestrator synthesis | Claude Opus 4.7 (synthesizer-role) | Read all three Phase-1 reports as input. Produce synthesis.md with agreement / disagreement / bias-detection sections. | Allowed — Phase 2 IS the cross-talk, but it's structured. |
| 3 — Operator review | Justin Jessup, AlphaOne LLC | Review the synthesis. Resolve disagreements via (a) re-probing the contested primitive, (b) operator override with rationale, or (c) filing as known cross-model uncertainty in v0.7.0 release notes. | n/a — biologic operator gates. |
The assessment prompt (prompt.md) defines a 22-probe matrix across five tiers. Each probe names a concrete file path in release/v0.7.0 @ 1cb95bbe7 and a question the evaluator must answer from direct examination, not training-prior recall.
| Tier | Probes | Focus |
|---|---|---|
| T1 — Substrate primitives | P1–P5 | Memory struct shape, link variants, schema migration ladder, recall pipeline, agent identity |
| T2 — Governance | P6–P10 | L1-6 rule engine, Ed25519 attestation, signed-events V-4 chain, namespace inheritance, permissions enforce-mode |
| T3 — Substrate interfaces | P11–P15 | MCP tool surface (74 advertised), HTTP route registry (88 routes / 74 unique paths), CLI subcommand table (79), config schema v2 sectioning, env-var precedence ladder |
| T4 — AI-NHI primitives | P16–P19 | Recursive-learning depth gate, atomisation engine, persona-as-artifact, context-offload TTL sweep |
| T5 — Operational hardening | P20–P22 | SSRF fail-CLOSED, governance fail-CLOSED, postgres + Apache AGE backend parity |
Each probe has a documented expected verdict and a documented bias-surface (the angle on which an evaluator's training distribution is most likely to inject a model-specific reading). The synthesis pass attends to whether evaluators converged on the same evidence-anchored verdict or diverged along the predicted bias-surface.
The Phase-1 Opus 4.7 report (full text →) returned SHIP-WITH-CAVEATS on 2026-05-24 morning, refined the same day with two of the caveats closed by filed PRs and three new follow-up issues opened. Surface findings:
memory_reflect MCP tool strips caller-supplied metadata.entity_id. Closure PR #1177 with 5-invariant regression suite.memory_capabilities.models.embedding_dim sources from tier preset instead of canonical KNOWN_EMBEDDING_DIMS table. Closure PR #1178 with 13-invariant regression suite covering known / unknown / tier-disabled / legacy-alias paths.local / global / const discipline (pm-v3.1 prime-directive addendum). Phase-1 inventory subagent audited 266 Rust source files + 233 integration tests and produced a 98-finding (37 MUST + 54 SHOULD + 7 NICE) report.source: "claude" on every reflection regardless of which AI NHI made the call. The substrate is heterogeneous-NHI by design; that hardcode is a monoculture defect. The hardcode was removed by commit 84ec159c1 per #1181; current source-stamp site uses DEFAULT_NHI_SOURCE from src/validate.rs:85.pending_action payload drops metadata on the pending→execute round-trip. Sibling defect to #1172 via a different code path.The Phase-1 Opus 4.7 finding that the substrate is the cognitive-checks-and-balances primitive for AI NHI — specifically the load-bearing claim that ai-memory is the externalised site of NHI cognition-across-time, with frozen weights + ephemeral processes as the load-bearing constraints rather than pathologies — is consistent with the substrate's own marketing and design rationale. Cross-evaluator confirmation of this framing (or principled disagreement with it) is one of the highest-stakes outputs of the eventual synthesis pass.