ai-memory uses the tracing crate with an EnvFilter-driven subscriber. 579 log call sites across 95 source files / 21 directories at v0.7.0 (was ~176 at v0.6.x; growth from the Wave-2 SAL split, the L1-5 Agent Skills surface, the K7 subscription-reliability drainer, the K11 governance-rules engine, the Form-4/5 provenance + confidence pipelines, and the recursive-learning + atomisation hooks). Every error path is named; every fanout failure is logged with the peer id; every governance verdict is loud. AI clients can grep the canonical lines below to reconstruct what the daemon was doing.
The daemon installs a tracing_subscriber::fmt layer with the standard EnvFilter chain. Default directives keep ai_memory + tower_http at info while honoring any RUST_LOG override.
To turn the volume up:
Named metadata targets (post-#1562). Most events emit under the module-path default, so an ai_memory=… directive matches them (including the named ai_memory::-prefixed targets: ai_memory::subscriptions, ai_memory::authz, ai_memory::handlers::skills, ai_memory::federation::sync, ai_memory::federation::push_dlq, ai_memory::governance::audit). But a set of subsystems emit under literal targets that do not start with ai_memory — an ai_memory=debug filter does NOT match these events; name them explicitly:
Counts pulled from grep -rnE '^\s*(tracing::)?(error|warn|info|debug|trace)!\s*\(' src/. Handlers, federation, and the SAL adapter family dominate — they are where user-visible behavior happens.
Every HTTP/MCP write goes through one of these handlers (modular split since v0.7 Wave-1: http.rs, federation_receive.rs, hook_subscribers.rs, transport.rs). Errors get an error!("handler error: {e}") and the response code; governance verdicts get an error!("governance error: {e}").
Boot + MCP dispatch + LLM client + subscriptions + reranker + curator. Mostly info! startup banners + warn-level fallback notices + the Ollama / OpenAI-compatible client error envelopes.
Quorum fanout (W of N) is the noisiest area. Every peer success is silent; every peer failure logs warn!("federation: peer {peer_id} failed for {id}: {reason}") so the operator can see where the network dropped traffic.
SAL adapter family — sqlite.rs + postgres.rs (Apache AGE Cypher). Logs adapter-bound issues, schema-migration ladder progress, and the postgres-route-gate verdicts on missing trait coverage.
SQLite SQL primitives + migrations.rs (v2→v57 ladder). Migration logs, FTS5 index rebuild events, vacuum completion. Most storage::* calls are silent on success.
MCP protocol layer + the 74-entry registry (15 calls under src/mcp/tools, 21 under src/mcp/tools/store alone). Logs malformed JSON-RPC, unknown tool names, method-not-found, and per-tool dispatch errors.
Pre-store / post-reflect hook pipeline (14 calls under src/hooks/pre_store; 19 across the rest). Logs hook chain failures, retry decisions, and the v0.7.0 auto-export panic-recovery path.
K9 permissions gate + K11 substrate rules engine. Logs every Deny verdict, every fail-CLOSED rule-consultation error, and every signed-rule cache invalidation.
CLI subcommand handlers (80 top-level subcommands in the default build at v0.7.x post-#1598 reembed; 82 under --features sal). Logs config-migration outcomes, schema-init progress, doctor reachability probes.
Sidechain transcripts + I2 transcript-link substrate. Logs zstd-3 compression boundaries and the auto-extract policy decisions.
Agent-id resolution chain. Logs precedence-chain fall-throughs and the anonymous-fallback path.
WT-1 atomisation engine + curator cycle. Logs LlmCurator scaffolding and cycle-skip rationale.
QW-3 context-offload (TTL sweep), Form-5 shadow observations, K10 SSE approvals + inbox notifications.
42 of the 579 call sites are debug!/trace! diagnostics (federation sync, postgres adapter, hooks executor, migrations) — silent under the default ai_memory=info filter until RUST_LOG raises them. Default production output is the three levels below.
Bootstrap milestones, model pulls completing, catchup cycle summary. Quiet by default — doesn't drown the operator.
Pulling Ollama embedding model 'nomic-embed-text'... Embedding model 'nomic-embed-text' pulled successfully catchup: pulled 142 memories from 3 peers in 2.1sDaemon kept running, request still succeeded (or fell back). Operator should look but no incident yet.
embedding generation failed: connection refused recall: embedder query failed, falling back to keyword-only: timeout SSRF guard rejected webhook URL https://10.0.0.1: private network federation: peer node-3 failed for abc-123: connect timeout register_agent fanout error (local committed): node-2 502The HTTP/MCP response is also an error. Logs the same condition the client gets back so server-side reconstruction is possible from logs alone.
handler error: validation failed: title cannot be empty governance error: agent not registered execute pending error: target memory not found detect_contradictions list error: db locked export error: tar archive write failedWhen an AI client is reading logs to figure out what happened, these are the lines to look for. Each pattern matches an exact tracing::*! call site in the source.
anyhow chain. Validators, db, governance, and federation all bubble into this single phrase. The HTTP response is 4xx/5xx; the log carries the full reason.
Pending verdict is silent — only Deny and replay failures log. Look for these when an AI client says "the write was rejected and I don't know why."
federation: peer {peer_id} failed for {memory_id}: {reason}. Quorum still completes if W of N succeed; this line tells you which N-W peers were missing.
register_agent, link, consolidate. The local row is durable; sync_pull from those peers will reconcile.
SSRF guard rejected webhook URL {url}: {reason}. Reasons include "private network," "loopback resolved," "scheme not http/https." Two SSRF defects were closed in commit 9eeb453 for v0.6.3.
unparseable memory from peer) when a peer ships malformed data — those rows are skipped, not retried.
approved but replay couldn't complete (deleted target, namespace gone, validator now rejects). Status remains approved; the row is loud in logs but the user-visible state is unchanged.
webhook POST to {url} failed: {error}. Subscription-side; not retried at the daemon (the recipient is responsible for its own retry policy).
v0.7.0 shipped the Hook Pipeline (arch-spec §2) and Sidechain Transcripts (§6). Both extended the trace surface:
info_span!("mcp_tool_call", tool=..., rpc_id=...) wraps every MCP tool dispatch (src/mcp/mod.rs) so production observability can attribute latency per toolsrc/transcripts/, transcripts.lifecycle trace target), queryable independent of the daemon loghooks.registered_count, transcripts.total_count, etc. (src/mcp/tools/capabilities.rs) so AI clients can answer "what's enforced here?" honestlytracing-opentelemetry bridge so traces can ship to a collector