Gemma 4 MTP — Empirical Bench (2026-05-17)

Reproducibility doc for the v0.7.0 / RFC #651 inference-backend evaluation. Pins a one-day measurement on an M4 Mac Mini at the exact Ollama/model versions present 2026-05-17 so a future agent can re-run and compare.

TL;DR

On gemma4:e4b Q4_K_M (the model ai-memory’s autonomous tier uses), setting OLLAMA_MLX_MTP_MAX_DRAFT_TOKENS=4 produces zero measurable speedup. Decode rate stays at 31 tok/sec, p50 wall stays at 2.9 s for the standard 80-token completion the curator’s auto-tag / contradiction passes drive. MTP needs the separately- published draft model (google/gemma-4-E4B-it-assistant) to have anything to speculate against, and that model ships only as HF safetensors — Ollama’s llama.cpp pull path returns 400: Repository is not GGUF or is not compatible with llama.cpp.

Until a GGUF-converted drafter lands (Ollama library, llama.cpp convert_hf_to_gguf.py, or a community port), MTP gating on this host is a no-op.

Test environment

Reproduction

# Path A — Direct Ollama latency bench (no curator)
cd /Users/fate/v07/v07-fixes/.local-runs/mtp-bench-2026-05-17
./bench.sh baseline-pre-mtp        # 20 calls, captures total/eval/prompt durations as JSONL

# Restart Ollama with MTP env (kills the .app's auto-spawned serve)
pkill -TERM -f 'Ollama|ollama serve'
sleep 4
nohup env OLLAMA_MLX_MTP_MAX_DRAFT_TOKENS=4 \
          OLLAMA_MODELS=/Users/fate/.ollama/models \
          /Applications/Ollama.app/Contents/Resources/ollama serve \
          > .local-runs/mtp-bench-2026-05-17/ollama-serve-mtp.log 2>&1 &
disown
sleep 4
ps eww -p "$(pgrep -f 'ollama serve' | head -1)" | tr ' ' '\n' | grep MTP   # confirm env

./bench.sh post-mtp                # same 20 calls under MTP

Both runs write <label>.jsonl (per-call timings) and <label>.summary.txt (model card + start/end wall times) to the same dir. Compare via:

jq -s '{
  calls: length,
  decode_tokens_per_sec: (([.[].eval_count] | add) * 1e9 / ([.[].eval_duration] | add)),
  p50_wall_ms: (sort_by(.wall_ns)[length/2|floor].wall_ns / 1000000),
  warm_avg_skip_outliers_ms: (sort_by(.wall_ns)[1:-1] | (map(.wall_ns) | add) / length / 1000000)
}' baseline-pre-mtp.jsonl
# repeat for post-mtp.jsonl

Numbers (this run, 2026-05-17 11:48–11:59 UTC)

                           Baseline (no MTP)     MTP enabled         Δ
decode tokens/sec          31.35                 31.01               −1% (noise)
p50 wall (ms)              2897                  2884                −0.4%
warm-avg skip outliers     2911                  2991                +2.7%
total wall 20 calls (ms)   56401                 68476               +21% (one 11.7 s outlier in MTP run)

1542 output tokens generated each side, same prompt set (verbatim from bench.sh). The 11.7 s spike on the 20th MTP-run call was a model unload/swap, unrelated to MTP behavior. Removing it pulls the warm averages closer (≤3% wall delta, ≤1% tok/sec delta — all within single-run noise).

End-to-end curator cycle delta

Measured cycle wall times from _curator/reports:

created_at                            dur_ms   pg  tagged contrad errs   notes
2026-05-17T12:01:24+00:00             98904    0   0      0       3      POST-MTP, new ai-memory daemon binary (PR #820)
2026-05-17T11:53:59+00:00             97841    0   0      0       3      pre-MTP
2026-05-17T11:47:21+00:00             117069   0   0      0       5      pre-MTP
2026-05-17T11:40:24+00:00             97222    0   0      0       3      pre-MTP
2026-05-17T11:33:46+00:00             60132    0   0      0       6      pre-MTP
2026-05-17T11:27:45+00:00             100476   0   0      0       3      pre-MTP

Pre-MTP steady-state cycle band: 60 – 117 s (range ≈ 57 s, mean ≈ 94 s). Post-MTP cycle: 98.9 s — squarely inside the pre-MTP band. Same personas_generated=0 (nothing new to mint), errors_total=3 (typical Ollama contradiction-detect timeouts), auto_tagged=0 (no new eligible memories on this idle DB).

The end-to-end null result mirrors the per-call bench. The MTP env var doesn’t engage on this model + this Ollama, so no speedup propagates to the cycle path either. A second validation: PR #820’s dispatch-deadlock fix is working in production — without it the cycle would never have persisted (the daemon would deadlock on the spawn-blocking thread’s first tracing::info!).

Re-run via:

SELECT created_at, json_extract(content,'$.cycle_duration_ms') AS dur_ms,
                   json_extract(content,'$.personas_generated') AS pg,
                   json_extract(content,'$.errors_total') AS errs
FROM memories WHERE namespace='_curator/reports'
ORDER BY created_at DESC LIMIT 10;

Why MTP didn’t help

OLLAMA_MLX_MTP_MAX_DRAFT_TOKENS gates Ollama’s MLX speculative decoding path. Speculative decoding requires both a target model and a smaller draft model — the draft predicts N tokens autoregressively, the target verifies N predictions in one parallel forward pass. Without a draft, the env var has no effect.

Two possible draft paths exist:

1. Integrated MTP heads on the target model (DeepSeek-V3 style — single weight file, multiple prediction heads). Not present in gemma4:e4b GGUF on this host (ollama show gemma4:e4b lists no draft/speculative capability, and the bench shows zero speedup).
2. A separate companion drafter. google/gemma-4-E4B-it-assistant on HuggingFace fits this shape (“MTP is implemented by extending the base model with a smaller, faster draft model”), but it’s safetensors-only — Ollama’s pull path rejects it with 400: not GGUF or not compatible with llama.cpp. Ollama’s Library doesn’t yet publish gemma4:e4b-draft or gemma4:e4b-assistant either.

So enabling MTP via env on this Ollama + this model is a no-op. The Ollama serve doesn’t error or warn — it just runs vanilla decode.

Paths forward

For most ai-memory deployments today (curator cycles already ≤2 min at steady state, recall-tier doesn’t hit Ollama), MTP is a nice-to-have, not load-bearing. The substantive curator throughput wins live in #651 Phase 1 backends (in-process Rust, no HTTP serialization overhead) and Phase 3 distillation.

Caveats

• MLX backend check. lsof -p <ollama-serve-pid> | grep mlx must show libmlx.dylib / libmlxc.dylib loaded for MTP to be reachable at all. Apple Silicon hosts get MLX auto-selected on Metal-4-capable chips (M2 Pro and later); Intel Macs / Linux NVIDIA hosts won’t have an MLX path regardless of env.
• 20-call sample size isn’t statistically rigorous — for a real benchmark, run ≥ 200 calls per condition with warm-up, randomized prompt order, and a coefficient-of-variation guard. The 20-call result here is sufficient to detect the null-result we found (decode rates differ by <1%) but wouldn’t surface a 5% true speedup confidently.
• Ollama 0.24.0 is the version we tested. The OLLAMA_MLX_MTP_* family of env vars was added in 0.23+; behavior may evolve in later Ollama releases. Re-run after any Ollama upgrade.
• Project hard rule on /tmp — this bench writes all artifacts under /Users/fate/v07/v07-fixes/.local-runs/mtp-bench-2026-05-17/, including the Ollama serve log when launched with the MTP env.

Artifacts on disk

.local-runs/mtp-bench-2026-05-17/
├── bench.sh                     # the bench driver
├── baseline-pre-mtp.jsonl       # 20 calls, no MTP env
├── baseline-pre-mtp.summary.txt # model card + wall times
├── post-mtp.jsonl               # 20 calls, OLLAMA_MLX_MTP_MAX_DRAFT_TOKENS=4
├── post-mtp.summary.txt
└── ollama-serve-mtp.log         # MTP-env-enabled Ollama serve stdout/stderr

Cross-references

• Issue #651 — RFC: pluggable inference backend trait (multi-platform GPU acceleration for autonomous + ULTRA-1 tiers).
• PR #820 — curator dispatch deadlock fix that landed in the same session; unblocks the end-to-end measurement (without that fix the curator daemon never reached Connection::open so cycle timings would never settle).
• v0.7.0 release notes: this doc joins the empirical-evidence set the v0.8+ inference RFC builds on.