We A/B tested agent memory against no memory at all — on a live fleet. Here is what broke.
Every memory layer for AI agents makes the same promise: your agent stops relearning the same lesson every session. The evidence offered for that promise is almost always a retrieval benchmark — recall@k on LoCoMo or LongMemEval. Those benchmarks answer whether a retriever finds the right row. They do not answer the question a team actually has, which is whether the agent does better work.
Answering that question requires a control arm: agents doing real work with the memory deliberately switched off. Nobody in this category publishes one, and the reason is structural — you need a production fleet you are willing to handicap, at random, for a month. We run one, so we ran the experiment. This post is the result, including the arm that failed, which happens to be the arm half this industry sells as intelligence.
The setup
For 30 days, every run on our agent fleet was assigned to one of three arms by hashing the run identifier into three equal-probability buckets — deterministic, re-derivable, invisible to the agent and to anyone watching fleet health. No task selection, no cherry-picked slice, no retrospective filtering.
| Arm | Configuration |
|---|---|
| A — memory off | no recall injected; the control |
| B — memory on | hybrid recall (vector + BM25), no score threshold, no type filter |
| C — selective | vector-only recall behind a 0.40 cosine gate and a type filter (procedure, incident, decision) |
Outcome measures: task success as judged by the same pipeline in all three arms, agent turns to completion, and model spend per run.
The results
| Arm | Runs | Success | Turns (mean) | Cost (mean) |
|---|---|---|---|---|
| A — memory off | 150 | 55.3% | 34.39 | $1.4082 |
| B — memory on | 148 | 66.9% | 27.81 | $1.2321 |
| C — selective | 137 | 55.5% | 34.49 | $1.1683 |
Memory-on beat memory-off on success rate by 11.6 points (χ²=4.19, p=0.041 uncorrected) and used 19% fewer turns (p=0.064). On cost there was no effect at all (p=0.38).
- The A-vs-B comparison was not pre-registered. The pre-registered primary metric was arm C cost against arm B cost — and on cost there is no effect.
- The pre-registered target was 400 runs per arm. We stopped at 150/148/137 and closed the experiment; these figures are final and will not be quietly replaced by a bigger sample later.
- Under Bonferroni correction for three pairwise tests, no comparison survives: the A-vs-B p-value rises to 0.12.
- What keeps us from calling it noise: arms were balanced week over week, and the direction held independently in both high-volume weeks. Consistency is not significance. It is a reason to treat this as a first signal, nothing more.
The part that failed is the useful part
Selective recall performed exactly like having no memory at all. 55.5% success against 55.3% for the control — χ²=0.001, p=0.98. Statistically the same result. Plain always-on recall beat it (p=0.048 uncorrected).
We built that arm on the premise that a memory which knows when to stay quiet beats one that always speaks — the premise behind every "intelligent filtering" pitch in this category. We spent a quarter on it. Our own measurement says it was worth nothing.
And the obvious explanation is the wrong one. Arm C was not starved of context. It injected more lessons per run than the winning arm — 4.05 against 3.29 — retrieved nothing at the same rate (12.4% vs 12.2%), at essentially the same match quality (mean cosine 0.561 vs 0.570). More retrieved context, and no benefit.
What actually separates the arms is the retrieval mechanism. The winning arm used hybrid retrieval — vector plus BM25, no gates. The failing arm used vector-only retrieval behind a similarity threshold and a type filter. On this evidence, the lever is how memory is retrieved, not how much of it you hand over. A vector-only path with quality gates returned more material that helped less.
If you build agent memory, here is what we think generalizes
First: measure end-task outcomes, not retrieval. Our retriever scored fine on the standard benchmarks in all three arms; the arms still differed by 11 points of task success. Retrieval quality and usefulness are not the same axis, and only a control arm tells you which side of that gap you are on.
Second: selectivity is not free intelligence. Score gates and type filters feel like rigor, and every failed retrieval they prevent is visible while the benefit they discard is not. The only way to see the discarded benefit is to run the ungated arm next to it.
Third: publish the arm that failed. Ours cost us a quarter of engineering and the headline feature of two releases. It is also the most useful number in this post for anyone deciding where to spend their own quarter.
Check our numbers
The de-identified per-run dataset (arm, week, outcome, turns, cost, retrieved count, mean cosine — no task identifiers, no role names, no project names) ships in the repository under benchmarks/mem_ab_v3/, with an analysis script that reproduces every number above using the Python standard library. The full protocol and caveats live in docs/EVIDENCE.md — including a disclosure section about the time a draft of that dataset was fabricated, what caught it, and the release gate that now makes that class of failure block the tag.
pgmnemo itself is a Postgres extension — memory for coding agents in the database you already run: CREATE EXTENSION pgmnemo CASCADE, no sidecar, no data egress, no model call on the write path. Where we lose is published too: BM25 still beats us on LongMemEval-S, and the graph layer ships for lineage, not ranking. pgmnemo.com has the rest.