Illustration: AI-generated, art-directed by the author.
TL;DR · 20 seconds
- More documents make AI worse: buried mid-stack, a model scored lower with 30 documents than with none.
- The most damaging noise is the almost-relevant kind: duplicates and stale drafts your retriever ranks highest.
- Let AI be the librarian: dedupe by meaning, expire contradictions, summarize archives, gate admissions.
Every company builds its AI knowledge base the same way: connect everything. Every drive, every wiki, every ancient PDF. The unspoken theory is that more knowledge means better answers, and storage is cheap, so why choose?
The research says the theory is backwards. Three separate lines of evidence, one conclusion: a knowledge base is not an archive. Feeding your AI everything is how you make it dumber.
The evidence, in ascending order of embarrassment
Models lose things in the middle. The famous "Lost in the Middle" study (Stanford and collaborators, published in TACL 2024) measured what happens when the passage containing the answer moves through a stack of retrieved documents. Accuracy follows a U-shape: strong when the answer sits first or last, collapsing in the middle. In the 30-document setting, GPT-3.5-Turbo scored 73.4% when the answer sat at the very top of the stack but only 50.5% with it buried mid-stack, worse than the 56.1% it scored with no documents at all. The retrieved pile made the model worse than its own memory.
Thirty documents scored worse than none
Accuracy by where the answer sits in the retrieved stack.
Answer near the top of the stack
No documents at all (closed book)
Answer buried mid-stack (30 documents)
View as table
| Condition | Accuracy |
|---|---|
| Answer near the top of the stack | 73.4% |
| No documents at all (closed book) | 56.1% |
| Answer buried mid-stack (30 documents) | 50.5% |
GPT-3.5-Turbo, 30-document setting. Source: Liu et al., “Lost in the Middle,” TACL vol. 12, 2024.
Everything degrades as input grows. In 2025, Chroma Research tested 18 models, frontier ones among them (the GPT, Claude, Gemini, and Qwen families) and found every single one degrades as input length grows, even on trivially simple tasks, and that degradation is continuous: there is no safe zone before the cliff. A single distractor document measurably reduced accuracy; four compounded it. (Disclosure your way: Chroma sells retrieval infrastructure, so they benefit from this conclusion. Their methodology is public, and it matches the academic work.)
The most dangerous noise is the almost-relevant kind. A SIGIR 2024 study ("The Power of Noise") quantified what a bloated corpus actually does at retrieval time: adding a single related-but-wrong document, exactly the near-duplicates and stale drafts a dump-everything corpus is full of, cut a tested model's accuracy from 0.56 to 0.43, with peak drops around 25%. The cruel mechanism: your retriever ranks those almost-relevant documents highest, because they look like the answer. (The same study found truly random, unrelated text can sometimes help. Randomness is not your problem. Your problem is the seven old versions of the pricing sheet.)
So the dump-everything knowledge base fails on physics, not effort: retrieval surfaces lookalikes, lookalikes displace the real answer, and long noisy stacks rot even the best model's attention. And every one of those wasted passages is also a token you pay for (the token bill) and a document that can leak (bringing the AI inside your walls).
Thirty documents scored worse than none. The pile lost to the model's own memory.
The librarian, and why it should be an AI
The fix is curation, and the honest objection is that nobody has time to curate: no team is reading ten years of shared drives. Correct. That is why the librarian should be the AI itself, doing the boring work it is genuinely good at, before anything gets vectorized:
- 01Deduplicate by meaning, not filename. The same policy exists as final_v2, final_FINAL, and a pasted copy inside a deck. On the meaning map they are neighbors; an embedding pass finds them in minutes. Keep one canonical version, tombstone the rest.
- 02Expire by evidence. Let the AI flag documents that contradict newer documents (old prices, dead org charts, retired products). Contradictions are exactly what poisons answers, because the retriever cannot tell which version is true.
- 03Summarize the archive instead of storing it raw. The 2019 project folder does not need 400 files in the index. It needs one AI-written summary with a pointer to the originals. History stays findable; noise leaves the retrieval path.
- 04Gate the door. New documents earn their way in through the same checks. A knowledge base with no admission standard returns to junk-drawer state in a quarter.
- 05Watch what retrieval actually uses. Logs show which documents get retrieved and which never do. A document that is never retrieved is a candidate for the archive tier; a wrong document that is retrieved often is an emergency.
Run that loop quarterly and the corpus gets smaller, sharper, and cheaper while everyone else's gets bigger and blurrier.
I run this on my own stack, and the uncomfortable part is how much of what I remove I had been proud to add. The first pass through my agents' knowledge base cut it by roughly a third, almost entirely superseded versions of documents I had written myself: three generations of the same pricing logic, two of the same onboarding steps. Nothing in there was wrong when it went in. That is the trap. A corpus does not rot because you added bad documents, it rots because you added good ones and never told the machine which of them stopped being true.
A knowledge base is not an archive. It is an attention budget.
The part your leadership meeting will resist
Someone will say deleting feels risky. Point them at the numbers above and reframe it: nothing is deleted; it is demoted out of the retrieval path. The risk runs the other way, because a corpus full of stale lookalikes is how you manufacture an answer that sounds right and is not. Deciding what belongs in the HR assistant's corpus and what belongs in finance's is not an IT task. It is the subject-matter half of the job, and it is why one tool per domain beats one assistant for everybody. The originals live in cold storage. What you are pruning is not information, it is the machine's reading list.
Demoted, not deleted
The same corpus, drawn twice: dumped in whole, then after the librarian’s pass.
Dump everything into the index
Retrieval reads the whole pile. The real answer is a sliver the retriever has to out-rank the lookalikes to find.
After the librarian’s pass
The retrieval path is short and mostly signal. The rest still exists; it is demoted off the reading list, not deleted.
Proportions are illustrative, not measured. The point is the ratio: curation shrinks what the model reads, it does not erase what you keep.
Because that is what a knowledge base actually is. Not an archive: an attention budget.
Storage is cheap. Attention, the model's and yours, is the expensive thing. The companies getting sharp answers did not find better AI. They gave the same AI a better reading list. When did your company last remove anything from its machine's?
Sources (3)
- 01Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," TACL vol. 12, 2024 (U-shape; 30-doc setting 50.5% mid-stack vs 56.1% closed-book; figures per arXiv version 2307.03172): aclanthology.org/2024.tacl-1.9/
- 02Chroma Research, "Context Rot," July 14 2025 (18 models degrade with input length; single distractor reduces accuracy; vendor interest disclosed): trychroma.com/research/context-rot
- 03Cuconasu et al., "The Power of Noise: Redefining Retrieval for RAG Systems," SIGIR 2024 (related-doc drop 0.5642 to 0.4283, peak ~-25%; random-docs caveat): arxiv.org/abs/2401.14887
