Why I didn't build RAG for a 4,000-token knowledge base

I run a chatbot that answers visitors' questions about a castle, and it has no retrieval. There's no vector database and no search index. Every question a visitor asks goes to the model with the same history sheet: 15 sections and 4,242 tokens, each section written from named sources and fact-checked on a stated date. The sources and the check dates sit beside each section in the database, and none of them ever reach the model.

The usual first move for an assistant that has to know one subject is to chunk the material and put a search index in front of it. I didn't, and for a body of facts this size I wouldn't next time either. My reasons have more to do with checking what the bot says than with cost.

What goes in the sheet and what stays out

The sheet draws on a handful of kinds of source: Historic England list entries, the Victoria County History, and Cassius Dio, the Roman historian. A list entry is Historic England's description of a protected building, while Dio is telling the story of events. They aren't the same kind of evidence, so anyone checking a fact in the sheet needs to know which one it came from.

That's why the sources sit next to each section rather than in it. The model only gets the text, which has nothing in it that isn't meant to be said to a visitor. The sources and the check date are there for whoever looks after the sheet. If someone challenges a date or a name, there's a line to go to.

Why I didn't build retrieval

I counted the sheet with Anthropic's token-counting endpoint, which is free: 4,242 tokens on Sonnet 5.5, the model the guide runs on. Anthropic's own post on contextual retrieval, from September 2024, puts the line at 200,000 tokens, about 500 pages: under that, it says, you can include the whole knowledge base in the prompt and skip retrieval. The guide's sheet is about 2% of that.

If the facts fit in the prompt, don't build retrieval. Count the body of knowledge in tokens before anyone opens a vector database tutorial, because the number often settles the design on its own.

Retrieval also adds failures of its own, and you don't always see them. Barnett and colleagues, in a January 2024 paper on seven failure points in RAG systems, list among them the answer being in the corpus but ranked too low to be fetched, and the right document being fetched but dropped before the model sees it. Anthropic's contextual retrieval post measured how often the top 20 chunks missed what was needed: 5.7% with plain embeddings, down to 1.9% with every improvement it tried, including a reranker. That's a good result. But on a sheet that fits in the prompt whole, it's still a step that can go wrong, and there's nothing for it to fix.

It's also one more component to look after. Retrieval needs its own eval set, one that asks whether the right passage came back, and it can regress whenever someone changes the chunk size, the embedding model or the reranker.

The post also shows why chunking hurts. A chunk that says revenue grew by 3% over the previous quarter doesn't say which company or which quarter. A castle history chunked by paragraph has the same problem. "It was rebuilt in the twelfth century" means nothing once it's lost the paragraph that says what "it" is.

The part I care about most as a lead is what happens when a reply is wrong. With the whole sheet in the prompt, a wrong fact has two places to come from: the text the model gets, which is the sheet and its rules, or the model itself. What the model saw is never in question, because it saw all of it. Retrieval adds a third suspect, the ranking, and it's gone by the time anyone looks unless you logged it. Even a good log only tells you what was fetched. When the passage was in the corpus and didn't come back, you still have to work out why: the chunking, the embedding or the ranking. That's one more thing to rule out on every complaint.

A second, cheaper model does read every reply before it posts, so each reply costs two calls. It isn't a suspect for a wrong fact. Its rubric tells it to judge a reply by its form and its subject, not by whether its facts are right.

Cost doesn't argue against the whole sheet either. Before the guide went live, I ran evals on the assistants that local businesses use on the same site, which share the guide's pipeline: 98 scripted chats against two made-up businesses, across six combinations of model and effort. On that first full run, 95 to 96% of the input came from the cache. The guide runs on Sonnet 5.5, and Anthropic's current pricing has cache reads on that model at 0.05 times the base input price. A sheet that's the same on every request is about as cache-friendly as a prompt gets.

Checking a reply needs a text to check it against

In those evals one model added "cakes and scones are on all day too" to a reply about a café. The café's text never said that. I could only call it invented because there was a text to compare it with. Five independent reviews then read every reply blind, with the models relabelled, and Sonnet 5.5 at low effort invented nothing in 126 replies. That count means something because every one of those replies could be held against the business's own text. I've written up what that eval set showed separately.

For the guide I used a different check. I took 399 of its replies from a calibration run of ordinary visitor questions and had four model checkers compare every factual claim with the sheet. The flags are theirs, not a ruling, and they flagged 55 replies. One was a plain contradiction: the guide called Boudica "the Roman queen", where the sheet says she was queen of the Iceni and led a revolt against Roman rule. Five gave details the sheet doesn't have, such as comparing the keep with the Tower of London as a whole. The other 50 were stretches, and a good share of them came from the sheet's own wording. It said "you can still see marks left by the boards" in the Roman vaults, and the guide turned that into telling visitors they could see the vaults, which the sheet never said. The sheet also said part of the keep was a county prison again from 1691, and seven replies turned that into "the county prison". Four called someone "the best-known" where the sheet ranks nobody.

One question went wrong in a different way. Asked "Who ordered those executions?" as the only message in the chat, the guide answered and then corrected an earlier answer it had never given, in all three runs: "One small correction to what I said before". It looked like the model making things up. The cause was in the sheet, which set Historic England's account (the two royalist commanders held in a room in the keep) beside the VCH's (executed outside the castle) as if they disagreed. The new version gives them as one sequence, held and then executed, and in the re-run the invented correction had gone.

So I changed the sheet and added one rule, that the guide only calls someone "the best-known" where the history does, then ran the same calibration set again before putting any of it live. The contradiction and the invented correction were gone, and fewer replies said "the county prison". The vaults didn't move. Between 12 and 15 of the 399 replies still told visitors they could see them, however I counted. The new sheet says nothing about seeing the vaults, so that one is the model, and it'll take a rule change. A search of the sheet rules it out.

One new fault came from the sheet itself. One source dates the building work as 1076 to 1086, and two replies in the first run had given 1086 as the start, so I added a sentence saying 1086 is where the work ended. The span still sat inside the sheet's list of when work began. Two replies gave it as the start again, which the checkers marked as contradictions, and three said the castle was finished by 1086. That one is the sheet, and the fix is to move the span out of that list.

Don't put much weight on the totals, though. Two separate runs of model checkers over the same 399 new replies flagged 45 and 72, because one brief named the patterns to look for and the other didn't. The patterns held up in both, and in a plain text search of the replies. I'd report the patterns and keep the totals for trend.

My read of the guide's last eval run before launch had already turned up the limit of the approach. Asked whether another AI checks its replies, the guide said "not that I know of", in all three runs. A second model does check every reply before it posts, but the guide isn't told that. The guide only knows what it's sent, so it answered truthfully from its text and still got the fact wrong. I didn't want the guide describing how the site works, so the fix was a sentence in its rules: asked how it works or how its replies are checked, it says it can't share how it's set up behind the scenes.

Keeping the sheet and the rules in step

Each safety probe run for the guide records a hash of the rules it ran against. Change a rule and the hash no longer matches, so the probes have to run again before anything publishes. Nobody has to remember to rerun them. The "best-known" rule went through exactly that: new rules, a new hash, and the probes ran again before the change could merge.

The sheet has the same kind of record, kept by section. One date for the whole sheet would only tell you when someone last touched it, not when a particular claim was last checked against its source. For the new version I fact-checked the seven sections that changed, and the other eight kept their earlier check dates. The database keeps every version, so if someone asks about an old reply, you can read it against the sheet as it was then.

Someone has to own each part. On a team I'd give the sheet to a content owner, a person who knows the subject and signs off every change to a section. The lead owns the rules, the gate and the evals. A complaint about a wrong answer goes to the content owner first, since the first question is what the sheet says, and the lead decides whether it's a sheet fault or a model fault. The vaults and the 1086 dates are one of each.

If you lead a team building an assistant over something similar, such as a returns policy or the part of a staff handbook an HR assistant answers from, I'd set it up the same way. Give the model the prose and keep the sources and check dates beside each section for the people who maintain it. Keep the versions, and don't let a change to the rules publish until the evals have run again.

Where the whole sheet stops working

The fair objection is that long prompts degrade. Liu and colleagues' "Lost in the Middle" paper from 2023 found models answered best when the relevant passage sat at the start or end of the input and worse when it sat in the middle, across 10 to 30 retrieved documents. Chroma's "Context Rot" report from July 2025 tested 18 models and found results grew less reliable as input length grew, with focused prompts of about 300 tokens beating full ones of about 113,000. That's far longer than a 4,242-token sheet, but it does mean the 200,000-token line isn't free. Anthropic's own September 2025 post on context engineering treats context as a finite resource with diminishing returns.

The hard case is in between. Take a 150-page policy set. On Anthropic's ratio of about 500 pages to 200,000 tokens, that's around 60,000 tokens: well under the line, and about half the size of the full prompts Chroma tested. It fits, and caching keeps repeat questions cheap, but every request still sends the whole set, and the answer to a typical question sits somewhere in the middle of it. I'd still start with the whole set in the prompt, because it's the cheapest thing to build and the easiest to check. I'd build the eval set first, though, with questions whose answers sit deep in the middle, and run it at full size. If those answers come out worse than the ones near the start and end, that's when retrieval, or splitting the set by topic, earns its place. Towards 150,000 tokens you're past Chroma's full prompts and close to Anthropic's line, and I'd expect that test to fail more often.

The approach has a cost of its own. A sheet only stays true if someone goes back to a section whenever its source changes.

I'd switch to retrieval when any of these holds:

• the facts no longer fit comfortably, or the evals show answers getting worse as the body grows

• different users should only see different parts of it

• it changes faster than anyone can rewrite a section and check it again

• a typical question needs a small piece of a large body rather than most of a small one

A castle's history doesn't do any of that.

Sources

• Introducing Contextual Retrieval, Anthropic, https://www.anthropic.com/news/contextual-retrieval

• Effective context engineering for AI agents, Anthropic Engineering, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

• Prompt caching, Claude Docs, https://platform.claude.com/docs/en/build-with-claude/prompt-caching

• Token counting, Claude Docs, https://platform.claude.com/docs/en/build-with-claude/token-counting

• Seven Failure Points When Engineering a Retrieval Augmented Generation System, Barnett et al., arXiv, https://arxiv.org/abs/2401.05856

• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv, https://arxiv.org/abs/2307.03172

• Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, https://www.trychroma.com/research/context-rot