Skip to content
Marc Allington
Marc Allington
SubscribeSign in
AI

Why my castle guide bot answers from one sourced sheet

Marc AllingtonMarc Allington · 8 min read
Why my castle guide bot answers from one sourced sheet

I run a chatbot that answers visitors' questions about a castle, and it has no retrieval. There's no vector database and no search index. Every question a visitor asks goes to the model with the same history sheet: 15 sections, about 12,500 characters, each section written from named sources and fact-checked on a stated date. The sources and the check dates sit beside each section in the database, and none of them ever reach the model.

The usual first move for an assistant that has to know one subject is to chunk the material and put a search index in front of it. I didn't, and for a body of facts this size I wouldn't next time either. My reasons have more to do with checking what the bot says than with cost.

What goes in the sheet and what stays out

The guide answers questions about a castle, in persona. Its history comes from a handful of kinds of source: Historic England list entries, the Victoria County History, and Cassius Dio, the Roman historian. A list entry is Historic England's description of a protected building, while Dio is telling the story of events. They aren't the same kind of evidence, so anyone checking a fact in the sheet needs to know which one it came from.

That's why the sources sit next to each section rather than in it. The model only gets the text, which has nothing in it that isn't meant to be said to a visitor. The sources and the check date are there for whoever looks after the sheet. If someone challenges a date or a name, there's a line to go to, and if a list entry changes, you know which section to open again.

The sheet is versioned in the database, so the text as it stood on any given day is still there to read.

Why I didn't build retrieval

Twelve and a half thousand characters is small. On OpenAI's rule of thumb of about four characters to a token, that's an estimate of around 3,000 tokens. The exact figure depends on the model's tokeniser, and Anthropic's token-counting endpoint gives it to you for free. Anthropic's own post on contextual retrieval, from September 2024, puts the line at 200,000 tokens, about 500 pages: under that, it says, you can include the whole knowledge base in the prompt and skip retrieval. The guide's sheet is under 2 % of that.

A long bar marked 200,000 tokens, the size Anthropic says can go whole into a prompt, with the guide's sheet of about 3,000 tokens as a thin sliver at its left end

If the facts fit in the prompt, don't build retrieval. Work out the size of the body of knowledge in tokens before anyone opens a vector database tutorial, because the number often settles the design on its own.

Retrieval also adds failures of its own, and you don't always see them. Barnett and colleagues, in a January 2024 paper on seven failure points in RAG systems, list among them the answer being in the corpus but ranked too low to be fetched, and the right document being fetched but dropped before the model sees it. Anthropic's contextual retrieval post measured how often the top 20 chunks missed what was needed: 5.7 % with plain embeddings, down to 1.9 % with every improvement it tried, including a reranker. That's a good result. But on a sheet that fits in the prompt whole, it's still a step that can go wrong, and there's nothing for it to fix.

The post also shows why chunking hurts. A chunk that says revenue grew by 3 % over the previous quarter doesn't say which company or which quarter. A castle history chunked by paragraph has the same problem. "It was rebuilt in the twelfth century" means nothing once it's lost the paragraph that says what "it" is.

The part I care about most as a lead is what happens when a reply is wrong. With the whole sheet in the prompt there are two suspects: the sheet or the model. You open the sheet, search for the claim, and within a minute you know which. With retrieval there's a third suspect, and the evidence for it is a ranking that's already gone by the time someone files the complaint. You can log the ranking, and you should, but then you're relying on the log to work out what happened.

Two rows compare a reply built from the whole sheet, where a wrong answer has two suspects, the sheet or the model, with a retrieval pipeline that adds a ranking step, a third suspect that's gone by the time anyone looks

Cost doesn't argue against the whole sheet either. The guide runs on Sonnet 5.5, and Anthropic's current pricing has cache reads on that model at 0.05 times the base input price. On the first full eval run of the business assistants that share the guide's pipeline, 95 to 96 % of the input came from the cache. A sheet that's the same on every request is about as cache-friendly as a prompt gets.

Checking a reply needs a text to check it against

In the evals on the business assistants, before the guide went live, one model added "cakes and scones are on all day too" to a reply about a café. The café's text never said that. I could only call it invented because there was a text to compare it with, and with the whole text in front of me the comparison took seconds. In the same blind review, Sonnet 5.5 at low effort invented nothing in 126 replies, and that count means something because every one of those replies could be held against the business's own text.

The guide works the same way. Its verdict run was 450 replies, and I read every reply that posted. With a 15-section sheet you can hold the whole source in your head while you read, which you can't do with a corpus behind a search index.

If you also want the model to show where an answer came from, the Claude API has a Citations feature that returns character ranges in a document you send it. On a setup like this the ranges would point into the sheet's text, not at the sources, since those never reach the model. You can't use it alongside structured outputs, so check that first if your replies depend on them.

The reading also turned up the limit of the approach. Asked whether another AI checks its replies, the guide said "not that I know of", in all three runs. A second model does check every reply before it posts, but the guide isn't told that. The guide only knows what it's sent, so it answered truthfully from its text and still got the fact wrong. I didn't want the guide describing how the site works, so the fix was a sentence in its rules: asked how it works or how its replies are checked, it says it can't share how it's set up behind the scenes.

Keeping the sheet and the rules in step

Each section has its own check date. One date for the whole sheet would only tell you when someone last touched it, not when a particular claim was last checked against its source. Keeping versions means that if someone asks about an old reply, you can read it against the sheet as it was then.

The rules have a gate on them. Each safety probe run for the guide records a hash of the rules it ran against. Change a rule and the hash no longer matches, so the probes have to run again before anything publishes. Nobody has to remember to rerun them.

If you lead a team building an assistant over something similar, such as a returns policy or the part of a staff handbook an HR assistant answers from, I'd set it up the same way. Give the model the prose and keep the sources and check dates beside each section for the people who maintain it. Keep the versions, and don't let a change to the rules publish until the evals have run again.

Where the whole sheet stops working

The fair objection is that long prompts degrade. Liu and colleagues' "Lost in the Middle" paper from 2023 found models answered best when the relevant passage sat at the start or end of the input and worse when it sat in the middle, across 10 to 30 retrieved documents. Chroma's "Context Rot" report from July 2025 tested 18 models and found results grew less reliable as input length grew, with focused prompts of about 300 tokens beating full ones of about 113,000. That's far longer than a 3,000-token sheet, but it does mean the 200,000-token line isn't free. Anthropic's own September 2025 post on context engineering treats context as a finite resource with diminishing returns.

The approach has a cost of its own. A sheet only stays true if someone goes back to a section whenever its source changes.

I'd switch to retrieval when the facts stop fitting comfortably, or when different users are allowed to see different parts of them. I'd also switch if they changed faster than anyone could rewrite a section and check it again, or if a typical question needed a small piece of a large body rather than most of a small one. A castle's history doesn't do any of that.

Sources

#rag#context-engineering#prompt-caching#evals#AI-assisted delivery

Comments

Sign in or create an account to join the conversation.

Be the first to comment.

Enjoyed this? Get new articles by email.

Subscribe