The Diminishing Returns of Context

Introduction

The context window was sold as pure upside. Each new model advertised a bigger one - eight thousand tokens, then a hundred and twenty-eight thousand, then a million - and the implied promise was simple: pour in more of your data and the answers get better. The measured evidence says the opposite. As you fill a model’s context with more tokens, and especially with tokens it does not need, its accuracy falls, it starts to miss the very facts you handed it, and it invents new ways to be wrong. More context does not buy a better answer. Past a point that arrives earlier than anyone selling context windows will tell you, it buys a worse one. The fix is not a bigger window. It is precision: giving the model only the slice of data the question actually requires - which is what an ontology is for.

• Length itself hurts: even when every relevant fact is handed to the model, accuracy falls sharply as the surrounding context grows longer.
• It is noise, not volume: a single plausible-looking but irrelevant document can poison an answer, while purely random text leaves it alone - the damage comes from relevant-seeming noise.
• Bloat breeds new failures: long context does not just lower recall; it makes models refuse valid requests and ignore instructions, at rates that climb with every added token.
• Precision beats capacity:feed the model less, chosen well, and accuracy rises while cost falls - but “chosen well” is engineering, and that is where an ontology earns its place.

The capacity myth

The last two years of model marketing turned the context window into a headline number, the way megapixels once sold cameras. A bigger window, the pitch went, means you can stop worrying about what to retrieve: paste in the whole manual, the entire ticket history, every row of the table, and let the model sort it out. It is a seductive idea because it promises to make a hard problem - deciding what information a question needs - disappear into raw capacity.

It does not disappear. A now-substantial body of work shows that a model’s ability to use its context degrades as that context grows, long before the advertised limit. The canonical results - that models get “lost in the middle” of long inputs, and that their effective context is a fraction of the size on the box - we survey in our companion research on the data foundation. This piece is about the practical consequence for anyone building on these models: the window is not free storage. It is a budget, and overspending it has a cost that shows up directly in the answer. The rest of this essay is the measurement of that cost, and of the discipline that avoids it.

When length is the problem

The tempting explanation for long-context failure is that retrieval was bad - the model missed the answer because the right passage was not in the window, or was buried. That happens. But it is not the whole story, and the cleanest experiment isolates the rest. In a 2025 study published in the Findings of EMNLP, researchers from the University of Illinois, Amazon, and the Hebrew University of Jerusalem did something deliberately unfair to the “just retrieve better” hypothesis: they guaranteed the model perfect retrieval - the answer was always present - and then simply varied how much surrounding text sat around it. Even with the relevant information certain to be there, and the input still well within the models’ claimed limits, accuracy still degraded substantially - by 13.9% to 85% across five open and closed models on math, question-answering, and coding, purely as a function of length.

Read that result slowly, because it reframes the whole problem. The answer was in the window. The window was not overfull. And the model got worse anyway, by as much as eighty-five percent, simply because there were more tokens around the fact it needed. Attention is a finite resource spread across everything in the context; the more you put in, the thinner it spreads, and the more the signal you care about competes with everything you piled on top of it. This is why “paste in everything” is not a neutral act. Every irrelevant token you add is a small tax on the model’s ability to find and use the relevant ones.

It is noise, not volume

If length alone were the whole story, the remedy would be blunt: send fewer tokens. The truth is more useful, and it comes from a 2024 study at SIGIR, the field’s premier information-retrieval venue, by a team from Sapienza University of Rome and collaborators. They took a retrieval-augmented question-answering system and asked a precise question: which retrieved documents actually help, and which hurt? The finding that matters is about the documents a retriever ranks highly because they look relevant, yet which do not contain the answer. Adding just one such related document caused a sharp drop in accuracy, with peaks of 0.24 - a 25% reduction. One plausible-looking, answerless passage was enough to knock a quarter off the score.

Then comes the counterintuitive half, and it is the key to the whole argument. When the same researchers padded the context instead with purely random documents - text with no relationship to the query at all - the model’s accuracy improved by up to 35%. Random noise helped; relevant-looking noise hurt. That rules out raw token count as the culprit. The damaging ingredient is not volume, it is plausible irrelevance: context that resembles the answer closely enough to pull the model toward it, but is wrong. A model cannot reliably tell a near-miss from a hit on its own. The only durable fix is to not put the near-miss in front of it - which means something upstream has to know what is genuinely relevant to this question about this entity. That “something” is the job an ontology does.

New ways to be wrong

The cost of getting this wrong is not abstract. In Mata v. Avianca, a U.S. federal judge sanctioned two attorneys and their firm $5,000 after they filed a brief citing six judicial decisions that did not exist, invented by a chatbot answering from memory with no grounding at all. A fabricated citation that looks exactly like a real one is the worst kind of failure, because nothing in the text warns you. For a field-service operator, the equivalent is a confidently wrong figure - an invoice total, a technician’s callback rate, a chemical dosage - rendered in the same clean sentence as a correct one.

And crucially, bloating the context does not merely lower a model’s recall. It manufactures entirely new failure modes. In a large study spanning more than two thousand retrieval-augmented experiments, Databricks Mosaic Research found Claude 3 Sonnet’s rate of wrongly refusing valid requests climbed from 3.7% at 16k tokens to 21% at 32k to 49.5% at 64k - nearly half of all answers lost to a spurious refusal, purely from a longer context. The same study found the model DBRX failing to follow instructions 5.2% of the time at 8k tokens, rising to 17.6% at 16k and 50.4% at 32k, typically by summarizing the retrieved context instead of answering the question it was asked. These are not subtle recall dips. They are the model behaving worse - refusing, drifting, ignoring the task - as a direct function of how much you stuffed into the window.

The behaviour is also not a clean, predictable curve you can simply plan around. The same researchers found the saturation point is model-specific and the decline non-monotonic: Llama 3.1 405B’s accuracy started to fall after 32k tokens and GPT-4-0125-preview after 64k, while a handful of the strongest models kept improving up to 100k. That unevenness is itself the warning. If you cannot predict where a given model stops benefiting from more context and starts suffering from it, you cannot safely treat the window as somewhere to dump data and hope.

The fix is precision

Put the failures together and they point at one remedy. If the problem is length, and specifically plausible-but-irrelevant length, then the answer is not a larger window; it is a smaller, cleaner one - the relevant slice and little else. The question is how you decide what is relevant, and the most striking evidence for the payoff comes from putting a structured layer between the model and the data. In a benchmark on a real enterprise insurance database, presented at the ACM GRADES-NDA workshop in 2024, GPT-4 answering questions directly against the raw SQL database scored 16.7%; the same model answering the same questions posed over a knowledge-graph representation of that database scored 54.2% - roughly three times the accuracy. On the hardest, most complex schemas the gap was total: querying SQL directly scored 0%, while the ontology-backed path scored 35.7%. (The study was authored by data.world, a knowledge-graph vendor, and peer-reviewed; we cite it for the mechanism, not as a neutral referee.) The model did not change. What changed was that a semantic layer - an ontology, with the entities and relationships of the business modeled explicitly - decided what the model saw.

The same principle holds without a formal graph: reduce the context to its informative core and quality goes up, not down. Microsoft researchers built a prompt-compression method that prunes low-information tokens around the question, and at ACL 2024 reported that it boosted GPT-3.5-Turbo on a multi-document benchmark by up to 21.4% while using around four times fewer tokens. Fewer tokens, a better answer, and a large cost saving in the same move. Relevance ordering does the same work from the other direction: Anthropic reported that adding a reranking step to its retrieval pipeline cut the failure rate of finding the right passages by 67%, from 5.7% to 1.9% (a first-party result on their own evaluation). The through-line across all three is one idea: the lever is the relevance and structure of what you pass in, not the raw amount. This is the same discipline we argue for in our research on grounding an AI system in the right evidence. Grounding decides that the answer must tie to a source; precision decides that the source must be the right one, and only that one.

Where the argument bends

An honest version of this argument has to mark where it stops, because “always send fewer tokens” is too strong and the evidence says so. When the model is strong enough and the long context is relevant, reading it all can beat retrieval. In a 2024 study from Google, across a suite of long-context benchmarks, long-context reading surpassed retrieval-augmented generation by 7.6% for Gemini 1.5 Pro, 13.1% for GPT-4o, and 3.6% for GPT-3.5-Turbo on average; the authors’ hybrid, which routes each query to whichever approach fits, mostly served to claw the cost back, cutting it 65% for Gemini 1.5 Pro and 39% for GPT-4o at comparable quality. Which method wins is not settled in favour of narrowing, either. On a controlled benchmark published at ICML 2025, retrieval beat long context by 6.48% and 38.12% on weaker models, but long context beat retrieval by 9.09% (GPT-4o) and 8.98% (Claude 3.5 Sonnet) on stronger ones. The authors’ own summary is the honest one: no silver bullet.

The structured layer has limits too, and they are worth stating plainly. An ontology or knowledge graph is only as good as its construction, and building one automatically is lossy. In a systematic 2026 evaluation, only about 65.8% of the answer entities on one benchmark and 65.5% on another actually appeared in the automatically constructed knowledge graph - a third of the answers were missing before retrieval even began. And on simple, single-hop questions, plain vector retrieval matched or beat the graph approach: vector RAG scored an F1 of 64.78 on one dataset against a graph method’s 63.01 and 54.48. Structure helps most on multi-step, connect-the-dots questions; it can actively lose fine-grained detail on lookups.

None of this refutes the claim; it sharpens it into the right one. “Precision” does not mean starving the model of tokens - the same body of work shows that when chunks are genuinely relevant, passing more of them can beat passing fewer, and even random padding was harmless. The lever is relevance, not scarcity. The defensible claim is this: bloating a context window with more, and especially with plausibly-irrelevant, tokens reliably degrades an AI system, and the durable remedy is a disciplined layer that feeds the model a well-chosen, well-structured, minimal-but-sufficient slice. Fewer tokens are a symptom of doing that right, not the goal itself.

Build for precision

The shape of the claim is now clear, and clearly bounded. Fill a model’s window with everything you have and its accuracy falls, sometimes below where it would sit with nothing at all; the failure is driven by relevant-looking noise, not raw size, and it arrives as confident fabrication and quiet refusal rather than an honest blank. Feed the same model a scoped, structured, minimal slice - the specific entities and facts the question needs - and accuracy rises while cost falls. The one honest caveat, that precision means the right context and not merely less of it, is not an escape hatch for the paste-it-all approach. It is the specification for building the retrieval layer properly.

This is the discipline Ardenus brings to the operators who run the physical economy - the field-service businesses most likely to be handed a chatbot pointed at all of their data at once. Rather than pour a company’s entire operational history into a window and hope the model copes, Ardenus models that data into an ontology - the technicians, routes, invoices, chemicals, and customers, and the relationships between them - and retrieves the precise, governed slice each question needs. The model reasons over signal, not noise. We make the companion arguments - why the data foundation is the part that actually fails, and why the value lives in the system around the model - in our companion research, and you can see the platform itself on the technology page. Do not buy the biggest window. Build the system that knows what to put in it.