Retrieval-Based Tool Output Cuts LLM Agent Accuracy

Background and Context

An LLM agent can compute the right value and still give the wrong final answer. A new arXiv study examines one engineering choice behind that failure. To save context, many agent harnesses no longer place long tool output in the conversation. They store it in files, and the agent searches or reads those files on demand.

The authors tested this choice with PHREEQC, the geochemical simulator from the USGS. It is a real scientific tool, and it is deterministic. Its reports reach 846.8k characters, so every printed number can be traced. Agents used it to answer 158 execution-verified multiple-choice questions.

The paper makes three contributions. It measures the accuracy cost of reading a report through retrieval instead of seeing it in context. It splits that cost into how often the agent computes the target value and which evidence the answer rests on. It also names the resulting failure mode, evidence substitution, and compares several mitigations.

Deep Analysis

The design is built to separate causes. In one condition, the agent sees the simulator report in context. In the other, it reaches the report only through a section index. Because PHREEQC returns the same output for the same input, the authors could rerun all 11,007 simulation calls recorded in the agent logs. For each call they checked whether the simulation itself produced the key value the question needed. That check separates two different error sources. In the first, the simulation was wrong. In the second, the simulation was right but the final answer did not come from the right evidence. The authors also ran a fixed-simulation analysis. It holds the simulation outcome constant and compares only how the evidence was used. Why does retrieval-based tool output reduce LLM agent accuracy? In this study, the loss did not come from worse calculation. When agents searched reports through a section index, GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 percentage points of accuracy. Claude Opus 4.6 and Sonnet 4.6 showed no detectable loss. Rerunning the simulations showed that the GPT models computed the target value about as often as before. What changed was the evidence behind the answer.

Under retrieval, the GPT models often skipped the report. They cited a higher-precision file of their own output and rounded to a neighboring option. The authors call this evidence substitution. With the simulation held fixed, retrieval still cost GPT-5.2 9.4 points. That figure points to evidence use, not simulation quality, as the source of the loss. GPT-5.4 is a subtler case. It made correct simulations more often, and that masked roughly ten points of evidence-use loss. The net change was indistinguishable from zero, although two opposing effects were at work.

Industry Impact

The first lesson is about evaluation. A headline accuracy score would suggest GPT-5.4 was unaffected by retrieval. The decomposition shows a real evidence-use penalty hidden by better simulation. Teams that test context-management strategies in an agent harness should therefore check where accuracy is lost, and not only read the total. The second lesson concerns deployment. Storing large tool output on disk and retrieving it is a common way to save tokens. This paper shows the practice is not free for every model. In the tested setup, the two GPT models paid a clear price while the two Claude models did not. Model choice and context strategy should be decided together.

The third lesson is that the fixes are cheap. Naming the authoritative output raised three GPT models by 6.3 to 7.9 points over plain retrieval. For GPT-5.1 and GPT-5.2, inlining output that fits in context worked about as well as naming it. Asking the agent to round correctly did not help. The failure is therefore about which evidence the agent chooses, and a rounding reminder does not touch that choice. The risk is sharpest in scientific computing. There, neighboring answer options can differ only by rounding. A wrong choice of source file is enough to produce a wrong answer, even when the underlying calculation is correct.

Outlook

The method is reusable. A deterministic simulator plus a replay of logged calls lets researchers decompose failures into computation and evidence use. The paper's setting is PHREEQC, but the same approach could be applied to other tools that produce long output, such as database queries or code execution environments. The source does not report such tests, so that extension remains a suggestion.

For tool designers, the practical step is to label the authoritative output clearly. A tool that writes several files with different precision gives an agent room to pick the wrong one. Marking one file as the source of truth, or returning short results inline, removes that room at low cost.

Some questions stay open. The study covers one tool family and 158 questions, and the authors report model-specific effects. GPT-5.4 shows that gains in one skill can hide losses in another, so future harness evaluations will need to report both parts. Until that becomes routine, retrieval-based reading of tool output should be treated as a model-dependent setting that needs testing, not as a default that is safe for every agent.