Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses

Yuvraj Sehgal; Mahsa Panahandeh; Naser Ezzati-Jivan

cs.SEOct 26, 2026

Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses

Authors: Yuvraj Sehgal, Mahsa Panahandeh, Naser Ezzati-Jivan

Organizations: Department of Computer Science Brock University St. Catharines, Ontario, Canada · School of Electrical Engineering and Computer Science University of Ottawa Ottawa, Ontario, Canada

Abstract

LLM agents are judged by two numbers: the size of the context window, and how often the final answer is correct. Both numbers miss a third thing: how much of a tool's output is lost before the model ever sees it. We measured this loss in a real tool-using agent. The agent finds the cause of injected faults by reading Linux kernel traces. The context window was almost empty: across 720 runs, the single largest prompt used only 5.75% of it. Yet a third of all tool results (3,827 of 11,189) were cut before they reached the model, and every cut result became broken JSON. The harness cut each result by slicing its JSON text at a fixed length. This does not shorten a result evenly. It deletes whatever data sits near the end. One field reached the model in only 4.7% of calls. A second field, in the very same results, reached the model in 99.3% of calls. Which field held the fault's evidence depended on when the faulty process started: early or late in the trace. The agent's diagnosis accuracy matched this same split. So we ran controlled experiments to test one question: did the data loss cause the drop in accuracy? This paper makes four contributions. First, we model an agent's context as a five-stage pipeline, with a measured failure at each stage. Second, we replay 864 real tool results, including one from a production trace where the shipped cut deletes all 128 records of the fault's evidence. Third, a controlled experiment shows that the cut's real cost appears first as extra work, not as wrong answers: 43% more tool calls, twice the computation, and yield falling from 0.61 to 0.31. The agent still finds the right answer when it has another way to reach it. Fourth, a survey of seven agent frameworks finds two that cut tool output the same unsafe way. Whether the data cut truly causes the accuracy drop is still an open question.

Figures & tables

	Boundary	Measured failure (field data)
S1	produce $\to$ deliver	$720$ -run eval: prefix slicing, $34\%$ of results cut, all broken; one field delivered $4.7\%$ vs $99.3\%$
S2	deliver $\to$ act	$554$ -worker campaign: the step budget is a wall, $51\%$ of workers stopped at exactly the limit
S3	act $\to$ retain	one before/after run: raising retention $80$ k to $150$ k chars moved it from $303$ k to $545$ k tokens
S4	retain $\to$ transfer	same $554$ -worker campaign: hand-off is voluntary, $98$ workers passed on nothing; silence reads as “nothing found”
S5	transfer $\to$ persist	by design, not measured: nothing survives the run (to keep runs independent)

TABLE I: The five boundaries, and the failure found at each one. S1–S4 come from the harness’s own field data, collected before the controlled experiment in Sec. IV ; S5 is a design choice, not a measured failure. These rows show how often each boundary fails. Whether that failure changes the final answer is a separate question, tested only by the controlled experiment.

Fig. 1: The delivery cliff. We build a result with the real tool’s shape and replay it through both cutting rules as it grows past the shipped

6{,}000

-character cap. No model is involved. Under the prefix slice, the late field arrives whole until the cap, then vanishes entirely and the payload breaks (bottom strip). The row-aware cut keeps every field at every size. The stars mark where the real production trace (Sec. III-B ) falls:

33{,}370

characters,

128

records, delivered or lost whole, exactly as predicted.

	correct	calls	cut/run	comp.	yield	tokens
repaired	14/15	53	0	18	0.61	(567k)
S1 defect	15/15	68	27	32	0.34	$+5\%$
S1 reversed	15/15	76	30	36	0.31	$+14\%$
order only	10/10	48	0	16	0.61	$-12\%$

TABLE II: The cut changes the price, not the answer. Medians over the

15

spawned-culprit runs per configuration (

5

recordings

\times

3

repeats; order-only has

10

). Yield = findings recorded

\div

computations run.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 250 papers

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Sep 4, 2026

Agents' Overreliance on Unreliable Tools

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with three tools (web search, an LLM sub-agent, and a code executor), corrupting their returns to examine whether agents overrely on unreliable tools. Agents adopt corrupted returns at high rates, with mean adoption exceeding one third for every tool and reaching 68.0% for web search. Corrupted returns also frequently override correct answers that agents give without tools, and this reliance persists in more complex tasks and in tasks that combine multiple tools. Agents often make more tool calls under corrupted returns, and reasoning traces show them questioning a return and even stating the correct answer. Nevertheless, they still pass the corrupted content to users, rarely warning them of the conflict. We further test interventions spanning prompts, tool metadata, post-training, and activation steering. Verification prompts reduce adoption across all three tools while largely preserving accuracy with correct returns, and reliability labels provide partial mitigation in web search. The tested post-training methods offer limited gains, and the effect of activation steering depends on the model and intervention strength. Overall, our findings provide a basis for diagnosing tool overreliance and for developing agents that assess tool returns rather than assuming their reliability.

Hoyeol Yang, Woojung Song, Taewon Kim +3

Seoul National University, Seoul, Republic of Korea

LLM Tool Use LLM Agent Reliability LLM Agent Evaluation

Ask PDF

Apr 30, 2026

Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents

Tool-augmented reasoning has become a popular direction for LLM-based agents, and it is widely assumed to improve reasoning and reliability. However, we demonstrate that this consensus does not always hold: in the presence of semantic distractors, tool-augmented reasoning does not necessarily outperform native CoT. To explain this performance gap, we propose a Factorized Intervention Framework that isolates the cost of prompt formatting, the overhead of the tool-calling protocol, and the actual gain from executing tools. Our analysis reveals a critical tradeoff: under semantic noise, the gains from tools often fail to offset the "tool-use tax", which is the performance degradation introduced by the tool-calling protocol itself. To address this, we introduce G-STEP, a lightweight inference-time gate to mitigate protocol-induced errors. While this yields partial recovery, our findings suggest that more substantial improvements still require strengthening the model's intrinsic reasoning and tool-interaction capabilities.

Kaituo Zhang, Zhen Xiong, Mingyu Zhong +4

University of Houston · University of Southern California · 3New York University +2

Tool-Augmented Language Model Agents Agentic Reasoning LLM Tool Use Causal Interventions in Language Models Language Model Robustness LLM Agent Reliability

Ask PDF

Abstract

LLM Context Management

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Agents' Overreliance on Unreliable Tools

Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents