Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Fault

Tianqi Bu; YuXuan Peng; Junteng Tu; Henghui Xiao

cs.AISep 26, 2026

Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Fault

Authors: Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao

Organizations: Rutgers University · Nanjing Tech University · Nanjing University of Posts and Telecommunications · New York University

Abstract

One variable sets what an upstream fault costs a staged pipeline of language-model agents: re-derivability, how much of what a stage needs it can rebuild from the original problem. Grounding an inspector agent in that problem is worth +0.608 [+0.517, +0.700] to +0.358 over a blind one on four open-weight backbones served with thinking disabled, and on the two Qwen backbones the blind inspector changes no item at all. That head-to-head is exploratory. One deterministic fault enters the first stage, and we re-expose the original problem to $k = 0,\dots,3$ of the downstream stages with agents, items, fault and topology held fixed, on 120 gsm_hard items per arm at temperature zero. Accuracy under fault rises on four of four backbones, from +0.233 to +0.392, the largest Holm-adjusted $p$ being $2.1\times10^{-6}$ . A registered kill test rules out tokens. Blanking every word holds the word slots fixed, and retention tracks the visible fraction on four of four, climbing from 0.221 to 0.692 on the primary. Those two families are confirmatory and everything else here is exploratory. The interaction excludes zero on two of four backbones under the registered pipeline, four of four under a three-stage pipeline, and three of four under full message history, the primary at +0.317. On Llama-3.1-8B the fault carries no detectable cost at any dose, so the other three carry every claim about what a fault costs. Re-derivability also sets what the architecture costs, and no decomposition we measured reliably beats one direct call. With no fault injected the registered pipeline loses to that call by -0.267, -0.125 and -0.317, and on Phi-4 reads +0.058 at $p = 0.118$ , which the test fails to separate from zero. The repair that works is cheap and front-loaded: the first re-grounded stage buys +0.394 of matched retention for +59.8 tokens per item on Qwen3-14B, and the stages after it buy nothing.

Figures & tables

Figure 1: Re-grounding a faulted pipeline; each panel’s title names what it draws. Every dose in (B) also runs without the fault at the same depth and with the same re-grounded stages, and that cell is the ceiling in (C). (C) and (D2) are the two registered confirmatory families; (D1) , (D3) and (E) are exploratory, (D3) on the primary backbone alone. Each fraction in (E) counts backbones out of four under the configuration named beside it. Every arm runs the same fixed 120-item gsm_hard subset (Section 2 ).

Figure 2: Every item in the dose experiment. (a) One column is one problem, one row is one dose

k

, and a row is the no-fault pipeline at that depth marked against its own faulted twin: pale means that row answers the item, saturated means the fault takes it away, a notch means the faulted arm wins it anyway, empty means neither reaches it. The saturated block is the deficit the dose has to close, and it narrows as

k

rises on all four backbones; Llama-3.1-8B has no block to narrow. (b) The two accuracies, filled for the faulted arm, open for its ceiling. (c)

\mathrm{PRR}_{\mathrm{matched}}

with 95% paired item bootstraps.

k=0

against

k=3

is the registered confirmatory contrast;

k=1

and

k=2

are exploratory.

Figure 3: The three controls, and what each one rules out. Left : the occlusion slider, the registered dominant-risk kill test (Family B). The circled point at full visibility is the same pipeline call as the dose arm at

k_{\max}

, so the two families are not independent evidence there. Right : redundancy, grounded against blind aggregators at one, three and five replicas. Bottom : the six cross-family tasks at

k_{\max}

, bars the moving-ceiling ratio on all 24 cells and hollow markers the deployed ratio beside them. The fault is weaker on the non-numeric families and one fault across families is not claimed (Section 9 ). Bands and bars are 95% paired item bootstraps.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Where the items went. Hue is the backbone and nothing else. Inside a raster a grey cell is one item answered right (darker in the two reference rows), and chroma marks what an arm did that its own comparison did not: on the dose ladder, the call that first answers an item; on the inspector rows, what that inspector wins against the strict pipeline; on the single-call row, the items it answers that no dose of re-grounding does. A tab at the top of a cell is an item lost to that same comparison. (a) Qwen3-14B’s 120 items ordered by the first

k

at which its faulted pipeline is right, and none is first right at

k=3

. (b) The reference and inspector rows only, in the hero’s column order, so one column is one problem in all four rasters; the dose ladder is drawn once, in (a). (c) Faulted accuracy against the matched no-fault cell at the same depth, and below it the fraction of its own ceiling each arm keeps. The ceiling moves with

k

, so retention can fall where accuracy rises (the ringed point is the one arm where it does), and on Phi-4 the ceiling is not even monotone. (d) Accuracy against the prompt budget it is bought with; on Phi-4 the prompt gets shorter , by

-3.1\%

. (e) The six families: no fault, and the two faulted doses the panel draws. 15 of the 72 family–backbone–arm cells sit at accuracy 1, and 5 of the 48 faulted cells land above their own no-fault arm. (f) Each arm against its own single call, with paired bootstrap intervals: circles are the dose ladder, the open square and triangle the two rigging controls. The 21 lying wholly below zero span

2.2

3.2\times

the cost; all three exceptions are controls, one of them at

3.4\times

Figure 5: Mechanism, per backbone, exploratory and two-sided. Points are paired differences with 95% paired item bootstrap intervals; the dotted line is zero. Top left : replacing the upstream message with a fixed uninformative line at

k_{\max}

. Top right : the fully re-grounded no-fault pipeline against a single-agent direct solve. Bottom left : the interaction

D(k_{\max})

on the registered pipeline, the re-grounding gain under fault minus the gain without it, which covers zero on Qwen3-14B and on Llama-3.1-8B; Figure 7 carries the same estimand on both measured pipelines. Bottom right : the four arm accuracies the contrasts are built from.

Figure 6: The flat regime on MATH500 and the cost of each re-grounded stage; primary backbone only, exploratory throughout. (a) Each point is one topology: predicted danger from the structural operator against measured retention, per fault family. (b) The tie battery: the operator’s rank correlation minus the one-line heuristic’s, with paired bootstrap intervals; the cell whose fault matches the headline campaign has not run, and the bank injects three other fault families. (c) Retrospective selection fronts. (d) The cost of each re-grounded stage on the same 120 items as the dose ladder, tokens counted as prompt plus completion per item and markers growing with

k

, so the smallest is

k=0

; the headline cost curve is Figure 7 c. Panels (c) and (d) carry their numbers on a line beneath the panel rather than inside it.

Figure 7: The second pipeline, the cost curve at the headline standard, and the occlusion token premise. (a) The fault-by-re-grounding interaction

D

under each pipeline at its own

k_{\max}

, with 95% paired item bootstrap intervals: it excludes zero on two of four backbones under the registered pipeline A (filled circle) and on four of four under pipeline B (open diamond), which is measured and not registered, so every number it produces is exploratory. (b) Pipeline B’s matched dose curve. (c) Matched retention against average prompt and completion tokens, markers growing with

k

. (d) Occlusion prompt tokens by visible fraction; on Phi-4 the masked arm is longer.

Figure 8: The deployed ratio, one shared vertical axis across all four panels.

\mathrm{PRR}_{\mathrm{deployed}}

divides by the deployed strict pipeline at

k=0

, a ceiling that does not move with

k

; the analysis plan calls it an operational number and never a restoration. Bands are 95% paired item bootstraps. The dotted line is the unit ratio: above it the faulty arm scores above the strict no-fault pipeline, which is a statement about the denominator and not about the fault. This panel set is in the appendix, and the matched ratio that every claim rests on is Figure 2 , so the two are never read off one axis.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 838 papers

May 21, 2026

Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines

When a multi-module LLM agent fails, the module most responsible for the failure is not necessarily the best place to intervene. We demonstrate this Diagnostic Paradox empirically: causal analysis consistently identifies the routing module -- which selects which tool to call next -- as the primary bottleneck across three independent agent families. Yet injecting prompt-level correction examples into this module consistently degrades performance, sometimes severely. Patching an upstream query-rewriting module instead reliably improves outcomes. The effect holds with statistical significance on two agent families and directional consistency on a third; alternative repair strategies at the routing module (instruction rewriting, model upgrade) are neutral, confirming that the harm is specific to correction-injection patching. We explain this asymmetry through the Linguistic Contract hypothesis: each downstream module implicitly adapts to its upstream's characteristic error distribution, so correcting the bottleneck breaks this implicit alignment in a way that upstream corrections do not. We operationalize this via a per-agent co-adaptation measure, derived from diagnosis alone, and show it is consistently associated with patching harm across agent families: higher co-adaptation co-occurs with harm, lower with safety. This trend holds across all three agent families, providing preliminary support for the hypothesis beyond a single-agent observation.

Yoon Jeonghun, Kim Dongchan

KAIST (Korea Advanced Institute of Science and Technology), Republic of Korea · NAVER Corp., Seongnam, Republic of Korea · NAVER Corp., Bellevue, WA, USA

LLM Agents Causal Interventions in Language Models Tool-Using Language Agents LLM Agent Reliability

Ask PDF

Oct 7, 2026

Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents

Tool-using agents are usually scored on whether they finish a task while the tools work. Deployments are less forgiving: services time out, endpoints disappear, parameter names change, and results come back well formed but wrong. Prior work has shown that language models over-trust tool outputs that fail silently; we ask how that over-trust plays out across the stages of failure handling in multi-turn agents. Wrapping the executable environments of an established function-calling benchmark in a fault-injection layer, we inject one of four typed faults at a controlled point in the trajectory and record whether the agent notices, changes plan, recovers the task, or repeats itself. Six models from three families, half of them reasoning variants, ran 1,920 trials over 24 multi-step tasks. Agents treat a failure as a problem in 91.3% of trials when the tool returns an explicit error, but in 58.8% of trials when it returns a plausible wrong value, against a 26.8% rate of reporting problems when nothing was wrong. Reasoning models are not better placed: paired against instruct siblings, they notice less (-9.3 points, p < .001) and change plan more (+10.4 points, p < .001), and recovery is unchanged (p = .512). Because agents are stochastic, two fault-free runs of the same task end in the same state only 63.3% of the time; against that baseline, only a missing tool clearly lowers recovery (39.9%), while timeouts, schema drift, and corruption stay within run-to-run variation. After a fault, agents return to the same tool three or more times in a row in up to 22.2% of trials, though strictly identical repeats are rare. A prompt line asking the agent to check each result did not move detection. Agents respond to the error channel rather than to the content of what a tool returns, so failures that stay inside the expected format pass through.

Obada Kraishan

College of Media and Communication Texas Tech University Lubbock, TX, USA

Tool-Using Agents LLM Agent Reliability LLM Agent Evaluation

Ask PDF

Jul 13, 2026

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Aritra Mazumder, Nusrat jahan Lia

University of Utah · University of Dhaka

AI Agent Reliability LLM Agent Evaluation LLM Agent Verification Tool-Using Agents LLM Agent Reliability

Ask PDF

Abstract

LLM Agent Reliability

Explore similar work

Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines

Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP