Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents

Yijun Liao; Fanwei Liang

cs.SEOct 28, 2026

Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents

Authors: Yijun Liao, Fanwei Liang

Organizations: Independent Researcher · South China University of Technology

Abstract

When an LLM agent fails, repair methods choose which action to replace and what to put in its place. They often score candidate repairs with greedy replay, a stronger model, an environment oracle, or a judge that reads the failed log. We ask whether these evaluators pick repairs that help the agent that will actually execute them. Holding failed trajectories and candidate single-step repairs fixed, we label every candidate with independent rollouts of the deployed policy and measure how much deployment value each evaluator's choice loses. On failures not screened for repair value, pre-specified tests show that greedy replay loses 0.08-0.13 in expected net success gain in ALFWorld and 0.06-0.13 in net reward in WebShop, relative to selection with the deployed agent's own rollouts. At an equal rollout budget, selecting repairs for Qwen2.5-7B with Qwen2.5-14B rollouts loses 0.25 in ALFWorld and 0.07 in WebShop, twenty 14B rollouts per candidate are worth no more than two of the 7B agent's own, and fresh rollouts confirm that the two models prefer opposite repairs on selected disagreement cases. Between similar-sized models from different families, the losses appear in both directions in both environments (0.07-0.17). The scoring executor also changes repair-location labels and method rankings. Repairs should be verified with the executor that will run them, and repair labels should name their executor.

Figures & tables

Figure 1: Which repair is best depends on who executes it. One failed ALFWorld trajectory of the deployed Qwen2.5-7B agent; each point is one of 18 candidate repairs at three steps, placed by its net gain (change in success probability over keeping the original action) when the 7B agent (30 rollouts per branch, horizontal) or Qwen2.5-14B (10 rollouts, vertical) executes the rest. The same action, going to the fridge, is the 7B agent’s best repair at step 0 but the 14B model’s at step 20; at step 0 the 14B model already succeeds without help, so the repair gains it nothing. Qwen3.8-flash succeeds from every branch (all net gains 0). An illustrative case from a screened pool.

Evaluator	Practice it stands for	Departs in
Deployed executor, 1–5 rollouts	few-rollout verification	budget
Same model at temperature 0.3	close decoding proxy	target
Greedy replay (temperature 0)	deterministic replay	target
Solvability oracle $F$	oracle / planner checks	target
Oracle, ties broken by slack	strongest environment signal	target
Planner action at earliest position	planner-based repair	target

Table 1: Evaluators compared against matched selection. Target : the evaluator estimates a different quantity than the deployed executor’s net gain. Budget : same target, fewer samples or a decision rule applied to the deployed executor’s own rollouts. The published decision rules re-implement only the verification step of each method within our fixed-candidate protocol; they are not full reproductions.

Pool (deployed)	Failures	Candidates	Rollouts per branch
Unscreened (primary)
ALFWorld, 7B	60/82	L, P, R (6.1)	7B 10, 14B 20, Llama 10
ALFWorld, Llama	60/109	L, P, R (6.0)	Llama 10, 7B 10
WebShop, 7B	60/72	C, L, P, R (11.5)	7B 10, 14B 20, Llama 10
WebShop, Llama	60/86	C, L, P, R (10.1)	Llama 10, 7B 10
Screened (controlled comparisons)

Table 2: Unscreened : the first failures of fresh episodes, not screened by repair value. Screened : failures in which screening rollouts showed some alternative helping. Candidates (mean per position): L, two proposals from each of three LLMs; P, privileged action (the planner in ALFWorld); R, random action; C, every clickable element. Rollouts per stochastic branch for each executor, plus one greedy rollout each; in the gridworld, also 20 rollouts of the 7B agent at temperature 0.3.

Figure 2: Deployment value lost by substitute evaluators. Each row is a paired difference: the deployed agent’s value of repairs selected with its own rollouts minus that of the substitute’s, both scored on its held-out rollouts. Unscreened pools (every pre-specified primary test, all Holm-confirmed): own selection with five rollouts per candidate versus greedy replay (diamonds) or five rollouts of another model (circles); nonparametric 90% intervals. Screened pools: ten rollouts per candidate on both sides; bias-corrected 90% intervals. ALFWorld and gridworld values are success probabilities, WebShop values rewards.

Same target, smaller budget or another rule
Evaluator	Gridworld (34)	ALFWorld (30)
Random (pos. control)	0.66 [0.57, 0.74]	0.75 [0.69, 0.81]
Deployed, 3 rollouts	0.11 [0.06, 0.16]	0.07 [0.03, 0.11]
Deployed, 10 rollouts	0.01 [ $-$ 0.02, 0.04]	0.02 [ $-$ 0.02, 0.05]
2-of-3 acceptance	0.32 [0.25, 0.39]	0.23 [0.16, 0.30]
Single-replay flip	0.43 [0.36, 0.50]	0.28 [0.22, 0.35]

Table 3: Selection loss on the screened pools, relative to matched selection with all of the deployed agent’s rollouts (20 per fold in the gridworld, 15 in ALFWorld; deployment net gain 0.662 and 0.851). Larger is worse; bias-corrected 90% intervals. The first block keeps the target and changes the budget or the decision rule; the published rules use the deployed agent’s own rollouts, so their losses come from the rule, not from mismatch. The second block changes the target. Other executors are compared at equal budget in Figure 2 .

Figure 3: More rollouts do not fix the executor, and preferences reverse on fresh rollouts. (a) Unscreened 7B ALFWorld pool: net gain for the 7B agent of repairs selected with

k

rollouts per candidate from each executor (the deployed agent’s own selection is cross-fitted, hence

k\leq 5

); bands and bars: 90% intervals. (b) Fresh rollouts (30 per branch and executor) on the 37 trajectories in which the 7B and 14B models’ best candidates differed in earlier rollouts (

r_{a}

: 7B’s,

r_{b}

: 14B’s) and on 15 control trajectories in which they agreed (

r_{b}

: 7B’s second best); small marks are trajectories, large marks group means with 90% intervals. Shaded: opposite preferences; dashed: equal preference.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Pool	Pre-specified	Fresh	Reported as
C1–C3; C4–C6	screened gridworld	yes	F	confirmatory (Holm, 3); secondary
Greedy with escape (E1, E2)	screened gridworld	yes, before running	R	secondary
B2–B4; B5–B9, informed judge	screened ALFWorld	yes	F	confirmatory (Holm, 3); secondary
Other executors X1, X2 (14B, flash)	both screened	yes, before running	R	secondary
Published decision rules	both screened	no	—	exploratory
Labels and rankings ( Section 6 )	both screened	no	—	exploratory

Appendix

Table 4: Status of every analysis. Pre-specified: written into the dated plan before the data it uses were collected or computed. Fresh: new failures (F) or new rollouts on existing branches (R). Confirmatory tests are Holm-corrected within the family shown; “secondary” tests use a lower bound above 0 without correction.

	Contrast	Estimate	Pre-specified	Bias-corrected	Rule	Decision
Screened gridworld (34 trajectories)
PC	matched $-$ random	0.656	[0.548, 0.709]	[0.574, 0.735]	lower $>0.10$	pass
C1	matched $-$ random solvable action, irreversible point	0.308	[0.193, 0.371]	[0.218, 0.396]	Holm	confirmed
C2	matched $-$ greedy replay	0.225	[0.136, 0.284]	[0.152, 0.300]	Holm	confirmed
C3	matched $-$ solvability oracle	0.186	[0.090, 0.235]	[0.115, 0.261]	Holm	confirmed
C4	matched $-$ 3 rollouts	0.109	[0.044, 0.145]	[0.060, 0.161]	lower $>0$	detected

Appendix

Table 5: All pre-specified tests. Estimates are differences in deployment net gain (first minus second; reward in WebShop). Screened pools: pre-specified (uncentered) and bias-corrected 90% intervals. Unscreened pools: nonparametric 90% intervals. Holm: primary test, Holm-corrected within its family at one-sided

\alpha=0.05

(screened pools: largest

p

-value 0.050 for B4 under the pre-specified bootstrap, 0.004 after correction; unscreened: P1–P2, H1–H3, and the five WebShop tests form three families, largest

p

-value 0.0005). Detected: lower bound

>0

; within margin: upper bound

<\delta=0.10

. Every decision is the same under both intervals.

Evaluator	Gridworld	ALFWorld
Random (positive control)	0.51 [0.43, 0.58]	0.74 [0.68, 0.78]
Deployed agent, 3 rollouts	0.10 [0.06, 0.14]	0.05 [0.02, 0.07]
Deployed agent, 5 rollouts	0.08 [0.04, 0.12]	0.02 [ $-$ 0.00, 0.04]
Deployed agent, 10 rollouts	0.02 [ $-$ 0.02, 0.05]	0.01 [ $-$ 0.02, 0.02]
Same model, temperature 0.3	0.02 [ $-$ 0.03, 0.08]	—
Greedy replay	0.19 [0.12, 0.25]	0.14 [0.08, 0.20]

Appendix

Table 6: Selection loss on the screened pools under the post-repair success objective (bias-corrected 90% intervals). Matched selection reaches post-repair success 0.72 (gridworld) and 0.97 (ALFWorld). The acceptance-based published rules are omitted because they have no natural post-repair value when they decline to repair.

	ALFWorld, 7B pool			ALFWorld, Llama pool
$k$	7B	14B	Llama	Llama	7B
1	0.63	0.38	0.55	0.68	0.57
2	0.77	0.49	0.67	0.82	0.68
3	0.80	0.53	0.72	0.85	0.74
5	0.82	0.57	0.75	0.87	0.77
10	—	0.60	0.78	—	0.79

Appendix

Table 7: Budget curves on the unscreened pools: deployment net gain of the repair selected with

k

rollouts per candidate from each executor, scored on the deployed agent (G: greedy replay). ALFWorld: success probability; WebShop: reward.

	14B, contested	14B, control	Llama, contested
Trajectories	37	15	34
$\Delta_{\mathrm{7B}}$	0.33 [0.23, 0.44]	0.13 [0.05, 0.23]	0.18 [0.10, 0.27]
$\Delta_{\mathrm{other}}$	$-$ 0.18 [ $-$ 0.25, $-$ 0.11]	0.12 [0.05, 0.20]	$-$ 0.02 [ $-$ 0.07, 0.03]
Difference	0.51 [0.38, 0.64]	0.01 [ $-$ 0.08, 0.11]	0.20 [0.12, 0.29]
Sig. opposite	8	0	2
Sig. aligned	0	3	3

Appendix

Table 8: Reversal checks on the unscreened 7B pool (only fresh rollouts; 30 per branch and executor).

\Delta_{e}=R_{e}(r_{a})-R_{e}(r_{b})

with

R_{e}

the net gain of Equation 1 , where

r_{a}

is the deployed 7B agent’s best candidate and

r_{b}

the other executor’s in the earlier rollouts; difference

=\Delta_{\mathrm{7B}}-\Delta_{\mathrm{other}}

. Control: trajectories where both executors’ best candidate was

r_{a}

, with

r_{b}

the 7B agent’s second best. Sig. opposite / aligned: both

\Delta

s have 90% intervals excluding 0, with opposite / equal signs.

	Deployed Qwen2.5-7B		Deployed Llama-3.1-8B
Contrast	Reward	Success	Reward	Success †
Positive control: matched $-$ random	0.163 [0.128, 0.200]	0.169 [0.114, 0.229]	0.314 [0.258, 0.373]	0.149 [0.090, 0.210]
Matched $-$ greedy replay (P1w; H3w)	0.061 [0.036, 0.086]	0.040 [0.014, 0.069]	0.131 [0.083, 0.181]	0.094 [0.055, 0.136]
Other model, same budget (H1w; H2w)	0.081 [0.046, 0.114]	0.075 [0.042, 0.112]	0.166 [0.117, 0.217]	0.080 [0.040, 0.124]
Matched $-$ privileged action, earliest (S1w)	0.144 [0.100, 0.190]	0.161 [0.094, 0.230]	0.248 [0.190, 0.305]	0.094 [0.040, 0.156]
ALFWorld-style: matched $-$ greedy	0.057 [0.032, 0.083]	0.049 [0.021, 0.082]	0.088 [0.057, 0.124]	0.073 [0.040, 0.110]

Appendix

Table 9: WebShop, unscreened pools (60 trajectories each): differences in net reward gain (primary) and net success gain (secondary), nonparametric 90% intervals. “Other model”: deployed agent’s own selection minus the other deployed model’s, five rollouts per candidate each. ALFWorld-style: only LLM-proposed, privileged, and random candidates, as in the ALFWorld pools. LLM-proposed: candidates proposed by at least one LLM. † The success-based positive control of the Llama pool falls just short of 0.10, so its success readouts are descriptive only.

Figure 4: Screened pools: selection loss against the number of deployed-agent rollouts per trajectory, when every candidate receives

k

rollouts (blue) or only the candidate preferred by the environment signal at each position does (orange). Shaded: 90% bootstrap intervals over trajectories. Settings were fixed on the gridworld and applied unchanged to ALFWorld.

Selection	Net gain	Rollouts
LLM locates and proposes; 2-of-3 check	0.32 [0.22, 0.40]	6
no check (first proposal)	0.22 [0.14, 0.30]	0
greedy check (first greedy success)	0.31 [0.22, 0.40]	2 (greedy)
more successes than the original action	0.32 [0.23, 0.40]	9
better proposal in hindsight	0.37 [0.29, 0.46]	—
LLM screen (first proposal per position), $k=1$ / 5	0.24 / 0.27	6 / 30

Appendix

Table 10: The LLM-located, verified pipeline of Section 7 and comparisons on the same 60 unscreened 7B ALFWorld failures (deployment net gain; nonparametric 90% intervals). Rollouts: deployed-agent rollouts per trajectory used for selection. The last three rows select among the fixed candidates of Section 3 at three positions.

Selection	Net gain	Rollouts
Human reads the log (one annotator)	0.335 [0.247, 0.423]	0
Random candidate	0.115 [0.099, 0.130]	0
LLM screen (first flash proposal), $k=5$	0.270 [0.187, 0.357]	30
Planner action, earliest position (privileged)	0.682 [0.605, 0.757]	0
Greedy replay	0.687 [0.618, 0.754]	21
Matched selection (5 rollouts)	0.822 [0.762, 0.878]	107

Appendix

Table 11: A human reader on the 60 unscreened 7B ALFWorld failures: net gain for the 7B agent, scored on all ten of its rollouts per branch (paired trajectory bootstrap, 10,000 replicates, 90% intervals). Rollouts: deployed-agent rollouts per trajectory used for selection.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Localizing Emergent Failures in Agentic AI: Recovering Minimal Repair Families via Counterfactual Replay
Aug 29, 2026Bingjie Li, Yumeng Song, Zhongming Yao +1Agent Failure Analysis Multi-Agent LLM Systems
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science, Aalborg University, Aalborg, Denmark · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents
Sep 14, 2026Mengyi Deng, Xin Li, Duyi Pan +4LLM Agent Self-Improvement LLM Agent Skill Learning
1Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China · 2The Hong Kong University of Science and Technology, Hong Kong SAR
Structured Feedback Improves Repair in an LLM Agent Loop
Jul 15, 2026Jaideep Ray, Ankit GoyalLLM Agent Workflow Optimization LLM Agent Evaluation
Independent Researcher

Aug 29, 2026J/K move · Enter open · S save

Localizing Emergent Failures in Agentic AI: Recovering Minimal Repair Families via Counterfactual Replay

Bingjie Li, Yumeng Song, Zhongming Yao, Tianyi Li

School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science, Aalborg University, Aalborg, Denmark · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Agent Failure Analysis Multi-Agent LLM Systems

Open paper AskPDF arXiv

Failures in agentic AI systems can arise from interactions among messages exchanged by multiple large language model (LLM) agents. Pointwise attribution cannot distinguish a jointly necessary repair from alternative singleton repairs. We formulate Minimal Repair Family Recovery (MRFR): recovering all inclusion-minimal event sets whose counterfactual replay restores task success within a declared size bound. We propose Graph-Constrained Joint Replay (GCJR), which slices failure-relevant events from an execution dependency graph, constructs graph-feasible singleton and pair candidates, and verifies them by replay with paired clean counterparts. For fixed replay outcomes, GCJR is exact within its declared graph domain. On 90 in-scope cases from a 120-DAG controlled benchmark, GCJR achieves 1.000 Family Exact Match while reducing mean replay calls from 56.3 to 25.3 (55.1%) relative to exhaustive search. On a 24-case, four-agent LLM pilot, it again achieves 1.000 Family Exact Match and reduces mean model calls from 21.0 to 10.0 (52.4%); single-event replay misses jointly necessary repairs.

Abstract

LLM Agent Evaluation

Explore similar work

Localizing Emergent Failures in Agentic AI: Recovering Minimal Repair Families via Counterfactual Replay

RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents

Structured Feedback Improves Repair in an LLM Agent Loop