When an LLM agent fails, repair methods choose which action to replace and what to put in its place. They often score candidate repairs with greedy replay, a stronger model, an environment oracle, or a judge that reads the failed log. We ask whether these evaluators pick repairs that help the agent that will actually execute them. Holding failed trajectories and candidate single-step repairs fixed, we label every candidate with independent rollouts of the deployed policy and measure how much deployment value each evaluator's choice loses. On failures not screened for repair value, pre-specified tests show that greedy replay loses 0.08-0.13 in expected net success gain in ALFWorld and 0.06-0.13 in net reward in WebShop, relative to selection with the deployed agent's own rollouts. At an equal rollout budget, selecting repairs for Qwen2.5-7B with Qwen2.5-14B rollouts loses 0.25 in ALFWorld and 0.07 in WebShop, twenty 14B rollouts per candidate are worth no more than two of the 7B agent's own, and fresh rollouts confirm that the two models prefer opposite repairs on selected disagreement cases. Between similar-sized models from different families, the losses appear in both directions in both environments (0.07-0.17). The scoring executor also changes repair-location labels and method rankings. Repairs should be verified with the executor that will run them, and repair labels should name their executor.
Figures & tables
Figure 1: Which repair is best depends on who executes it. One failed ALFWorld trajectory of the deployed Qwen2.5-7B agent; each point is one of 18 candidate repairs at three steps, placed by its net gain (change in success probability over keeping the original action) when the 7B agent (30 rollouts per branch, horizontal) or Qwen2.5-14B (10 rollouts, vertical) executes the rest. The same action, going to the fridge, is the 7B agent’s best repair at step 0 but the 14B model’s at step 20; at step 0 the 14B model already succeeds without help, so the repair gains it nothing. Qwen3.8-flash succeeds from every branch (all net gains 0). An illustrative case from a screened pool.
Evaluator
Practice it stands for
Departs in
Deployed executor, 1–5 rollouts
few-rollout verification
budget
Same model at temperature 0.3
close decoding proxy
target
Greedy replay (temperature 0)
deterministic replay
target
Solvability oracle F
oracle / planner checks
target
Oracle, ties broken by slack
strongest environment signal
target
Planner action at earliest position
planner-based repair
target
Table 1: Evaluators compared against matched selection. Target : the evaluator estimates a different quantity than the deployed executor’s net gain. Budget : same target, fewer samples or a decision rule applied to the deployed executor’s own rollouts. The published decision rules re-implement only the verification step of each method within our fixed-candidate protocol; they are not full reproductions.
Pool (deployed)
Failures
Candidates
Rollouts per branch
Unscreened (primary)
ALFWorld, 7B
60/82
L, P, R (6.1)
7B 10, 14B 20, Llama 10
ALFWorld, Llama
60/109
L, P, R (6.0)
Llama 10, 7B 10
WebShop, 7B
60/72
C, L, P, R (11.5)
7B 10, 14B 20, Llama 10
WebShop, Llama
60/86
C, L, P, R (10.1)
Llama 10, 7B 10
Screened (controlled comparisons)
Table 2: Unscreened : the first failures of fresh episodes, not screened by repair value. Screened : failures in which screening rollouts showed some alternative helping. Candidates (mean per position): L, two proposals from each of three LLMs; P, privileged action (the planner in ALFWorld); R, random action; C, every clickable element. Rollouts per stochastic branch for each executor, plus one greedy rollout each; in the gridworld, also 20 rollouts of the 7B agent at temperature 0.3.
Figure 2: Deployment value lost by substitute evaluators. Each row is a paired difference: the deployed agent’s value of repairs selected with its own rollouts minus that of the substitute’s, both scored on its held-out rollouts. Unscreened pools (every pre-specified primary test, all Holm-confirmed): own selection with five rollouts per candidate versus greedy replay (diamonds) or five rollouts of another model (circles); nonparametric 90% intervals. Screened pools: ten rollouts per candidate on both sides; bias-corrected 90% intervals. ALFWorld and gridworld values are success probabilities, WebShop values rewards.
Evaluator
Gridworld (34)
ALFWorld (30)
Random (pos. control)
0.66 [0.57, 0.74]
0.75 [0.69, 0.81]
Same target, smaller budget or another rule
Deployed, 3 rollouts
0.11 [0.06, 0.16]
0.07 [0.03, 0.11]
Deployed, 10 rollouts
0.01 [ − 0.02, 0.04]
0.02 [ − 0.02, 0.05]
2-of-3 acceptance
0.32 [0.25, 0.39]
0.23 [0.16, 0.30]
Single-replay flip
0.43 [0.36, 0.50]
0.28 [0.22, 0.35]
Table 3: Selection loss on the screened pools, relative to matched selection with all of the deployed agent’s rollouts (20 per fold in the gridworld, 15 in ALFWorld; deployment net gain 0.662 and 0.851). Larger is worse; bias-corrected 90% intervals. The first block keeps the target and changes the budget or the decision rule; the published rules use the deployed agent’s own rollouts, so their losses come from the rule, not from mismatch. The second block changes the target. Other executors are compared at equal budget in Figure 2 .
Figure 3: More rollouts do not fix the executor, and preferences reverse on fresh rollouts. (a) Unscreened 7B ALFWorld pool: net gain for the 7B agent of repairs selected with k rollouts per candidate from each executor (the deployed agent’s own selection is cross-fitted, hence k≤5 ); bands and bars: 90% intervals. (b) Fresh rollouts (30 per branch and executor) on the 37 trajectories in which the 7B and 14B models’ best candidates differed in earlier rollouts ( ra : 7B’s, rb : 14B’s) and on 15 control trajectories in which they agreed ( rb : 7B’s second best); small marks are trajectories, large marks group means with 90% intervals. Shaded: opposite preferences; dashed: equal preference.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Analysis
Pool
Pre-specified
Fresh
Reported as
C1–C3; C4–C6
screened gridworld
yes
F
confirmatory (Holm, 3); secondary
Greedy with escape (E1, E2)
screened gridworld
yes, before running
R
secondary
B2–B4; B5–B9, informed judge
screened ALFWorld
yes
F
confirmatory (Holm, 3); secondary
Other executors X1, X2 (14B, flash)
both screened
yes, before running
R
secondary
Published decision rules
both screened
no
—
exploratory
Labels and rankings ( Section 6 )
both screened
no
—
exploratory
Appendix
Table 4: Status of every analysis. Pre-specified: written into the dated plan before the data it uses were collected or computed. Fresh: new failures (F) or new rollouts on existing branches (R). Confirmatory tests are Holm-corrected within the family shown; “secondary” tests use a lower bound above 0 without correction.
Contrast
Estimate
Pre-specified
Bias-corrected
Rule
Decision
Screened gridworld (34 trajectories)
PC
matched − random
0.656
[0.548, 0.709]
[0.574, 0.735]
lower >0.10
pass
C1
matched − random solvable action, irreversible point
0.308
[0.193, 0.371]
[0.218, 0.396]
Holm
confirmed
C2
matched − greedy replay
0.225
[0.136, 0.284]
[0.152, 0.300]
Holm
confirmed
C3
matched − solvability oracle
0.186
[0.090, 0.235]
[0.115, 0.261]
Holm
confirmed
C4
matched − 3 rollouts
0.109
[0.044, 0.145]
[0.060, 0.161]
lower >0
detected
Appendix
Table 5: All pre-specified tests. Estimates are differences in deployment net gain (first minus second; reward in WebShop). Screened pools: pre-specified (uncentered) and bias-corrected 90% intervals. Unscreened pools: nonparametric 90% intervals. Holm: primary test, Holm-corrected within its family at one-sided α=0.05 (screened pools: largest p -value 0.050 for B4 under the pre-specified bootstrap, 0.004 after correction; unscreened: P1–P2, H1–H3, and the five WebShop tests form three families, largest p -value 0.0005). Detected: lower bound >0 ; within margin: upper bound <δ=0.10 . Every decision is the same under both intervals.
Evaluator
Gridworld
ALFWorld
Random (positive control)
0.51 [0.43, 0.58]
0.74 [0.68, 0.78]
Deployed agent, 3 rollouts
0.10 [0.06, 0.14]
0.05 [0.02, 0.07]
Deployed agent, 5 rollouts
0.08 [0.04, 0.12]
0.02 [ − 0.00, 0.04]
Deployed agent, 10 rollouts
0.02 [ − 0.02, 0.05]
0.01 [ − 0.02, 0.02]
Same model, temperature 0.3
0.02 [ − 0.03, 0.08]
—
Greedy replay
0.19 [0.12, 0.25]
0.14 [0.08, 0.20]
Appendix
Table 6: Selection loss on the screened pools under the post-repair success objective (bias-corrected 90% intervals). Matched selection reaches post-repair success 0.72 (gridworld) and 0.97 (ALFWorld). The acceptance-based published rules are omitted because they have no natural post-repair value when they decline to repair.
ALFWorld, 7B pool
ALFWorld, Llama pool
k
7B
14B
Llama
Llama
7B
1
0.63
0.38
0.55
0.68
0.57
2
0.77
0.49
0.67
0.82
0.68
3
0.80
0.53
0.72
0.85
0.74
5
0.82
0.57
0.75
0.87
0.77
10
—
0.60
0.78
—
0.79
Appendix
Table 7: Budget curves on the unscreened pools: deployment net gain of the repair selected with k rollouts per candidate from each executor, scored on the deployed agent (G: greedy replay). ALFWorld: success probability; WebShop: reward.
14B, contested
14B, control
Llama, contested
Trajectories
37
15
34
Δ7B
0.33 [0.23, 0.44]
0.13 [0.05, 0.23]
0.18 [0.10, 0.27]
Δother
− 0.18 [ − 0.25, − 0.11]
0.12 [0.05, 0.20]
− 0.02 [ − 0.07, 0.03]
Difference
0.51 [0.38, 0.64]
0.01 [ − 0.08, 0.11]
0.20 [0.12, 0.29]
Sig. opposite
8
0
2
Sig. aligned
0
3
3
Appendix
Table 8: Reversal checks on the unscreened 7B pool (only fresh rollouts; 30 per branch and executor). Δe=Re(ra)−Re(rb) with Re the net gain of Equation 1 , where ra is the deployed 7B agent’s best candidate and rb the other executor’s in the earlier rollouts; difference =Δ7B−Δother . Control: trajectories where both executors’ best candidate was ra , with rb the 7B agent’s second best. Sig. opposite / aligned: both Δ s have 90% intervals excluding 0, with opposite / equal signs.
Deployed Qwen2.5-7B
Deployed Llama-3.1-8B
Contrast
Reward
Success
Reward
Success †
Positive control: matched − random
0.163 [0.128, 0.200]
0.169 [0.114, 0.229]
0.314 [0.258, 0.373]
0.149 [0.090, 0.210]
Matched − greedy replay (P1w; H3w)
0.061 [0.036, 0.086]
0.040 [0.014, 0.069]
0.131 [0.083, 0.181]
0.094 [0.055, 0.136]
Other model, same budget (H1w; H2w)
0.081 [0.046, 0.114]
0.075 [0.042, 0.112]
0.166 [0.117, 0.217]
0.080 [0.040, 0.124]
Matched − privileged action, earliest (S1w)
0.144 [0.100, 0.190]
0.161 [0.094, 0.230]
0.248 [0.190, 0.305]
0.094 [0.040, 0.156]
ALFWorld-style: matched − greedy
0.057 [0.032, 0.083]
0.049 [0.021, 0.082]
0.088 [0.057, 0.124]
0.073 [0.040, 0.110]
Appendix
Table 9: WebShop, unscreened pools (60 trajectories each): differences in net reward gain (primary) and net success gain (secondary), nonparametric 90% intervals. “Other model”: deployed agent’s own selection minus the other deployed model’s, five rollouts per candidate each. ALFWorld-style: only LLM-proposed, privileged, and random candidates, as in the ALFWorld pools. LLM-proposed: candidates proposed by at least one LLM. † The success-based positive control of the Llama pool falls just short of 0.10, so its success readouts are descriptive only.
Figure 4: Screened pools: selection loss against the number of deployed-agent rollouts per trajectory, when every candidate receives k rollouts (blue) or only the candidate preferred by the environment signal at each position does (orange). Shaded: 90% bootstrap intervals over trajectories. Settings were fixed on the gridworld and applied unchanged to ALFWorld.
Selection
Net gain
Rollouts
LLM locates and proposes; 2-of-3 check
0.32 [0.22, 0.40]
6
no check (first proposal)
0.22 [0.14, 0.30]
0
greedy check (first greedy success)
0.31 [0.22, 0.40]
2 (greedy)
more successes than the original action
0.32 [0.23, 0.40]
9
better proposal in hindsight
0.37 [0.29, 0.46]
—
LLM screen (first proposal per position), k=1 / 5
0.24 / 0.27
6 / 30
Appendix
Table 10: The LLM-located, verified pipeline of Section 7 and comparisons on the same 60 unscreened 7B ALFWorld failures (deployment net gain; nonparametric 90% intervals). Rollouts: deployed-agent rollouts per trajectory used for selection. The last three rows select among the fixed candidates of Section 3 at three positions.
Selection
Net gain
Rollouts
Human reads the log (one annotator)
0.335 [0.247, 0.423]
0
Random candidate
0.115 [0.099, 0.130]
0
LLM screen (first flash proposal), k=5
0.270 [0.187, 0.357]
30
Planner action, earliest position (privileged)
0.682 [0.605, 0.757]
0
Greedy replay
0.687 [0.618, 0.754]
21
Matched selection (5 rollouts)
0.822 [0.762, 0.878]
107
Appendix
Table 11: A human reader on the 60 unscreened 7B ALFWorld failures: net gain for the 7B agent, scored on all ten of its rollouts per branch (paired trajectory bootstrap, 10,000 replicates, 90% intervals). Rollouts: deployed-agent rollouts per trajectory used for selection.
Failures in agentic AI systems can arise from interactions among messages exchanged by multiple large language model (LLM) agents. Pointwise attribution cannot distinguish a jointly necessary repair from alternative singleton repairs. We formulate Minimal Repair Family Recovery (MRFR): recovering all inclusion-minimal event sets whose counterfactual replay restores task success within a declared size bound. We propose Graph-Constrained Joint Replay (GCJR), which slices failure-relevant events from an execution dependency graph, constructs graph-feasible singleton and pair candidates, and verifies them by replay with paired clean counterparts. For fixed replay outcomes, GCJR is exact within its declared graph domain. On 90 in-scope cases from a 120-DAG controlled benchmark, GCJR achieves 1.000 Family Exact Match while reducing mean replay calls from 56.3 to 25.3 (55.1%) relative to exhaustive search. On a 24-case, four-agent LLM pilot, it again achieves 1.000 Family Exact Match and reduces mean model calls from 21.0 to 10.0 (52.4%); single-event replay misses jointly necessary repairs.
Bingjie Li, Yumeng Song, Zhongming Yao +1
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science, Aalborg University, Aalborg, Denmark · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Language agents increasingly rely on reusable skills, but post-failure repair is often handled by opaque one-shot reflection: a model generates a skill patch without explicitly maintaining how failure explanations relate to candidate repairs or how unsuccessful retests should influence later edits. We introduce RESKILL, a structured repair framework that maintains an explicit repair state across repair rounds. Given a failed rollout, the framework links failure hypotheses to candidate skill patches, selects local repairs through coverage-based attribution, retests the edited skill set in the environment, and uses retest outcomes to guide subsequent repair updates. The language model supplies structured repair factors, while the repair procedure records them, compares local skill patches by how well they address active failure explanations, and carries unsuccessful retest outcomes into later repair rounds. We evaluate RESKILL on ALFWorld and TextCraft across three model sizes under fixed repair budgets. RESKILL obtains the strongest final success in all six benchmark-model settings, improving average final success by 3.7 percentage points over direct repair and 3.3 points over hypothesis-conditioned repair. These results suggest that explicit attribution alone is insufficient; durable improvement emerges when attribution is integrated with repair selection and persistent retest-conditioned update.
Mengyi Deng, Xin Li, Duyi Pan +4
1Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China · 2The Hong Kong University of Science and Technology, Hong Kong SAR
LLM agents often retry after external validation rejects a candidate, but the interface between validation and the next model call remains underspecified. We introduce VeriHarness, a code-controlled agent loop in which models generate candidates while external validators control acceptance, budgets, and traces. We use it to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games under a four-call cap, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 points). Ablations locate most of the gain in the admissible alternatives: feedback containing only the location and observed value remains near the raw diagnostic baseline. Presenting the complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. The ordering persists across the tested call budgets and one sampled-decoding setting.