Published explanations of why long-horizon agents fail are commonly taxonomies labeled over completed trajectories: a judge reads a transcript and names the failure. We argue that a claim attributing an agent's behavior to its history should carry intervention-verified evidence, per instance or via a validated estimator: freeze an executable mid-trajectory state, alter only the history the agent sees, and continue - executed rather than inferred from the transcript, at a state whose restoration has been measured, with a replay certificate matched to the estimand. Each of the four manipulation classes we name has a precedent, and several methods already execute interventions from checkpoints, re-executed prefixes or recorded conversation states (for example DoVer, REFLECT, Min et al., Who&When Pro, Causal Agent Replay), but none reports a measured replay certificate. Two cautionary measurements, neither applying the standard, follow: a carrier taxonomy whose "concentration" could not be separated from benchmark construction (adjusted odds ratio 1.15, interval crossing 1), and hash-identical snapshots that, restored into identically seeded processes, agreed with one another in 24 of 24 states while differing from the live continuation in 9 and 20 of 24. Restoring the captured position of a random stream made all 24 agree with the live run.
Figures & tables
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
service_response
local_file_state
Task pairs (of 266 labeled)
225 (84.6%)
41 (15.4%)
Labeled attack trajectories
13,500
2,460
Attack-positive trajectories (of 9,199)
7,810
1,389
Share of unsafe actions
0.8490 [0.8008, 0.8940]
0.1510 [0.1060, 0.1992]
Adjusted odds ratio
1.1484 [0.6275, 2.0160]
1 (reference)
First-exposure recovery
13,048/13,500 = 96.65%
1,705/2,460 = 69.31%
Appendix
Table 1: Measurement 3.1, analyzed carrier estimates. Intervals are 95% task-clustered bootstrap intervals at seed 20260901 (within-scenario dependence between tasks is not represented), with 5,000 replicates for shares and concentration indices and 2,000 for the adjusted model. Attack method is nearly collinear with carrier (only context injection contains both classes: 195 service, 12 local pairs), so the adjusted odds ratio is estimated under limited common support. The adjusted odds ratio is a logistic carrier contrast with fixed effects for 15 risk-behavior classes and the benchmark’s attack method, fit on all 15,960 labeled rows, with 0 failed bootstrap fits.
Analysis
Result
Leave-one-model-out, harness fixed
15/15 computable folds; τ -b median 1.000, minimum −1.000 (14 folds at +1 , one at −1 )
Leave-one-harness-out, model fixed
6/6 computable folds; τ -b median 1.000, minimum 1.000
Per-configuration share matrix
20 configurations × 2 classes, 40 cells
Log-evidence-only proxy
7,150 events; service share 0.8236; adjusted OR 0.8973 [0.4715, 1.7043]
Clean-twin negative control
benign-task failure 13.49% (607/4,500) for service twins vs. 8.78% (72/820) for local twins; 5,320 rows, 266 pairs
Drop configurations below 200 usable
all 20 had 798 usable labeled attack trajectories; 0 dropped, estimates unchanged
Appendix
Table 2: Replication folds and sensitivity analyses; same seed, 2,000 replicates for the proxy model. Leave-one-out replication ran on behavior- and attack-method-adjusted carrier log-odds rankings along both released axes; with two surviving classes τ -b can take only ±1 .
Exp.
Rate
Machinery
Comparison / metric
Div./24
Fraction [95% CI]
B1
0.05
naive
five-clone / observable
17
0.71 [0.54, 0.88]
B1
0.20
naive
five-clone / observable
22
0.92 [0.79, 1.00]
B2
0.05
naive
live vs. restored / observable
11
0.46 [0.25, 0.67]
B2
0.05
naive
live vs. restored / failure pattern
11
0.46 [0.25, 0.67]
B2
0.05
naive
live vs. restored / final hash
11
0.46 [0.25, 0.67]
B2
0.05
naive
live vs. restored / final grader
0
0.00 [0.00, 0.00]
Appendix
Table 3: Replay divergence in the unseeded experiment, all measured conditions (B1: five-clone experiment; B2: live-vs-restored and live-vs-live experiment; the seeded replication is in Table 4 ). “Naive” restores the serialized state only; “captured” additionally restores the random-stream position; live-vs-live rows restore nothing and are the unseeded re-execution baseline of the run named in the Machinery column. 10,000 state resamples per interval, seed 20260901; every fraction and interval endpoint is an integer multiple of 1/24 and is shown to two decimals. Intervals are the released state-resampling bootstrap intervals over the 24 factorial cells and are descriptive: the cells are not a random sample of the census, so no population coverage is claimed; at a zero count the bootstrap is degenerate (a binomial Clopper–Pearson 95% upper limit would be 0.14 [ 8 ] , under a sampling model the design does not supply).
Arm
0.05
0.20
Two fresh runs, same seed (re-execution baseline)
0/24
0/24
State rebuilt by replaying the recorded prefix (seeded)
Byte-identical snapshot restored, seeded process, one seed per state
9/24
20/24
Byte-identical snapshot restored, unseeded process (separate run)
14/24
21/24
Snapshot with captured random-stream state restored
0/24
0/24
Appendix
Table 4: Seeded replication: states (of 24) whose continuation diverged from the live continuation, by arm and tool-failure rate. Counts only; the 24 states are factorial cells. The last two rows report clone-vs-clone disagreement (both seed regimes) and final-grader divergence.
Figure 1: Seeded replication, one seed per state: divergence from the live continuation (fraction of 24 states) for the arms of Table 4 at tool-failure rate 0.05 (left) and 0.20 (right); error bars are exact binomial 95% intervals drawn for orientation only, since the states are factorial cells; the unseeded bar is the separate unseeded run of Table 4 .
curves.csv
marginal.csv
Rows matched
285/285
45/45
Point-estimate values compared
1,710
45
Max ∣ shipped − fresh ∣ , point estimates
1.33×10−15
1.53×10−16
Interval endpoints compared
570
90
Max ∣ shipped − fresh ∣ , endpoints
7.22×10−16
2.78×10−16
Non-overlapping intervals
0
0
Appendix
Table 5: Shipped versus fresh bootstrap-bearing tables. All 330 intervals overlap, and no interval changes its qualitative classification.
#
Arm
Error content
Source role
Isolates
1
Short verified state
none
—
fresh-context baseline
2
Clean-long
inert, token/turn-matched
self
length tax L (with 1)
3
Healed own
errors corrected in place
self
error-content counterfactual
4
Untouched own
natural errors
self
Δerr (with 3)
5
Natural foreign
matched real foreign errors
prior-agent
provenance × natural content
6
Synthetic error
injected matched errors
self
reproduction of the synthetic-injection result [ 24 ]
Appendix
Table 6: History arms. Only assistant-side text differs; the frozen executable state (environment and process state), the user turns and all tool outputs are identical across arms. Arms 3–6 instantiate four of the eight crossed content-by-role cells (healed and synthetic content appear only under the self wrapper); arms 7–8 carry the provenance contrast on a fixed packet outside the four content levels; arm 5 is the natural-content, prior-agent cell and is not a clean provenance contrast.
Table 7: Estimands. All are risk differences in percentage points of exact continuation success, estimated pairwise within forked states and cluster-bootstrapped over tasks, except S , which is dimensionless and reported unclipped. The reproduction tests compare Δforeign and Δsynth against Δerr as ±5 -point equivalence tests (both interval bounds inside the margin), not as failures to reject. S is secondary, reported only when its denominator is at least 15 points (the floor assumed in the power sketch below) and its numerator sign-stable; the claim that contaminated history explains less than half of decay requires the 95% upper bound below 0.5.
Probe
Signature
Repair implied
Error content
Δerr (late) large; foreign and synthetic arms reproduce it inside the ±5 -point margin; Δprov small
history editing
Provenance
Δprov large with the bytes held fixed; the reproduction tests fail
role-wrapper handling
Length tax
L large; contaminated arms barely move once token and turn counts are matched
neither; the effect was context length
None of the three
no contrast estimated precisely enough to locate the decay (the design is not powered for a 5-point equivalence margin)
not located by this design
Appendix
Table 8: What each probe’s sign would imply, and the repair it points to. Because each sign implies a different repair, the intended result is the set of contrasts, not the sign of any single one.
Code set
Stage 1
screen from title, TL;DR and tags; include if the paper (a) studies LLM agents or multi-turn LLM systems and (b) contributes primarily a taxonomy of failure modes, a failure-attribution or failure-localization method or benchmark, or an empirical diagnosis of why agents fail. Exclude capability benchmarks without failure analysis, methods papers whose failure discussion is incidental, surveys that do not themselves categorize failures, and position papers with no coded failure data
Q1 label source
human annotation; LLM judge; rule-or-automatic; mixed; none; not stated in card
Q2 intervention
none; system-or-task change scored end-to-end; history-or-state manipulation at an intermediate point with continuation; other
Q3 class
healed history; provenance swap; placebo or length-matched synthetic history; counterfactual injection; environment-or-state reset; replay from checkpoint without content change; other
Q4 frozen state
does the paper restore an executable intermediate state (environment plus process) and continue the same agent from it? yes; prefix re-execution only; no; not determinable from card
Q5 evidence
one verbatim quote ( ≤30 words) from the card supporting Q2 and Q4
Appendix
Table 9: The rubric as frozen before coding. Q3 is asked only when Q2 is a history-or-state manipulation, and admits multiple codes.
Coder 1
Coder 2
Q2 = history-or-state manipulation with continuation
22
13
Q2 = system-or-task change scored end-to-end
13
22
Q2 = none
6
6
Q4 = yes (frozen executable state, same agent continued)
1
3
Q4 = prefix re-execution only
1
1
Q4 = not determinable from card
3
2
Appendix
Table 10: Stage 2 counts, the two coders side by side ( N=41 each). Q2 and Q4 each partition the 41 cards.
Table 12
#
arXiv
Short title
Q2
Q4
Coder 2 differs
1
2608.06503
Toward Reliable Context Compression
man.
yes
—
2
2607.24167
Falsifiable Commitment Planning
man.
n.d.
Q2 e2e; Q4 no
3
2601.23045
The Hot Mess of AI
e2e
no
—
4
2509.09677
The Illusion of Diminishing Returns
man.
no
—
5
2608.11772
Diagnosis Before Recovery
e2e
no
—
6
2607.02606
ChainSWE: multi-bug maintenance
man.
no
Q4 yes
Appendix
Table 13: The 41 included cards, part 1 of 2 (cards 1–21). Q2: man. = history-or-state manipulation with continuation, e2e = system-or-task change scored end-to-end. Q4: pre. = prefix re-execution only, n.d. = not determinable from card. The last column shows coder 2’s code wherever it differs. Short titles are abridged from each card’s frontmatter.
#
arXiv
Short title
Q2
Q4
Coder 2 differs
22
2505.00212
Which Agent Causes Task Failures?
none
no
—
23
2606.06324
From Failed Trajectories to Reliable Agents
e2e
no
—
24
2505.23840
Measuring Sycophancy in Multi-turn Dialogues
e2e
no
—
25
2606.22528
Governance Decay
man.
no
—
26
2607.09510
Failure as a Process
none
no
—
27
2603.24631
Coherence Collapse
e2e
no
—
Appendix
Table 14: The 41 included cards, part 2 of 2 (cards 22–41). Columns as in Table 13 .