cs.LGAug 20, 2026
SaveCredit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
Organizations: University of Southern California Los Angeles, CA, USA
Abstract
Audited against policy-conditional ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals we audit -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- shows reliable incremental fidelity beyond its own marginal-matched shuffled control. Correcting for replay-target reliability leaves implicit fidelity bounded near zero and judge fidelity inconclusive at the achieved target reliability. Existing evaluations grade these signals against annotated step correctness; we audit them against step contribution -- what re-sampling the policy's own alternatives at each decision point and rolling forward changes about the outcome -- and they come apart. The ground truth is structured: 30.5% of decision points where it is defined exhibit a nonzero replay contrast at the achieved sampling resolution, and measurability is model-dependent -- the fraction of points with no policy-supported counterfactual differs twofold (13.1% vs. 26.8%) between two similar-scale policies. The failure mode is identifiable: implicit credit echoes the policy's fluency (median rank correlation +0.75, replicating at +0.70 in a second family under a corrected instrument), while outcome conditioning adds no causal information (partial correlation -0.004, Qwen). A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn (14.0% per trajectory). In a seven-arm pre-registered training experiment, no arm reliably outperforms the untrained policy, and the checkpoints' apparent instrument signature is statistically consistent with mediation by effective training dose in this design -- sparser credit retains fewer examples, an order-of-magnitude spread in optimizer steps -- not credit content. Comparisons of credit rules must match effective sample size, or they measure dose, not credit.
Figures & tables
Figure 1: The audit verdict: in both model families, credit is indistinguishable from its own shuffled control. Each family’s rank-fidelity estimate against executed-replay ground truth (median within-trajectory Spearman vs. , Section 2 ; filled markers) beside its own marginal-matched shuffled control (open markers, paired within family; the figure plots rank fidelity only — the precision-at-pivotal lift discussed in Section 4 is a different statistic and is not plotted): Qwen2.5-7B implicit ( of 50; own control ) and judge ( , 32 computable; own control ), data runs/collect_v2/analysis_summary.json ; Llama-3.1-8B implicit under the corrected instrument ( trajectories retained of 28; own control ), data runs/xfam_ext/redo/redo_family_analysis.json , coverage 88.3% 1 . Under the frozen controls-first verdict order, every family–control pair overlaps under the controls-first order: verdict H3 (placebo-level, frozen verdict order, Section 4 ) throughout; the binding comparison is each family against its own control.
| Family | (median) [95% CI] | Shuffle [95% CI] | |
|---|---|---|---|
| Implicit (HCAPO ) | 37 of 50 | ||
| Judge (Qwen2.5-72B) | 37 (32 comp.) |
Table 1: Both credit families are placebo-level at within-trajectory credit ranking under the frozen verdict order (H3): each is indistinguishable from its own marginal-matched shuffled control. The implicit estimate is centred on zero, and bounded near zero once corrected for replay-target reliability; the judge estimate is weak, possibly nonzero, yet indistinguishable from its own control, and inconclusive after the same correction — the families are reported separately and earn the same verdict class. Estimator: median within-trajectory Spearman vs. , 10k bootstrap (estimator note: Appendix A ); controls: marginal-matched within-trajectory shuffles; instruments: original (Qwen), corrected (Llama; coverage 88.3% 1 ); data: runs/collect_v2/analysis_summary.json , runs/xfam_ext/redo/redo_family_analysis.json . Sign agreement: per-step vs trajectory-median splits, Wilson 95%; implicit , judge ; = trajectories retained by the frozen exclusion rules.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Question | Failure instance | Defence |
|---|---|---|---|
| Identity | is the object checked the object the system uses ? | the gate verified a different physical copy than the stages loaded — hashing the wrong file passes perfectly | realpath binding: a verification claim names the physical path the stage will load |
| Creation-time validity | were the bytes correct when written ? | a NUL-filled file’s hash is a valid hash of a NUL-filled file; every replica verifies | structural assertions against intent , not against the writer’s own output |
| Persistence | is every item still readable now ? | a file-count check passes while one file is unreadable | per-file content hashing — you cannot hash what you cannot read |
| Diagnosability | does the check preserve the evidence needed to act ? | the guard reported a verdict and discarded the errno; EIO and EDQUOT call for opposite responses | log the raw errno, never the verdict |
Table 2: The four dimensions of the integrity taxonomy, each with the incident that named it and the defence adopted. No single check covers all four dimensions; the failure mode throughout is assuming one check answers another’s question.
| class | turns | contrast at this design |
|---|---|---|
| constant pool (all pooled outcomes identical) | 1,204 | forced to zero |
| arithmetically forced | 509 | forced nonzero |
| genuinely stochastic | 55 | nonzero with probability |
Table 3: Divisibility classes of the replay estimand at under Qwen2.5-7B: which turns could register a zero contrast at all, before the environment is consulted. Read the third column as a property of the counts, not of the task. complete turns; runs/collect_v2/ class counts, E-B1B3.
| Family | Estimate | Median | Bootstrap 95% CI | Undefined | |
|---|---|---|---|---|---|
| Implicit | uncorrected (published) | 37 | — | ||
| Judge | uncorrected (published) | 32 | — | ||
| Implicit | Route 1, literal rule ( ), primary | 22 | 15 | ||
| Implicit | Route 1, machine-zero sensitivity | 21 | 16 | ||
| Judge | Route 1, literal rule ( ), primary | 21 | 11 | ||
| Judge | Route 1, machine-zero sensitivity | 20 | 12 |
Table 4: Step 2: corrected rank fidelity, Qwen2.5-7B, original instrument. Each row is a family median of per-trajectory corrected Spearman with its bootstrap 95% interval; the upper bound is the interval’s upper edge (97.5th percentile), on which the pre-registered rule keys. Route 1 is the primary; Route 2 rows are reported as the pre-registration requires and are disqualified for any bound or claim (known-answer failure, floor ). Uncorrected rows are the published medians of Section 4 on their own . Undefined = trajectories with (or under the sensitivity treatment), counted and excluded from the median. Source: A2_step2_output.txt (governance branch, commit 3e00d92 ; sha256 aa33b9b1 … 8721 ).
| arm | s0 | s1 | s2 | mean | between-seed SD |
|---|---|---|---|---|---|
| 1 outcome-only | 0.438 | 0.445 | 0.438 | 0.440 | 0.5 pp |
| 2 implicit ( ) | 0.352 | 0.414 | 0.344 | 0.370 | 3.9 pp |
| 3 judge | 0.359 | 0.531 | 0.445 | 0.445 | 8.6 pp |
| 4 shuffled- | 0.375 | 0.383 | 0.422 | 0.393 | 2.5 pp |
| 5 inverted- | 0.406 | 0.391 | 0.094 † | 0.297 † | 17.6 pp † |
| 6 pivotal-masked | 0.469 | 0.344 | 0.320 | 0.378 | 8.0 pp |
Table 5: Seven-arm training outcome: every pre-registered confirmatory comparison is inconclusive under the frozen pp band (C1–C4 TOST equivalence; C5–C6 one-sided superiority; Holm over the six), and no arm reliably beats the untrained base policy (0.422; 54/128). Top: final-round win rates on the frozen 128 held-out tasks (greedy; per arm-seed; seed-paired CRN seeds {0,1,2}). arm 5 seed 2 is the format-collapse instrument note of Appendix G, excluded from primary analysis by pre-registration; the printed arm-5 mean and SD include it and inherit the flag, and the registered-exploratory sensitivity dropping it leaves all six comparisons inconclusive. Bottom: paired comparisons; the design’s minimum detectable effect at this and seed count is 11.8 pp (Appendix F), so “inconclusive” is the frozen reading, never “no difference”. Data: runs/ks2/eval/final/<arm>/<seed>/eval.json (win rates), runs/ks2/eval/r1/base_control/s0/eval.json (base), runs/ks2/CONFIRMATORY_RESULT.json (comparisons).
| Partition | Stage / statistic | (B W) | ||
|---|---|---|---|---|
| Instrument | Stage 1, turn-1 canonical agreement | |||
| Information | Stage 1, turn-1 canonical agreement | |||
| Instrument | Stage 2, 500-state bank JS | |||
| Information | Stage 2, 500-state bank JS |
Table 6: The pre-registered instrument partition separates the checkpoints; the information partition does not. Each row reports mean between-partition minus mean within-partition JS divergence over cross-arm checkpoint pairs, under the partition named in column 1; positive means checkpoints sharing a group are more similar to each other than to the other group. Partitions, criterion, and the reading to record were frozen in a3b-prereg-v1 before the matrix was seen; each partition’s is a restricted permutation test under the pre-registered within-seed scheme (10,000 draws), the scheme attribution following the pre-registration. Both partitions divide the same 171 cross-arm pairs; the split differs, so the two rows carry different . Data: a3b_stage1_structure.json , a3b_stage2_structure.json .
Figure 2: Perturbation strength does not order the checkpoints’ distributional positions: the position readout is not a dose meter for information destruction along the magnitude axis. Each contrast pair’s JS position relative to the within-arm band (%), across training rounds r1–r3 (marker shade), against the measured strength of its perturbation — fraction of credit positions actually changed: shuffle- , shuffle-judge , inversion ( trajectories each; runs/pc2/a3b_perturbation_strength.json ). Position values are the nine round-by-contrast cells of runs/pc2/a3b_position_monotonicity.json ; the pre-registered monotonicity test returns (10k perm; adjacent comparisons; grid , randomization unit = contrast, ). Four qualifiers ride this figure, per its record: the strongest perturbations sit in a saturated regime of the readout; separation is monotone in information destruction ; the non-monotonicity is confined to the magnitude axis; and the judge-shuffle pair carries no positional signal.
Figure 3: [Exploratory — R_ROUTE] The confidence router recovers pivotal turns at chance level at every pre-registered threshold — its recall never exceeds the matched random-routing baseline; its two cost granularities move in opposite directions, so neither may be read alone. Left: recall of pivotal turns (pivotal stored ; zeros are resolution-bounded, Section 2 ) across the six pre-registered threshold multipliers, with Wilson 95% band ( pivotal turns at ); the dashed curve is the matched random-routing baseline (the routed fraction), and the dotted line marks the baseline at the registered threshold , against recall — chance-level, never exceeding the matched baseline at any of the six multipliers. Right: judge-call reduction at per-turn and per-trajectory granularity; the curves cross directions across the sweep, and neither curve may be read as a single unlabelled cost figure. Sweep is sensitivity only; the single main result is multiplier 1.0. Data: route_pack/artifacts/route_theta_sweep.json , route_pack/route_retrospective.json (frozen prereg 1814166… ).
Figure 4: Measurability of replay ground truth is model-dependent in both directions: the family with more undefined counterfactuals is the one with more causal signal where they exist. Bars: fraction of turns with no policy-supported counterfactual (Qwen % , intervened; Llama % , ; corrected instrument 1 ) and fraction of complete turns that are pivotal (Qwen % , ; Llama % , ); Wilson 95% intervals; the two divergences run in opposite directions. The absorbing axis is near-flat across families and is reported here rather than plotted: at % of Qwen’s and % of Llama’s complete turns; every turn carries the resolution bound of Section 2 (unexcluded at one-sided 95%). Data: measurability_counts_{ks1,xfam_redo}.json ; full three-axis map: Figure 5 (appendix).
Figure 5: Where per-step replay ground truth can be measured depends on the model — in both directions. For each family, three fractions of the replay map: turns with no policy-supported counterfactual (Qwen2.5-7B 13.1%, intervened turns; Llama-3.1-8B 26.8%, , corrected instrument 1 ), pivotal turns ( : 30.5%, ; 38.3%, ), and absorbing turns ( : 86.0%, ; 80.6%, ). Error bars are Wilson 95% intervals throughout — Qwen: , , ; Llama: , , respectively. The large divergence is the first bar — Llama lacks a policy-supported counterfactual at twice Qwen’s rate — yet the pivotal fraction runs the other way, so neither family is uniformly harder to audit. All zero-valued turns are resolution-bounded ( , Section 2 ). Data: measurability_counts_ks1.json , measurability_counts_xfam_redo.json .
Figure 6: An occupancy-style model of the policy’s own action statistics ranks turns by measurability but does not level-calibrate. KS1/Qwen; the cross-family replicate is reported numerically in §5. Single-family presentation is the figure’s specification: the predictor is reported as a one-family observation, and the ranking-vs-level contrast (observed completion rising across predicted deciles against a 20-percentage-point aggregate level miss) is stated in Section 3 . intervened turns; data runs/ks1xfam/measurability_predictor.json ( ks1 key).
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,132 papers