Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay

Haiyue Zhang

cs.LGAug 20, 2026

Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay

Authors: Haiyue Zhang

Organizations: University of Southern California Los Angeles, CA, USA

Abstract

Audited against policy-conditional ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals we audit -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- shows reliable incremental fidelity beyond its own marginal-matched shuffled control. Correcting for replay-target reliability leaves implicit fidelity bounded near zero and judge fidelity inconclusive at the achieved target reliability. Existing evaluations grade these signals against annotated step correctness; we audit them against step contribution -- what re-sampling the policy's own alternatives at each decision point and rolling forward changes about the outcome -- and they come apart. The ground truth is structured: 30.5% of decision points where it is defined exhibit a nonzero replay contrast at the achieved sampling resolution, and measurability is model-dependent -- the fraction of points with no policy-supported counterfactual differs twofold (13.1% vs. 26.8%) between two similar-scale policies. The failure mode is identifiable: implicit credit echoes the policy's fluency (median rank correlation +0.75, replicating at +0.70 in a second family under a corrected instrument), while outcome conditioning adds no causal information (partial correlation -0.004, Qwen). A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn (14.0% per trajectory). In a seven-arm pre-registered training experiment, no arm reliably outperforms the untrained policy, and the checkpoints' apparent instrument signature is statistically consistent with mediation by effective training dose in this design -- sparser credit retains fewer examples, an order-of-magnitude spread in optimizer steps -- not credit content. Comparisons of credit rules must match effective sample size, or they measure dose, not credit.

Figures & tables

Figure 1: The audit verdict: in both model families, credit is indistinguishable from its own shuffled control. Each family’s rank-fidelity estimate against executed-replay ground truth (median within-trajectory Spearman vs.

A_{\mathrm{replay}}

, Section 2 ; filled markers) beside its own marginal-matched shuffled control (open markers, paired within family; the figure plots rank fidelity only — the precision-at-pivotal lift discussed in Section 4 is a different statistic and is not plotted): Qwen2.5-7B implicit

0.0193

[-0.109,0.081]

(

n_{\mathrm{eff}}{=}37

of 50; own control

[0.005,0.114]

) and judge

0.1142

[0.027,0.168]

(

n{=}37

, 32 computable; own control

[-0.049,0.117]

), data runs/collect_v2/analysis_summary.json ; Llama-3.1-8B implicit under the corrected instrument

-0.043

[-0.125,-0.016]

(

n{=}21

trajectories retained of 28; own control

[-0.102,+0.024]

), data runs/xfam_ext/redo/redo_family_analysis.json , coverage 88.3% 1 . Under the frozen controls-first verdict order, every family–control pair overlaps under the controls-first order: verdict H3 (placebo-level, frozen verdict order, Section 4 ) throughout; the binding comparison is each family against its own control.

Family	$\hat{\rho}$ (median) [95% CI]	$n_{\mathrm{eff}}$	Shuffle [95% CI]
Implicit (HCAPO $\rho_{t}$ )	$0.0193$ $[-0.109,0.081]$	37 of 50	$[0.005,0.114]$
Judge (Qwen2.5-72B)	$0.1142$ $[0.027,0.168]$	37 (32 comp.)	$[-0.049,0.117]$

Table 1: Both credit families are placebo-level at within-trajectory credit ranking under the frozen verdict order (H3): each is indistinguishable from its own marginal-matched shuffled control. The implicit estimate is centred on zero, and bounded near zero once corrected for replay-target reliability; the judge estimate is weak, possibly nonzero, yet indistinguishable from its own control, and inconclusive after the same correction — the families are reported separately and earn the same verdict class. Estimator: median within-trajectory Spearman vs.

A_{\mathrm{replay}}

, 10k bootstrap (estimator note: Appendix A ); controls: marginal-matched within-trajectory shuffles; instruments: original (Qwen), corrected (Llama; coverage 88.3% 1 ); data: runs/collect_v2/analysis_summary.json , runs/xfam_ext/redo/redo_family_analysis.json . Sign agreement: per-step vs trajectory-median splits, Wilson 95%; implicit

267/527=50.7\%

[46.4,54.9]

, judge

84/139=60.4\%

[52.1,68.2]

;

n_{\mathrm{eff}}

= trajectories retained by the frozen exclusion rules.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Question	Failure instance	Defence
Identity	is the object checked the object the system uses ?	the gate verified a different physical copy than the stages loaded — hashing the wrong file passes perfectly	realpath binding: a verification claim names the physical path the stage will load
Creation-time validity	were the bytes correct when written ?	a NUL-filled file’s hash is a valid hash of a NUL-filled file; every replica verifies	structural assertions against intent , not against the writer’s own output
Persistence	is every item still readable now ?	a file-count check passes while one file is unreadable	per-file content hashing — you cannot hash what you cannot read
Diagnosability	does the check preserve the evidence needed to act ?	the guard reported a verdict and discarded the errno; EIO and EDQUOT call for opposite responses	log the raw errno, never the verdict

Appendix

Table 2: The four dimensions of the integrity taxonomy, each with the incident that named it and the defence adopted. No single check covers all four dimensions; the failure mode throughout is assuming one check answers another’s question.

class	turns	contrast at this design
constant pool (all pooled outcomes identical)	1,204	forced to zero
arithmetically forced	509	forced nonzero
genuinely stochastic	55	nonzero with probability $0.5055$

Appendix

Table 3: Divisibility classes of the replay estimand at

K=4

under Qwen2.5-7B: which turns could register a zero contrast at all, before the environment is consulted. Read the third column as a property of the counts, not of the task.

n=1{,}768

complete turns; runs/collect_v2/ class counts, E-B1B3.

Family	Estimate	Median	$n$	Bootstrap 95% CI	Undefined
Implicit	uncorrected (published)	$0.0193$	37	$[-0.109,0.081]$	—
Judge	uncorrected (published)	$0.1142$	32	$[0.027,0.168]$	—
Implicit	Route 1, literal rule ( $\rho_{xx}>0$ ), primary	$-0.034$	22	$[-0.295,0.174]$	15
Implicit	Route 1, machine-zero sensitivity	$-0.093$	21	$[-0.300,0.128]$	16
Judge	Route 1, literal rule ( $\rho_{xx}>0$ ), primary	$0.328$	21	$[-0.081,0.532]$	11
Judge	Route 1, machine-zero sensitivity	$0.285$	20	$[-0.162,0.486]$	12

Appendix

Table 4: Step 2: corrected rank fidelity, Qwen2.5-7B, original instrument. Each row is a family median of per-trajectory corrected Spearman with its bootstrap 95% interval; the upper bound is the interval’s upper edge (97.5th percentile), on which the pre-registered rule keys. Route 1 is the primary; Route 2 rows are reported as the pre-registration requires and are disqualified for any bound or claim (known-answer failure, floor

0.600

). Uncorrected rows are the published medians of Section 4 on their own

n

. Undefined = trajectories with

\rho_{xx}\leq 0

(or

\leq 10^{-12}

under the sensitivity treatment), counted and excluded from the median. Source: A2_step2_output.txt (governance branch, commit 3e00d92 ; sha256 aa33b9b1 … 8721 ).

arm	s0	s1	s2	mean	between-seed SD
1 outcome-only	0.438	0.445	0.438	0.440	0.5 pp
2 implicit ( $\rho$ )	0.352	0.414	0.344	0.370	3.9 pp
3 judge	0.359	0.531	0.445	0.445	8.6 pp
4 shuffled- $\rho$	0.375	0.383	0.422	0.393	2.5 pp
5 inverted- $\rho$	0.406	0.391	0.094 †	0.297 †	17.6 pp †
6 pivotal-masked	0.469	0.344	0.320	0.378	8.0 pp

Appendix

Table 5: Seven-arm training outcome: every pre-registered confirmatory comparison is inconclusive under the frozen

\pm 3

pp band (C1–C4 TOST equivalence; C5–C6 one-sided superiority; Holm over the six), and no arm reliably beats the untrained base policy (0.422; 54/128). Top: final-round win rates on the frozen 128 held-out tasks (greedy;

n{=}128

per arm-seed; seed-paired CRN seeds {0,1,2}).

\dagger

arm 5 seed 2 is the format-collapse instrument note of Appendix G, excluded from primary analysis by pre-registration; the printed arm-5 mean and SD include it and inherit the flag, and the registered-exploratory sensitivity dropping it leaves all six comparisons inconclusive. Bottom: paired comparisons; the design’s minimum detectable effect at this

n

and seed count is

\approx

11.8 pp (Appendix F), so “inconclusive” is the frozen reading, never “no difference”. Data: runs/ks2/eval/final/<arm>/<seed>/eval.json (win rates), runs/ks2/eval/r1/base_control/s0/eval.json (base), runs/ks2/CONFIRMATORY_RESULT.json (comparisons).

Partition	Stage / statistic	$\Delta$ (B $-$ W)	$p$	$n_{W}/n_{B}$
Instrument	Stage 1, turn-1 canonical agreement	$+0.0291$	$0.0124$	$30/141$
Information	Stage 1, turn-1 canonical agreement	$-0.0077$	$0.7205$	$36/135$
Instrument	Stage 2, 500-state bank JS	$+0.0346$	$0.0002$	$30/141$
Information	Stage 2, 500-state bank JS	$-0.0126$	$0.9777$	$36/135$

Appendix

Table 6: The pre-registered instrument partition separates the checkpoints; the information partition does not. Each row reports mean between-partition minus mean within-partition JS divergence over cross-arm checkpoint pairs, under the partition named in column 1; positive means checkpoints sharing a group are more similar to each other than to the other group. Partitions, criterion, and the reading to record were frozen in a3b-prereg-v1 before the matrix was seen; each partition’s

p

is a restricted permutation test under the pre-registered within-seed scheme (10,000 draws), the scheme attribution following the pre-registration. Both partitions divide the same 171 cross-arm pairs; the split differs, so the two rows carry different

n_{W}/n_{B}

. Data: a3b_stage1_structure.json , a3b_stage2_structure.json .

Figure 2: Perturbation strength does not order the checkpoints’ distributional positions: the position readout is not a dose meter for information destruction along the magnitude axis. Each contrast pair’s JS position relative to the within-arm band (%), across training rounds r1–r3 (marker shade), against the measured strength of its perturbation — fraction of credit positions actually changed: shuffle-

\rho

0.9043

, shuffle-judge

0.2602

, inversion

1.0000

(

n{=}288

trajectories each; runs/pc2/a3b_perturbation_strength.json ). Position values are the nine round-by-contrast cells of runs/pc2/a3b_position_monotonicity.json ; the pre-registered monotonicity test returns

p{=}0.9413

(10k perm;

n{=}6

adjacent comparisons; grid

3{\times}3

, randomization unit = contrast,

k{=}3

). Four qualifiers ride this figure, per its record: the strongest perturbations sit in a saturated regime of the readout; separation is monotone in information destruction ; the non-monotonicity is confined to the magnitude axis; and the judge-shuffle pair carries no positional signal.

Figure 3: [Exploratory — R_ROUTE] The confidence router recovers pivotal turns at chance level at every pre-registered threshold — its recall never exceeds the matched random-routing baseline; its two cost granularities move in opposite directions, so neither may be read alone. Left: recall of pivotal turns (pivotal

=

stored

A_{\mathrm{replay}}\neq 0

; zeros are resolution-bounded, Section 2 ) across the six pre-registered threshold multipliers, with Wilson 95% band (

n{=}540

pivotal turns at

\theta^{*}

); the dashed curve is the matched random-routing baseline (the routed fraction), and the dotted line marks the baseline

0.1307

at the registered threshold

\theta^{*}

, against recall

0.1185

[0.0939,0.1485]

— chance-level, never exceeding the matched baseline at any of the six multipliers. Right: judge-call reduction at per-turn and per-trajectory granularity; the curves cross directions across the sweep, and neither curve may be read as a single unlabelled cost figure. Sweep is sensitivity only; the single main result is multiplier 1.0. Data: route_pack/artifacts/route_theta_sweep.json , route_pack/route_retrospective.json (frozen prereg 1814166… ).

Figure 4: Measurability of replay ground truth is model-dependent in both directions: the family with more undefined counterfactuals is the one with more causal signal where they exist. Bars: fraction of turns with no policy-supported counterfactual (Qwen

13.1

[11.7,14.6]

n{=}2{,}034

intervened; Llama

26.8

[24.2,29.5]

n{=}1{,}082

; corrected instrument 1 ) and fraction of complete turns that are pivotal (Qwen

30.5

[28.4,32.7]

n{=}1{,}768

; Llama

38.3

[34.9,41.7]

n{=}792

); Wilson 95% intervals; the two divergences run in opposite directions. The absorbing axis is near-flat across families and is reported here rather than plotted:

\sigma_{\mathrm{floor}}{=}0

86.0

[84.3,87.6]

of Qwen’s and

80.6

[77.7,83.2]

of Llama’s complete turns; every

A_{\mathrm{replay}}{=}0

turn carries the resolution bound of Section 2 (unexcluded

|\Delta p|\leq 0.632

at one-sided 95%). Data: measurability_counts_{ks1,xfam_redo}.json ; full three-axis map: Figure 5 (appendix).

Figure 5: Where per-step replay ground truth can be measured depends on the model — in both directions. For each family, three fractions of the replay map: turns with no policy-supported counterfactual (Qwen2.5-7B 13.1%,

n{=}2{,}034

intervened turns; Llama-3.1-8B 26.8%,

n{=}1{,}082

, corrected instrument 1 ), pivotal turns (

A_{\mathrm{replay}}\neq 0

: 30.5%,

n{=}1{,}768

; 38.3%,

n{=}792

), and absorbing turns (

\sigma_{\mathrm{floor}}{=}0

: 86.0%,

n{=}1{,}768

; 80.6%,

n{=}792

). Error bars are Wilson 95% intervals throughout — Qwen:

[11.7,14.6]

[28.4,32.7]

[84.3,87.6]

; Llama:

[24.2,29.5]

[34.9,41.7]

[77.7,83.2]

respectively. The large divergence is the first bar — Llama lacks a policy-supported counterfactual at twice Qwen’s rate — yet the pivotal fraction runs the other way, so neither family is uniformly harder to audit. All zero-valued

A_{\mathrm{replay}}

turns are resolution-bounded (

|\Delta p|\leq 0.632

, Section 2 ). Data: measurability_counts_ks1.json , measurability_counts_xfam_redo.json .

Figure 6: An occupancy-style model of the policy’s own action statistics ranks turns by measurability but does not level-calibrate. KS1/Qwen; the cross-family replicate is reported numerically in §5. Single-family presentation is the figure’s specification: the predictor is reported as a one-family observation, and the ranking-vs-level contrast (observed completion rising

0.650\to 0.981

across predicted deciles against a 20-percentage-point aggregate level miss) is stated in Section 3 .

n{=}2{,}034

intervened turns; data runs/ks1xfam/measurability_predictor.json ( ks1 key).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Mar 6, 2026

The Trace Is the State: Exact Credit Assignment for LLM Agent Teams

Credit assignment for a team of LLM agents, what each message was worth, has mostly been treated as prediction: a learned critic, a trajectory-level score, or an agent ablation stands in for the counterfactual that defines credit, which is rarely run. Teams that communicate through a shared context are different: when everything a downstream agent reads is written into the trace, the trace is the state, and that counterfactual can be executed. A credit signal can then be judged as any estimator is: by bias, variance, and agreement with an independent reference. C3, credit assignment by counterfactual continuation, substitutes one message at a decision point and continues the run to the terminal reward, so its credit is unbiased, exact up to Monte Carlo error. Given the sampled alternatives, that error's variance follows a derived law with no term for the number of agents, and the observed noise follows the law on 6 workflows of 2 to 10 decision points. On the 2-agent chain, from 4 continuations, C3 ranks alternatives at 0.69 rank correlation against a 16-continuation reference, near the 0.73 at which 2 such references agree; a critic trained on the same rollouts reaches 0.29. Used as the advantage in training a 2-agent team, C3 beats MAGRPO, the stronger baseline in our comparison, and spends 37% fewer training tokens than MAPPO, since only the messages downstream of a decision point are regenerated. When the trace is the state, credit need not be predicted; it can be exact.

Yanjun Chen, Yirong Sun, Hanlin Wang +5

Eastern Institute of Technology · The Hong Kong Polytechnic University · Harbin Institute of Technology

Credit Assignment Counterfactual Credit Assignment LLM Agent Training Multi-Agent LLM Systems Multi-Agent Reinforcement Learning

Ask PDF

Jun 6, 2026

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

When an LLM agent fails -- issues a refund it should not have, calls the wrong tool, leaks data -- existing tooling answers what happened (observability) or whether it passed (evaluation), but not which step caused the failure. The obvious heuristics are wrong: the step that executes the harmful action is usually not the step that decided on it, and LLM-judge attribution is correlational and unreliable (state-of-the-art step-level accuracy on the Who&When benchmark is about 14%). We present Causal Agent Replay (CAR), which answers the question by intervention: it models an agent run as a structural causal model, applies a do-operation to a step, and re-executes the trajectory forward under the same stochastic policy, measuring the shift in the outcome distribution. We define an intervention algebra over agent steps, a single-step contrastive estimator whose point-of-commitment rule resolves a confound specific to stochastic run-forward, and a budget-bounded Monte-Carlo Shapley estimator that splits credit across interacting steps. Every effect is reported with confidence intervals. We validate against synthetic structural causal models with planted ground truth: the contrastive estimator recovers the pivotal step, and Shapley recovers a two-step interaction (0.44, 0.45, ~0; efficiency sum 0.909 versus the analytic 0.91). CAR is open source and runs on hosted or free local models.

Jaineet Shah

Agent Failure Analysis AI Agent Reliability Counterfactual Evaluation Causal Attribution Failure Attribution for LLM Agents Causal Inference+1

Ask PDF

Apr 10, 2026

From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

Reinforcement learning (RL) for large language models (LLMs) increasingly relies on sparse outcome rewards, yet such rewards say little about which token, reasoning step, tool call, memory operation, or agent caused an outcome. This credit assignment (CA) problem spans reasoning RL and becomes sharper in agentic RL, where environment interaction introduces transition non-closure, partial observability, limited replay, heterogeneous actions, weak intermediate verifiability, and agent coupling. We synthesize a unified corpus of 69 papers published from January 2024 through July 31, 2026: 56 core CA methods and 13 adjacent or boundary enablers, selected from 92 deduplicated screening records. We retain the original granularity-by-methodology taxonomy and add a six-diagnostic framework mapping assumption breaks to identification barriers, estimators, and evaluation controls. A source-located full-text audit covers a fixed 42-core-paper subset. Two algorithm researchers independently and blindly cross-coded 252 diagnostic cells, agreeing on 223 (88.5%); per-diagnostic Cohen's kappa ranges from .543 to .909, and principal-family agreement is 42/42 (kappa=1.000). Beyond taxonomy, we establish when restored-state comparisons identify a protocol-specific causal contrast, show that text-only histories can leave even the sign of credit unidentified, and introduce a reusable CA-ID Card linking each claim to its estimand, evidence provenance, and falsification test. An atomic reporting audit describes comparator, budget parity, ablation, overhead, uncertainty, and replay coverage without constructing a cross-paper leaderboard. The companion repository hosts a living catalog and decision aids; a dated release of the frozen audit bundle is planned there separately from the minimal arXiv source.

Chenchen Zhang

Independent Researcher

Credit Assignment in RL RL for Language Models Fine-Grained Credit Assignment RL for LLM Agents Causal RL Counterfactual Credit Assignment

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

The Trace Is the State: Exact Credit Assignment for LLM Agent Teams

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models