Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Zeropy Surio; Tobias Engel; Mei-Lin Chou; Anika Raman; Samuel Okafor

cs.LGOct 22, 2026

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Authors: Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor

Organizations: University of Sheffield · Carnegie Mellon University · University of Michigan

Abstract

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-ρ$ , where $ρ$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $Δ$ falls from $Θ(Δ^{-2})$ to $Θ(Δ^{-1})$ . A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.

Figures & tables

Figure 1: Why twins (exact curves). (a) Variance of credit for a decision with

Q{=}0.55

V{=}0.45

: subtracting a Monte Carlo baseline from the realized return is floored by the return’s own variance; independent pairs decay as

1/J

; twin pairs are lower by the factor

1-\rho

(Prop. 2 ). (b) Probability of assigning the wrong sign to a

\Delta{=}0.1

advantage with

J

pairs. (c) Pairs needed for 95% sign accuracy:

\propto\Delta^{-2}

for independent pairs,

\propto\Delta^{-1}

for twins (Prop. 3 ). (d) Shared Gumbel noise makes two nearby next-token distributions emit the same token almost as often as an optimal coupling (Lemma 1 ); independent noise does not.

Figure 2: Twice in one training step. Top: every factual token draws Gumbel noise keyed by (episode, turn, line, position). Bottom: the twin of turn 2 swaps in an alternative drawn with fresh noise and continues with the factual keys, sharing the downstream luck $\xi_{3}$ ; under shared noise, nearby next-token distributions emit the same token (inset). Right: the outcome difference divided by the branch probability is the turn’s credit, and branch probabilities follow a square-root rule of predicted discordance over cost; the actor is updated with PPO/GRPO on factual tokens and deployed alone.

Qwen2.5-3B
	General QA			Multi-hop QA
Method	NQ †	TriviaQA	PopQA	HotpotQA †	2Wiki	MuSiQue	Bamb.	Avg.	Tok.
PPO	41.2	56.4	41.1	33.8	32.6	11.4	28.0	34.9	1.00 $\times$
GRPO	43.1	57.6	41.8	37.2	34.5	13.9	33.6	37.4	1.00 $\times$
GRPO ( $2\times$ )	43.8	58.1	42.3	38.1	35.4	14.5	34.4	38.1	2.00 $\times$
IGPO	44.0	57.9	42.6	39.0	37.1	15.2	35.2	38.7	1.00 $\times$

Table 1: Exact match (%) on seven QA benchmarks (mean of three seeds; † : in-domain; Bamb. is Bamboogle). Groups: outcome-level credit, process signals without extra rollouts, tree rollouts, counterfactual branching, ours. Tok.: generated tokens per update relative to GRPO, including auxiliary rollouts. Bold / underline: best / second best.

Method	Tok.	s/step	Rev. $\downarrow$	MSE $\downarrow$	Corr. $\uparrow$
GRPO	1.00 $\times$	96	31.8	1.00	0.18
GRPO ( $2\times$ )	2.00 $\times$	171	31.5	0.97	0.19
GiGPO	1.00 $\times$	99	27.4	0.84	0.29
Tree-GRPO	1.00 $\times$	94	24.9	0.71	0.37
VinePPO	3.20 $\times$	262	18.6	0.52	0.51
CRR	1.70 $\times$	149	21.7	0.61	0.46

Table 2: Credit quality and cost (Qwen2.5-3B, 8

\times

H100). Credit is audited on 1,036 decisions against an oracle advantage from 128 rollouts each. Tok.: generated tokens per update relative to GRPO; Rev.: % of decisions with

|A^{\star}|\geq 0.05

whose credit has the wrong sign; MSE: relative to GRPO; Corr.: Pearson correlation with

A^{\star}

Figure 3: Credit quality and coupling (Qwen2.5-3B, 400 held-out multi-hop questions). (a) Share of wrong-sign credit with

J{=}(b{+}1)/2

pairs per decision. (b) Mean squared error to the oracle; the dashed line is the independent curve scaled by

1-\bar{\rho}

from the measured twin correlation. (c) Twin correlation by oracle effect size for three noise layouts. (d) Outcomes of the pairs used for training.

Twin worlds
Configuration	Avg.	$\Delta$	MH	$\bar{\rho}$	Rev.
Twice (full)	42.7	–	36.9	0.57	6.9
independent twins	41.2	$-1.5$	35.0	0.02	20.4
per-episode seed	41.6	$-1.1$	35.5	0.29	14.8
no line re-sync	42.3	$-0.4$	36.4	0.49	8.6
$n$ -gram keys (inexact)	42.4	$-0.3$	36.6	0.61	6.1

Table 3: Ablations (Qwen2.5-3B). Avg. / MH: seven-benchmark / multi-hop exact match;

\bar{\rho}

: mean twin correlation; Rev.: wrong-sign credit (%). Budgets are matched.

Figure 4: Dynamics and budget (Qwen2.5-3B). (a) In-domain dev exact match over updates (

\pm 1

s.d., three seeds). (b) The same runs against generated tokens, auxiliary rollouts included. (c) Branch probability and realized twin discordance by the prompt’s group success rate. (d) Calibration of the discordance predictor.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Turn/line keys	Episode stream	Independent
Bias of credit, mean $\pm$ s.e. ( $\times 10^{-3}$ )	$1.1\pm 0.7$	$0.9\pm 1.0$	$0.3\pm 1.4$
Slope of credit on exact $A$	1.001	1.012	1.010
MSE to exact $A$	0.110	0.253	0.472
Twin correlation $\hat{\rho}$	0.769	0.474	0.025
MSE ratio: measured / predicted $1-\hat{\rho}$	0.233 / 0.231	0.537 / 0.526	1 / 1
Tie rate $\Pr(D{=}0)$	0.875	0.731	0.512

Appendix

Table 4: Numerical verification on the synthetic search MDP (exact values and Monte Carlo). All three layouts are unbiased (Prop. 1 ); the measured variance ratio matches

1-\hat{\rho}

(Prop. 2 ); twins reverse the sign of credit 17 times less often and have a four-times larger large-deviation rate (Prop. 3 ); and the square-root rule lowers total Horvitz–Thompson variance by 30% at equal cost (Prop. 4 ).

Figure 5: Numerical verification (exact MDP). (a) Sign accuracy of credit with

J

pairs on 2,000 decisions with

|A|\geq 0.05

; ties are broken at random, so single pairs are equally accurate, and twins pull ahead as discordant pairs accumulate. (b) Mean squared error to the exact advantage; black marks are the prediction

(1-\hat{\rho})\cdot\mathrm{MSE}_{\mathrm{ind}}

Dataset	Train	Eval
NQ [ Kwiatkowski et al., 2019 ]	79,168	3,610
TriviaQA [ Joshi et al., 2017 ]	–	11,313
PopQA [ Mallen et al., 2023 ]	–	14,267
HotpotQA [ Yang et al., 2018 ]	90,447	7,405
2Wiki [ Ho et al., 2020 ]	–	12,576
MuSiQue [ Trivedi et al., 2022 ]	–	2,417

Appendix

Table 5: Benchmarks: single-hop (top) and multi-hop (bottom). NQ and HotpotQA are in-domain.

Setting	Value
Updates; questions/update	400; 256
Rollouts $G$	5 (10 for GRPO $2\times$ )
Learning rate; warm-up	$10^{-6}$ ; 20 updates
PPO mini-batch; epochs	64; 1
Clip $\epsilon$ ; KL coefficient	0.2; 0.001
Train temperature; top- $p$	1.0; 1.0

Appendix

Table 6: Hyperparameters. Shared settings are identical for all methods.

Qwen2.5-3B
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamb.	Avg.
GRPO	51.2	64.3	45.7	47.5	40.2	22.2	42.6	44.8
GiGPO	52.8	65.3	47.0	50.1	44.1	24.4	45.4	47.0
Tree-GRPO	54.1	66.1	47.6	51.4	46.5	25.3	48.0	48.4
AT 2 PO	54.4	66.4	47.9	51.9	47.2	25.8	48.8	48.9
VinePPO	53.7	65.7	47.5	52.3	48.1	26.4	48.2	48.8

Appendix

Table 7: Token-level F1 (%), mean of three seeds. Bamb. is Bamboogle.

Method	Qwen2.5-3B	Qwen2.5-7B
PPO	$34.9\pm 0.52$	$40.0\pm 0.47$
GRPO	$37.4\pm 0.41$	$42.0\pm 0.35$
GRPO ( $2\times$ )	$38.1\pm 0.38$	$42.7\pm 0.33$
IGPO	$38.7\pm 0.44$	$43.1\pm 0.39$
GiGPO	$39.3\pm 0.37$	$43.6\pm 0.34$
Tree-GRPO	$40.5\pm 0.36$	$44.4\pm 0.31$

Appendix

Table 8: Seven-benchmark exact match, mean

\pm

s.d. over three seeds.

Qwen3-4B
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamb.	Avg.
GRPO	44.8	60.3	43.6	40.9	39.8	16.7	43.2	41.3
Tree-GRPO	46.7	61.5	44.9	43.4	43.9	18.6	45.6	43.5
AT 2 PO	47.1	61.8	45.2	43.9	44.6	19.0	46.4	44.0
CRR	46.4	61.2	44.6	43.2	44.1	18.8	45.6	43.4
Twice	48.2	62.9	46.1	45.8	47.3	21.4	49.6	45.9

Appendix

Table 9: Other backbones: exact match (%), mean of three seeds. Bamb. is Bamboogle.

	Qwen2.5-1.5B		Qwen2.5-7B
Method	ALFWorld	WebShop	ALFWorld	WebShop
GRPO	72.6	56.3	77.9	65.8
GiGPO	86.0	66.9	90.1	74.6
CRR	84.1	64.2	88.2	72.0
Twice	89.4	70.6	92.7	78.1

Appendix

Table 10: ALFWorld and WebShop: success rate (%), mean of three seeds.

Method	F1	Calls	Utility
GRPO	46.9	2.41	34.9
Tree-GRPO	50.6	2.46	38.3
VinePPO	51.8	2.33	40.2
CRR	51.1	2.29	39.7
Twice	53.9	1.92	44.3

Appendix

Table 11: Cost-aware utility on HotpotQA (Qwen2.5-3B): F1 (%), search calls per question, and utility

=

-

\times

calls (in %).

Twin budget $\beta$ (auxiliary tokens / policy tokens)
$\beta$	0.20	0.35	0.70	1.00	1.40
Twice	40.9	41.8	42.7	42.9	43.1
CRR	39.2	39.8	40.3	40.7	41.0
Pairs per branched decision $J$ (budget scaled with $J$ )
$J$ ( $\beta$ )	1 (0.7)	2 (1.4)	4 (2.8)
Twice	42.7	43.0	43.2

Appendix

Table 12: Sensitivity (Qwen2.5-3B, seven-benchmark exact match).

Figure 6: Sensitivity and coupling over training (Qwen2.5-3B). (a) Twin budget

\beta

; Twice at

\beta{=}0.35

matches AT 2 PO. (b) Twin-credit weight

\omega

. (c) Retrieval noise

\epsilon

(each passage replaced by a random one of the top 20 with probability

\epsilon

): coupling weakens (dashed, right axis) but the margin over CRR remains. (d) The share of self-twins and the mean twin correlation grow as the policy sharpens.

Tool noise $\epsilon$	0.0	0.1	0.3
$\bar{\rho}$ ( Twice )	0.57	0.49	0.36
GRPO	37.4	37.0	36.1
CRR	40.3	39.9	39.1
Twice	42.7	42.1	40.8

Appendix

Table 13: Nondeterministic retrieval (Qwen2.5-3B, seven-benchmark exact match).

Configuration	Avg.	MH	Tok.
PPO	34.9	26.5	1.00 $\times$
PPO + twin credit	40.6	34.2	1.70 $\times$
IGPO	38.7	31.6	1.00 $\times$
Twice + IGPO turn rewards	43.1	37.5	1.70 $\times$
Tree-GRPO	40.5	33.9	1.00 $\times$
Tree-GRPO + coupled siblings	41.8	36.1	1.00 $\times$

Appendix

Table 14: Composability (Qwen2.5-3B): seven-benchmark and multi-hop exact match.

Method	1.5B	3B	7B	14B
GRPO	27.6	37.4	42.0	45.6
AT 2 PO	31.2	41.0	44.8	47.9
Twice	33.4	42.7	46.5	49.3
$\Delta$ vs. AT 2 PO	$+2.2$	$+1.7$	$+1.7$	$+1.4$

Appendix

Table 15: Scale: seven-benchmark exact match for Qwen2.5-Instruct backbones (3B and 7B from Table 1 ).

	$H$ (MH)		$G$ (Avg.)
	4	8	3	5	8
GRPO	29.8	30.6	36.5	37.4	38.0
CRR	33.9	34.8	–	–	–
Twice	36.9	38.2	42.0	42.7	43.0
$\bar{\rho}$ / AUC	0.57	0.50	0.74	0.79	0.81

Appendix

Table 16: Horizon and group size (Qwen2.5-3B). Left: multi-hop exact match with at most

H

search calls. Right: seven-benchmark exact match with

G

rollouts per question.

Backbone	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamb.
Qwen2.5-3B	0.004	0.012	0.003	<0.001	<0.001	0.002	0.21
Qwen2.5-7B	0.009	0.018	0.006	<0.001	<0.001	0.003	0.19

Appendix

Table 17: Paired bootstrap

p

-values, Twice vs. AT 2 PO (10,000 resamples over questions). Bamb. is Bamboogle.

Failure	GRPO	Twice	GRPO	Twice
	Share (%)		Per 100 q.
evidence missing	31	33	21.8	20.8
premature answer	27	14	19.0	8.8
wrong despite evidence	18	26	12.6	16.4
query drift	15	14	10.5	8.8
search limit	9	13	6.3	8.2

Appendix

Table 18: Error analysis of 200 failed multi-hop episodes per method (Qwen2.5-3B): share of failures and failures per 100 multi-hop questions.

Method	Calls (1-hop)	Calls (multi)	Tok./ep.
GRPO	1.35	2.31	412
Tree-GRPO	1.29	2.52	437
VinePPO	1.31	2.61	446
CRR	1.30	2.57	441
Twice	1.21	2.64	428

Appendix

Table 19: Search behavior (Qwen2.5-3B): mean search calls per question and generated tokens per episode.

Question (gold)	Turn	Factual decision	Twin alternative	$U$ / $U^{\prime}$	$D$	$D_{\mathrm{CRR}}$
Which magazine was started first, Arthur’s Magazine or First for Women? (Arthur’s Magazine)	1	search: Arthur’s Magazine first published	search: Arthur’s Magazine First for Women history	1 / 1	0	$+1$
	2	search: First for Women magazine launch year	answer now: First for Women	1 / 0	$+1$	$+1$
Who was president of the United States in the year that Citibank was founded? (James Madison)	1	search: Citibank founded year	search: Citibank history	1 / 0	$+1$	$+1$
	2	search: president of the United States in 1812	search: US presidents list 1800s	1 / 1	0	$+1$
Were Scott Derrickson and Ed Wood of the same nationality? (yes)	2	search: Ed Wood nationality	search: Ed Wood filmmaker (same passages)	1 / 1	0 (skip)	0
	3	search: Scott Derrickson American director	answer now: yes	1 / 1	0	$+1$

Appendix

Table 20: Twins in training episodes (Qwen2.5-3B).

U/U^{\prime}

: factual and twin utilities.

D

is twin credit;

D_{\mathrm{CRR}}

is CRR’s independent difference. Twins tie when a decision does not matter under the shared noise, whereas independent counterfactuals assign spurious

\pm 1

credit to such decisions.

Step-Level Credit Assignment in RL

Step-level credit assignment in reinforcement learning links delayed rewards to the actions or reasoning steps that produced them.

See all 165 papers

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
Jun 30, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +6Agentic RL Step-Level Credit Assignment in RL
1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University +1
ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning
Sep 28, 2026Yang Li, Jinhan Yang, hai liu +8Agentic RL Credit Assignment in RL
University of Hong Kong · The Chinese University of Hong Kong · Jiangxi Science and Technology Normal University +1
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology

Jun 30, 2026J/K move · Enter open · S save

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

Yuanda Xu, Zhengze Zhou, Hejian Sang, Xiaomin Li+5

1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University+1

Agentic RL Step-Level Credit Assignment in RL Credit Assignment Policy Learning Reward Design for RL Fine-Grained Credit Assignment+1

Open paper AskPDF arXiv

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional

10.4\%

and

14.8\%

relative to GRPO.

Abstract

Step-Level Credit Assignment in RL

Explore similar work

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning

CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning