Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

Qifan Shi; Zhaolu Kang; Chenghua Zhu

cs.AIAug 21, 2026

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

Authors: Qifan Shi, Zhaolu Kang, Chenghua Zhu

Abstract

Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport operator that converts this evidence into token-level advantages, and an update geometry that turns advantages into policy changes. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture-agnostic. Fixed-discount GAE applies a stationary geometric kernel along token time; group-relative methods broadcast an outcome statistic across an entire response. Neither operator represents the trajectory-specific computation used by the Transformer policy itself. We introduce computation-conditioned credit transport (CCT), a general framework in which a detached statistic of the behavior policy's internal computation parameterizes the causal kernel that transports downstream value through a rollout. Our concrete algorithm, CompPO, maps native attention concentration to a bounded per-token retention gate, uses the gate in both the one-step bootstrap and a path-dependent generalized-advantage trace (Comp-GAE), and co-designs a transport-aligned critic (TAC) that reuses the actor's hidden states and routing information without a second same-scale Transformer. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches 61.4% final held-out accuracy (95% CI [60.8,62.0]) versus 53.8% [52.9,54.7] for tuned GRPO. Neither Comp-GAE with a standard critic (55.2%) nor TAC with a fixed gate (56.4%) matches the full model (interaction +2.4 [1.9,2.9]). Shuffle and position controls confirm trajectory-specific alignment; CompPO is stable in 10/12 PPO-grid runs versus 3/12. Frozen evaluation improves over GRPO by 4.3 and 3.9 greedy pass@1 macro points on Qwen3-4B and Llama-3.1-8B-Instruct.

Figures & tables

Panel of Figure 1 — Figure 1: Credit assignment factorized into evidence, transport, and update geometry (top), with three transport coordinates used in LLM RL (bottom). Fixed GAE uses a stationary temporal kernel; group-relative methods broadcast response-level evidence; CompPO lets behavior-policy computation parameterize the causal transport path.

Panel of Figure 2 — Figure 2: CompPO changes the credit-estimation path while preserving the task reward and outer clipped policy update. A native policy-computation statistic yields a detached gate $\kappa_{t}$ . The same gate controls one-step bootstrap, eligibility-trace continuation, and the local/global geometry of a critic that reuses actor features.

Panel of Figure 3 — Figure 3: Main training evidence. CompPO improves both best and final endpoints over the selected GRPO baseline, and the factorial experiment shows that the trajectory-specific gate and aligned critic are complementary rather than interchangeable.

Variant	Best	Final	Last-20	AULC	Peak–final	EV	TD $\rho_{s}$	MAE	Clamp sat.	Step to 56.4
Fixed gate + standard critic	53.4	52.6	52.0	47.4	0.8	$-0.15$	0.11	0.41	0.24	never
Comp-GAE + standard critic	56.0	55.2	54.7	50.2	0.8	$-0.08$	0.16	0.37	0.19	never
Fixed gate + aligned critic	57.2	56.4	55.9	53.1	0.8	0.45	0.52	0.20	0.05	112
Full CompPO	61.7	61.4	60.8	58.9	0.3	0.52	0.57	0.18	0.03	33

Table 1: Five-seed gate-by-critic factorial. Best and final means have 95% Student-

t

intervals: Fixed+Std

53.4[52.9,53.9]/52.6[52.0,53.2]

; Comp-GAE +Std

56.0[55.6,56.5]/55.2[54.6,55.8]

; Fixed+Aligned

57.2[56.6,57.8]/56.4[55.8,57.0]

; Full

61.7[61.3,62.1]/61.4[60.8,62.0]

Panel of Figure 4 — Figure 4: Specificity and robustness controls. Destroying token-wise alignment or replacing it with a fixed positional profile reduces performance, while CompPO occupies a wider stable region in the tested PPO grid.

Backbone	Base	GRPO	CompPO	$\Delta$
Qwen3-4B, greedy	58.2	57.5	61.7	+4.3
Qwen3-4B, majority@8	69.7	68.7	71.0	+2.3
Llama-3.1-8B, greedy	31.1	39.0	42.8	+3.9
Llama-3.1-8B, majority@8	36.5	47.8	52.0	+4.2

Table 2: Unweighted frozen-benchmark macro-averages.

\Delta

is CompPO minus GRPO. These are final-checkpoint point estimates; complete per-benchmark results are in Tables 10 and 11 .

Family	Evidence or baseline	Transport geometry	Update or value mechanism	Relation to this work
PPO/GAE	learned scalar value	stationary temporal trace	token-ratio clipping; critic	fixed-kernel ancestor
GRPO/RLOO	group or leave-one-out outcome	response-level broadcast	critic-free token update	outcome-coordinate control
DAPO / Dr. GRPO / GSPO	improved sampling and normalization	mainly response/group credit	clipping, loss normalization, or sequence ratios	orthogonal optimizer advances
PRIME/PRM	process or implicit step evidence	reward placement plus downstream estimator	process/reward model	complementary evidence axis
VinePPO / TEMPO	continuation or prefix-tree values	rollout- or tree-based TD	Monte Carlo or nonparametric value	alternative fine-grained transport
OAR / S-trace / GRPO- $\lambda$	influence, entropy, or trace proxy	reshaping or selective traces	critic-free update	neighboring token-credit methods

Table 3: Positioning by the evidence–transport–update factorization. Categories identify the primary intervention, not an exclusive taxonomy; many systems combine multiple axes.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Actor LR	KL	Best	Final
GRPO	$5\mathrm{e}{-7}$	0	52.8	51.2
GRPO	$5\mathrm{e}{-7}$	$1\mathrm{e}{-3}$	54.0	52.4
GRPO	$5\mathrm{e}{-7}$	$2\mathrm{e}{-3}$	53.2	51.8
GRPO	$1\mathrm{e}{-6}$	0	54.6	52.8
GRPO	$1\mathrm{e}{-6}$	$1\mathrm{e}{-3}$	56.4	53.8
GRPO	$1\mathrm{e}{-6}$	$2\mathrm{e}{-3}$	55.0	53.2

Appendix

Table 4: Single-seed common

3\!\times\!3

screen used to select the five-seed configurations. Frozen benchmarks were not used.

Seed	GRPO Best	GRPO Final	CompPO Best	CompPO Final
17	55.6	52.9	61.2	60.8
42	56.4	53.8	61.8	61.4
123	55.8	53.4	61.5	61.2
256	56.6	54.2	61.9	61.6
2026	56.8	54.7	62.1	62.0
Mean	56.2	53.8	61.7	61.4

Appendix

Table 5: Five-seed selected-configuration confirmation.

	Fixed+Std		Comp-GAE +Std		Fixed+Aligned		Full
Seed	Best	Final	Best	Final	Best	Final	Best	Final
17	52.8	51.9	55.6	54.6	56.6	55.7	61.2	60.8
42	53.8	53.1	56.6	55.8	57.8	56.9	61.8	61.4
123	53.2	52.4	55.8	54.9	57.0	56.1	61.5	61.2
256	53.6	52.8	56.2	55.4	57.4	56.6	61.9	61.6
2026	53.6	52.8	56.0	55.3	57.2	56.7	62.1	62.0

Appendix

Table 6: Seed-level gate-by-critic factorial endpoints.

Variant	Best	Final	Last-20	AULC
Global shuffle	59.4 [58.9,59.9]	58.7 [58.0,59.4]	58.1	57.2
Position-only	60.2 [59.8,60.6]	59.7 [59.2,60.2]	–	–
Full CompPO	61.7 [61.3,62.1]	61.4 [60.8,62.0]	60.8	58.9

Appendix

Table 7: Five-seed gate controls. Brackets are 95% Student-

t

intervals.

Statistic	Value	Statistic	Value
Mean / median	0.61 / 0.63	IQR	0.18
Early / middle / late	0.52 / 0.63 / 0.68	Correct / incorrect	0.64 / 0.57

Appendix

Table 8: Realized gate statistics from five final full-method checkpoints. Archived tail and boundary-proximity summaries are excluded pending reconciliation of the exact implementation map with the printed smooth-map equation.

Method	Actor LR	KL	Best	Final	Stable seeds
PPO	$2\mathrm{e}{-7}$	0	48.2	44.8	1/2
PPO	$2\mathrm{e}{-7}$	$1\mathrm{e}{-3}$	49.0	46.2	1/2
PPO	$5\mathrm{e}{-7}$	0	46.8	31.0	0/2
PPO	$5\mathrm{e}{-7}$	$1\mathrm{e}{-3}$	48.4	43.6	1/2
PPO	$1\mathrm{e}{-6}$	0	44.2	18.4	0/2
PPO	$1\mathrm{e}{-6}$	$1\mathrm{e}{-3}$	49.8	22.6	0/2

Appendix

Table 9: Two seeds per LR–KL cell. Across 12 runs, PPO is stable in 3 and CompPO in 10.

		Greedy pass@1			Majority@8
Benchmark	$N$	Base	GRPO	CompPO	Base	GRPO	CompPO
AIME25	30	43.3	50.0	43.3	66.7	66.7	66.7
AIME2024	30	63.3	53.3	70.0	76.7	73.3	80.0
MATH500	500	67.2	66.6	69.8	79.8	79.2	81.0
GSM8K	1319	91.7	93.1	94.1	93.5	94.3	94.5
LiveBench	200	46.5	50.5	53.5	61.0	61.5	67.0

Appendix

Table 10: Frozen Qwen3-4B final-checkpoint point estimates.

	Greedy pass@1			Majority@8
Benchmark	Base	GRPO	CompPO	Base	GRPO	CompPO
AIME25	3.3	13.3	16.7	6.7	23.3	30.0
AIME2024	13.3	20.0	23.3	16.7	33.3	36.7
MATH500	53.4	64.2	69.6	64.4	76.6	80.6
GSM8K	81.3	86.4	87.6	85.6	90.4	91.2
LiveBench	22.5	35.0	39.0	29.0	44.5	49.0

Appendix

Table 11: Frozen Llama-3.1-8B-Instruct final-checkpoint point estimates.

Method	Trajectories/update	ID Best	ID Final	Frozen greedy macro
GRPO $4\!\times\!4$	16	56.4	53.8	57.5
GRPO $8\!\times\!8$	64	58.0	56.2	60.2
CompPO reference	16	61.8	61.4	61.7

Appendix

Table 12: Single-run stronger-group control. The CompPO row is the submitted-seed reference.

Item	GRPO	CompPO
Backbone	Qwen3-4B	same
Transfer backbone	Llama-3.1-8B-Instruct	same
Maximum response	16K	16K
Batch / responses / minibatch	4 / 4 / 4	same
Actor learning rate	$10^{-6}$	$10^{-6}$
Critic learning rate	–	$10^{-5}$

Appendix

Table 13: Selected primary settings. Both methods were screened on the same actor-LR/KL grid.

Phase	KL	Clip	Actor LR	Value-loss clip	Critic LR mult.
Stable	0.0020	0.20	$10^{-6}$	0.5	1.0
Warning	0.0035	0.16	$7\times 10^{-7}$	0.4	0.9
Strong	0.0045	0.14	$5\times 10^{-7}$	0.3	0.7
Hard-stop	0.0070	0.09	$2\times 10^{-7}$	0.2	0.5

Appendix

Table 14: Adaptive stability phases. The same controller is held fixed across the factorial, so it cannot induce the gate-by-critic interaction.

Token-Level Credit Assignment

Token-level credit assignment allocates sequence-level rewards across the words, actions, or reasoning steps that shape a model’s outcome.

See all 55 papers

Sep 22, 2026

PACT: From Credit Assignment to Critic Alignment

Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.

Jiayan Fu, Hang Xu, Yong Zhang +4

AllSpark Team

Reinforcement Learning Token-Level Credit Assignment RL for Language Model Reasoning Step-Level Credit Assignment in RL RL Post-Training Actor-Critic Methods+1

Ask PDF

May 29, 2026

ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate

Token-level credit assignment for language-model reinforcement learning is usually formulated as if the policy were fully trainable, while practical LLM-RL pipelines often rely on parameter-efficient fine-tuning, especially LoRA. We argue that this separation hides a structural failure mode. Under LoRA, the policy is restricted to a low-rank neighborhood of the reference model, so the per-token output-distribution differences used by common intrinsic credit signals, surprisal, entropy reduction, and policy divergence, can become degenerate after within-trajectory normalization, either approaching uniform weights or concentrating on a small set of task-agnostic positions. We formalize this behavior and propose measuring it directly with concentration diagnostics such as weight Gini and effective-token ratio. We then introduce Adapter-Residual Credit Assignment (ARCA), a lightweight alternative that derives token salience from the adapter's own hidden-state residual,

\|h^{\text{adapted}}_t - h^{\text{base}}_t\|_2

. ARCA asks where the adapter actually changes the model, rather than where the output distribution appears uncertain or shifted, and requires no learned reward model, value head, or tree construction. In a compact MATH/Qwen3-1.7B GRPO sweep, ARCA exhibits the predicted non-degenerate middle-regime credit distribution under matched rollout budgets and remains competitive with rank-matched baselines.

Rodney Lafuente-Mercado

Scale AI

Token-Level Credit Assignment RL for Language Model Reasoning Low-Rank Adaptation Credit Assignment Fine-Grained Credit Assignment

Ask PDF

Jul 3, 2026

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.

Zijun Xie, Yuyang You, Yongzhi Li +6

Peking University · Baidu Inc. · Kuaishou Inc.

Token-Level Credit Assignment RL for Language Model Reasoning Step-Level Credit Assignment in RL Maximum Entropy RL Policy Gradient Methods Sparse-Reward RL+1

Ask PDF

Abstract

Token-Level Credit Assignment

Explore similar work

PACT: From Credit Assignment to Critic Alignment

ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy