$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training

Nan Qiao; Yebin Yang; Weinong Wang; Shuning Wang; Shangpin Peng; Fengyuan Lu; Xinming Wang; Zhehan Kan

cs.AISep 26, 2026

$T^5$ : Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training

Authors: Nan Qiao, Yebin Yang, Weinong Wang, Shuning Wang, Shangpin Peng, Fengyuan Lu, Xinming Wang, Zhehan Kan,

Organizations: Tsinghua University · Tencent · Shanghai Jiao Tong University · Central South University ·

Abstract

Reinforcement mid-training lets language models learn internal thoughts from unlabeled text, but efficient token-level credit assignment remains challenging. Existing group-relative methods require costly repeated generation. Learned critics offer single-rollout feedback, but accurate return prediction alone does not ensure reliable policy updates. Our analysis shows how training--inference mismatch and PPO clipping prevent a common offset in advantage estimates from cancelling out, introducing additional update drift. We propose \tfour{}, a twin-critic method that calibrates token-level advantages from a single generated trajectory. After warmup and held-out qualification, the critics provide two advantage estimates, combined using action-dependent weights learned through a conditional-moment saddle-point objective. This objective brings the average advantage at each prefix toward zero, while a signal-retention constraint prevents the correction from erasing the learning signal. Sharing information across text positions avoids repeated sampling of each prefix. Theoretically, we characterize optimal mixing under the signal-retention constraint and establish an upper bound on residual mean-induced drift. Experiments show that, compared with the state-of-the-art critic-free method, \tfour{} improves mean benchmark performance by 7.8% and reduces mean training-step time by up to 63.4%.

Figures & tables

Figure 1: Naive PPO gains (nats) over training steps. Left: CSQA (blue) and GSM8K (orange) thought gains. Right: CSQA mixed gain (teal dashed). Shading: 20-step critic warmup with the actor frozen.

Figure 2: Why calibrate twin-critic advantages? Schematic conditional distributions at a fixed prefix, showing normalized advantage against a scalar projection of the masked update factor $v$ . (a) Centered advantages retain an action-contrast signal. (b) A common offset adds a mean-induced component when the mean update factor is nonzero. (c) Twin-advantage mixing targets the offset while retaining aggregate signal.

Panel of Figure 3 — Figure 3: Experimental comparisons. (a) Selected Qwen3.5-2B results (PIQA: accuracy; others: pass@1). (b) Earlier controlled comparison with a shared Quiet-STaR architecture: benchmark scores (%, left) and performance over training time from the common Base checkpoint (right). Legend labels PPO and GRPO denote Quiet-STaR (w/ PPO) and Quiet-STaR (w/ GRPO), respectively.

Model	RLP	$\mathbf{T}^{5}$
Qwen3.5-2B	54.01	31.79
Qwen3.5-9B	497.83	182.18

Table 1: Mean step time (s).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Parameters	Storage
Qwen/Qwen3.5-2B-Base	2.274B	4.56 GB
Qwen/Qwen3.5-9B-Base	9.653B	19.32 GB

Appendix

Table 2: Target Base-model initializations.

Source	Share
FineWeb: CC-MAIN-2025-13/18/21/26	85%
2025–2026 open-index/open-arxiv titles and abstracts	10%
OpenWebMath	5%

Appendix

Table 3: Training-corpus composition. The four web crawls are from 2025.

Split	Sequences	Corpus tokens
Training	12,800	6,553,600
Critic calibration	1,024	524,288
Development	1,024	524,288

Appendix

Table 4: Corpus splits, with 512 tokens per sequence. Counts exclude generated thoughts and auxiliary continuations.

Method	Group	Update	Controlled role
Base	–	None	Original checkpoint with no additional corpus exposure.
NTP	–	Next-token likelihood	Controls for continued exposure to the same training mixture.
Quiet-STaR (w/ PPO)	1	Single-critic clipped PPO	Uses the hidden-thought interface with a standard learned-value update.
RLP	8	GRPO-style group-relative update	Samples eight thoughts and uses next-token information-gain rewards.
Quiet-STaR (w/ GRPO)	8	Group-relative update	Applies GRPO to the same hidden-thought architecture.
$\mathrm{T}^{5}$	1	Twin-critic clipped PPO	Uses predictive qualification and constrained saddle-point calibration.

Appendix

Table 5: Methods in the reported Qwen3.5-2B comparison. A dash indicates no grouped thought sampling. All GRPO-based comparisons in this paper use group size 8.

Release	Benchmark	Items	Focus
2025	AIME 2025	30	Competition mathematics
2026	AIME 2026	30	Competition mathematics
2021	GSM8K test	1,319	Grade-school mathematics
2021	MATH-500	500	Competition mathematics
2019	CommonsenseQA validation	1,221	General commonsense
2020	PIQA validation	1,838	Physical commonsense

Appendix

Table 6: Primary evaluation suite and frozen item counts.

Benchmark	Base	NTP	Quiet-STaR (w/ PPO)	RLP	Quiet-STaR (w/ GRPO)	$\mathrm{T}^{5}$
CSQA	66	67	64	74	67	75
PIQA	75	75	75	76	69	81
ARC-Challenge	74	74	75	88	86	90
GSM8K	79	78	71	82	77	85
MATH-500	38	36	39	64	56	85
AIME 2025	30	33	27	36	33	38

Appendix

Table 7: Qwen3.5-2B primary results on the seven retained benchmarks, rounded to integer percentages. Bold and underlined entries identify the best and second-best unrounded values in each row. The final row rounds the unweighted mean of the seven source values.

Benchmark	Metric	Primary	Pass@4	Coverage	Avg. tokens
AIME 2025	pass@1	37.50	53.33	100.00	3849.1
AIME 2026	pass@1	34.17	50.00	100.00	3922.4
ARC-Challenge	Accuracy	89.59	–	100.00	609.2
CSQA	Accuracy	74.53	–	100.00	649.2
GSM8K	pass@1	84.80	93.18	100.00	645.7
MATH-500	pass@1	85.45	89.60	100.00	1673.6

Appendix

Table 8: Qwen3.5-2B

\mathrm{T}^{5}

evaluation diagnostics. Primary scores, pass@4, and coverage are percentages. A dash marks tasks for which pass@4 was not supplied.

Figure 4: Qwen3.5-2B

\mathrm{T}^{5}

diagnostics. (a) Paired pass@1 and pass@4 with percentage-point differences. (b) Average reasoning tokens per response, with rounded endpoint labels. Coverage is 100% on all seven tasks.

Figure 5: Qwen3.5-2B

\mathrm{T}^{5}

warmup-parameter ablation on AIME 2026. Scores are plotted against measured warmup time, with point labels identifying the score and

\tau

Benchmark	Base	Quiet-STaR (w/ GRPO)	$\mathrm{T}^{5}$
MATH500	48.17	57.96	58.10
GSM8K	54.47	72.42	75.00
AMC23	26.19	30.76	31.70
Minerva	15.50	19.40	21.00
MMLU	50.61	42.40	55.80
MMLU pass@1	45.10	40.49	51.90

Appendix

Table 9: Complete preliminary benchmark summary (%). Bold indicates the largest reported value in each row. The three additional rows report pass@1.

Figure 6: Recorded per-step training times for RLP and

\mathrm{T}^{5}

on Qwen3.5-2B and Qwen3.5-9B. Faint lines show raw timers, solid lines show 15-step rolling medians, and dotted lines mark the reported means. The shaded 2B region is the final continuous

\mathrm{T}^{5}

segment used for its mean;

J_{1}

and

J_{2}

mark segment joins. The logarithmic vertical axis reports seconds.

Figure 7: Qwen3.5-9B comparison of Base, NTP, and

\mathrm{T}^{5}

. Labels preserve the supplied scores in percent. The four mathematics tasks use pass@1, and the three multiple-choice tasks use accuracy. CSQA abbreviates CommonsenseQA. The vertical axis starts at 70, and the footer reports unweighted means over all seven tasks.

Figure 8: Common-method comparison between Qwen3.5-2B and Qwen3.5-9B. Each panel pairs a method’s scores on the same seven tasks. Light bars show 2B and darker bars show 9B, with identical score axes across panels. The 2B bars use unrounded source values, with labels rounded to one decimal. Headings report the seven-task mean at each scale.

Figure 9: Historical naive-PPO ablation-style diagnostic at seven checkpoints. (a) Thought gain (left axis) and CSQA mixed gain (right axis). (b) Useful-position fraction (left axis) and CSQA worst-1% thought gain (right axis). The no-thought prediction has zero gain by construction. Shading marks the 20-step critic warmup with the actor frozen; endpoint labels are rounded values reconstructed from the original vector figure.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 437 papers

The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
May 9, 2026Tianhao Cheng, Zeyu Huang, Zihan Qiu +5Reinforcement Learning Token-Level Credit Assignment
1Fudan University · 3The University of Edinburgh · 6Qwen Team, Alibaba Group +4
Best Practice Critic Optimization
Aug 24, 2026Penghui Qi, Xiangxin Zhou, Wee Sun LeeRL for Language Model Reasoning Actor-Critic Methods
National University of Singapore · Tencent Hunyuan
Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
May 14, 2026Langzhou He, Junyou Zhu, Yue Zhou +7Agentic RL Token-Level Credit Assignment
University of Illinois Chicago · Potsdam Institute for Climate Impact Research · Technical University of Berlin +3

May 9, 2026J/K move · Enter open · S save

The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits

Tianhao Cheng, Zeyu Huang, Zihan Qiu, Yu Cheng+4

1Fudan University · 3The University of Edinburgh · 6Qwen Team, Alibaba Group+4

Reinforcement Learning Token-Level Credit Assignment RL for Language Model Reasoning Critic-Free RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

A commonly accepted explanation of critic-free RL for LLMs, based on sequence-level rewards, is that it reinforces successful rollouts with a positive advantage while penalizing failed ones. In contrast, we study critic-free RL from a token-level perspective, revealing the token-flipping phenomenon: positive and negative rollouts exhibit remarkably similar proportions of tokens whose probabilities are boosted or suppressed during RL training. To explain this phenomenon, we further show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role. Specifically, these token coupling effects occur primarily between identical tokens that are both predicted with low confidence. Building upon this analysis, we propose the cancellation hypothesis: as a result of coupling, opposing signals cancel out for tokens shared by positive and negative rollouts, while tokens more specific to successful rollouts receive stronger reinforcement, thereby inducing hidden token-level credit assignment from rollout-level rewards. We support this hypothesis with complementary empirical evidence. (1) Compared with training on only positive rollouts, critic-free RL shifts updates from template and formatting tokens toward reasoning tokens; (2) Tokens boosted by critic-free RL consistently demonstrate higher value than suppressed tokens, regardless of whether they originate from positive or negative rollouts. Guided by this view, we implement two batching interventions to encourage or preserve cancellation in critic-free RL training: query-preserved mini-batching and reward-balanced batching. Despite their simplicity, these interventions improve RLVR training across multiple model scales, supporting cancellation as both an explanatory principle and a practical design criterion for critic-free RL training.

Abstract

RL for Language Models

Explore similar work

The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits

Best Practice Critic Optimization

Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy