MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

Zhixia Zhang; Binqian Xu; Huaiyang Wang; Yikun Ban; Dianbo Liu

cs.LGOct 23, 2026

MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

Authors: Zhixia Zhang, Binqian Xu, Huaiyang Wang, Yikun Ban, Dianbo Liu

Organizations: Beihang University · National University of Singapore

Abstract

Scientific, engineering, and mathematical reasoning tasks can admit multiple valid solutions, while exhaustive annotation of the full solution set is often prohibitively costly. Test-time reinforcement learning (TTRL) applies reinforcement learning to unlabeled test instances, offering a label-free approach to multiple-solution reasoning. In this setting, generation frequency serves two opposing objectives: correctness estimation favors frequent outcomes, whereas outcome-level diversity preservation favors less frequent outcomes. We term this tension the Frequency Conflict. To resolve this conflict, we propose MS-TTRL, which obtains frequency-independent correctness estimates through Outcome-Level Self-Verification while reserving generation frequency solely for diversity preservation. Three-State Outcome Cache and the Candidate State support verification reuse across repeated observations and confidence-aware handling of ambiguous outcomes, while Confidence-Gated Inverse Probability Scaling selectively strengthens the learning signals of rare, reliably verified outcomes. Experiments across two benchmarks, four multiple-solution tasks, and three backbone models, each evaluated with five random seeds, show that MS-TTRL improves Recovery Rate by 19.13 percentage points on average over the best-performing label-free baseline and approaches the performance of the supervised IPS-GRPO.

Figures & tables

Figure 1: Frequency conflict and signal decoupling in MS-TTRL. Left: Frequency-based correctness estimation favors common outcomes, whereas diversity preservation favors rare ones. Middle: Self-verification estimates correctness, while outcome frequency promotes diversity. Right: MS-TTRL achieves higher Recovery Rate than TTRL and TTRL-CoCoV across four tasks.

Figure 2: Overview of MS-TTRL. For each unlabeled problem, generated solution outcomes are self-verified and maintained in a three-state outcome cache. Their stored status is then combined with gated inverse-probability scaling to reward outcomes, thereby decoupling correctness estimation from diversity preservation with limited cost of self-verification.

Qwen3-1.7B
	Causal (@16)		3D (@27)		Boolean (@20)
Method	RR(%) $\uparrow$	ORR(%) $\uparrow$	RR(%) $\uparrow$	ORR(%) $\uparrow$	RR(%) $\uparrow$	ORR(%) $\uparrow$
IPS-GRPO †	$51.20\pm 0.70$	$28.19\pm 0.53$	$27.75\pm 0.15$	$16.35\pm 0.04$	$46.15\pm 0.41$	$37.38\pm 0.57$
TTRL	$36.94\pm 0.00$	$17.42\pm 0.00$	$9.07\pm 0.00$	$5.28\pm 0.00$	$31.03\pm 0.13$	$22.63\pm 0.13$
TTRL-CoCoV	$37.54\pm 0.11$	$17.63\pm 0.04$	$9.07\pm 0.00$	$5.28\pm 0.00$	$32.88\pm 1.38$	$25.11\pm 0.74$
MS-TTRL	$45.89\pm 0.49$	$23.26\pm 0.20$	$28.39\pm 0.04$	$16.93\pm 0.03$	$46.00\pm 1.10$	$36.49\pm 0.51$

Table 1: Main results over 5 random seeds on the three HypoSpace subsets. RR and ORR denote Recovery Rate and Outcome Recovery Rate, respectively. Higher is better for both metrics. IPS-GRPO, as shown in light gray, is a label-supervised reference rather than a baseline . “Best” denotes the better-performing result between TTRL and TTRL-CoCoV for each metric.

Method	RR (%) $\uparrow$	ORR (%) $\uparrow$
IPS-GRPO †	$68.19\pm 0.49$	$55.26\pm 0.57$
TTRL	$34.39\pm 0.25$	$22.75\pm 0.21$
TTRL-CoCoV	$33.75\pm 0.25$	$22.34\pm 0.12$
MS-TTRL	$44.26\pm 0.56$	$34.74\pm 0.56$
$\Delta$ (MS-TTRL $-$ Best)	$\mathbf{+9.87}$	$\mathbf{+11.99}$

Table 2: Performance on the MACE-Math benchmark. Higher is better for all metrics.

Table 3: Training time and peak GPU memory on Qwen3-1.7B across three tasks. Lower is better.

Figure 3: Ablation of the Three-State Outcome Cache on Qwen3-1.7B across the Causal, 3D, and Boolean tasks. Removing the cache mechanism substantially increases the number of verified samples throughout training, showing that reusing previously stored verification states effectively reduces repeated self-verification and lowers the additional inference cost.

Configuration	Causal
Configuration	ACC(%) $\uparrow$	RR(%) $\uparrow$	ORR(%) $\uparrow$
w/o Gated IPS	$59.59\pm 0.68$	$49.46\pm 0.79$	$27.68\pm 0.54$
w/o Candidate Set (high)	$99.93\pm 0.06$	$37.15\pm 0.28$	$17.50\pm 0.10$
w/o Candidate Set (mid)	$74.80\pm 0.90$	$41.79\pm 0.77$	$20.55\pm 0.49$
w/o Candidate Set (low)	$63.59\pm 0.62$	$45.12\pm 0.44$	$23.81\pm 0.49$
MS-TTRL	$93.75\pm 0.46$	$45.89\pm 0.49$	$23.26\pm 0.20$

Table 4: Ablation study of Gated IPS and the Candidate Set on Qwen3-1.7B for the Causal task. Results are averaged over five random seeds.

Verification	Causal		3D
Verification	RR (%) $\uparrow$	ORR (%) $\uparrow$	RR (%) $\uparrow$	ORR (%) $\uparrow$
4	41.54	20.64	27.08	15.63
8	45.89	23.26	28.41	16.88
16	46.40	26.13	27.11	15.78

Table 5: Sensitivity to the number of self-verification trials on Qwen3-1.7B for Causal and 3D Structure Reconstruction.

$\lambda$	1.00	0.95	0.90	0.85	0.80	0.75
RR (%) $\uparrow$	10.35	26.05	28.39	27.20	20.69	18.14
ORR (%) $\uparrow$	5.52	15.52	16.93	15.82	12.28	10.56

Table 7: Sensitivity to the diversity scaling coefficient

\lambda

on Qwen3-1.7B for the 3D Structure Reconstruction task.

	Causal (@16)			3D (@27)			Boolean (@20)
Method	Acc(%) $\uparrow$	RR(%) $\uparrow$	ORR(%) $\uparrow$	Acc(%)	RR(%)	ORR(%)	Acc(%)	RR(%)	ORR(%)
Self-Verify	93.41	46.92	25.00	97.57	27.10	15.78	87.86	46.28	37.39
TTRL	99.97	36.94	17.42	99.87	9.07	5.28	81.79	31.03	22.63
TTRL-2	18.95	19.26	10.06	22.10	2.81	2.19	66.25	35.44	26.33
TTRL-3	1.29	1.73	0.94	0.00	0.00	0.00	52.14	36.80	30.61
TTRL-4	1.42	2.62	1.82	2.68	1.71	1.52	45.54	39.07	31.01

Table 8: Effect of extending TTRL from majority voting to the top-

k

most frequent outcomes on Qwen3-1.7B across three multiple-solution tasks. TTRL-

k

treats the top-

k

generated outcomes as positive pseudo-labels. All methods use inverse-probability scaling when assigning rewards.

Figure 4: Learning signals assigned to rare valid outcomes on Qwen3-1.7B. For each rollout group containing at least two distinct valid outcomes, we identify the valid outcome(s) with the lowest generation frequency. Each subfigure reports the mean advantage and positive advantage rate across training epochs for one task.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-1.7B
Method	Causal ACC(%) $\uparrow$	3D ACC(%) $\uparrow$	Boolean ACC(%) $\uparrow$
IPS-GRPO †	$95.14\pm 0.63$	$99.82\pm 0.13$	$88.54\pm 0.79$
TTRL	$99.97\pm 0.03$	$99.87\pm 0.07$	$81.79\pm 0.81$
TTRL-CoCoV	$97.73\pm 0.18$	$99.97\pm 0.05$	$77.79\pm 0.91$
MS-TTRL	$93.75\pm 0.46$	$99.13\pm 0.53$	$90.54\pm 0.99$
Qwen3-8B

Appendix

Table 9: Accuracy over 5 evaluation seeds on the three HypoSpace subsets. Higher is better.

Method	ACC (%) $\uparrow$
IPS-GRPO †	$82.83\pm 0.64$
TTRL	$65.99\pm 0.29$
TTRL-CoCoV	$62.86\pm 0.35$
MS-TTRL	$67.00\pm 0.35$

Appendix

Table 10: Accuracy on the MACE-Math benchmark using Qwen3-1.7B. Higher is better.

Panel of Figure 5 — Figure 5: Training dynamics on the Causal (top), 3D (middle), and boolean (bottom) tasks. From left to right: (a) accuracy, (b) recovery rate, (c) outcome recovery rate.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning
Oct 12, 2026Yi Li, Linan Yue, Weibo GaoTest-Time RL RL for Language Model Reasoning
Southeast University · The Hong Kong Polytechnic University
Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting
May 19, 2026Hongxiang Lin, Zhirui Kuai, Erpeng Xue +1Reinforcement Learning RL for Language Model Reasoning
Meituan
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model Reasoning Test-Time RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University

Oct 12, 2026J/K move · Enter open · S save

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Yi Li, Linan Yue, Weibo Gao

Southeast University · The Hong Kong Polytechnic University

Test-Time RL RL for Language Model Reasoning

Open paper AskPDF arXiv

Test-Time Reinforcement Learning (TTRL) enables large language models to adapt directly on unlabeled test problems by constructing pseudo-labels from multiple sampled responses. However, existing TTRL methods typically apply the same expensive sampling and optimization procedure to all test problems, despite substantial differences in their training utility. In this paper, we study whether such problems can be identified before costly test-time training, without access to ground-truth answers. We uncover a stable relationship between internal response representations and problem-level empirical accuracy. Specifically, we define COS as the cosine similarity between a model's response representation and a correctness-related direction constructed from an independent question-answering dataset. COS consistently correlates with problem solvability across different models, datasets, and experimental settings, and we further provide a theoretical analysis explaining this relationship. Building on this finding, we propose SelectTTRL, a representation-guided data selection method that ranks test problems using COS and applies TTRL only to a selected percentile interval. SelectTTRL leaves the standard TTRL pipeline unchanged for the selected problems, while avoiding unnecessary multi-response sampling and reinforcement learning updates on the remaining ones. Experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL. Code is available at https://github.com/heyiliyi/SelectTTRL-public.

Abstract

Test-Time RL

Explore similar work

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time