SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Yi Li; Linan Yue; Weibo Gao

cs.LGOct 12, 2026

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Authors: Yi Li, Linan Yue, Weibo Gao

Organizations: Southeast University · The Hong Kong Polytechnic University

Abstract

Test-Time Reinforcement Learning (TTRL) enables large language models to adapt directly on unlabeled test problems by constructing pseudo-labels from multiple sampled responses. However, existing TTRL methods typically apply the same expensive sampling and optimization procedure to all test problems, despite substantial differences in their training utility. In this paper, we study whether such problems can be identified before costly test-time training, without access to ground-truth answers. We uncover a stable relationship between internal response representations and problem-level empirical accuracy. Specifically, we define COS as the cosine similarity between a model's response representation and a correctness-related direction constructed from an independent question-answering dataset. COS consistently correlates with problem solvability across different models, datasets, and experimental settings, and we further provide a theoretical analysis explaining this relationship. Building on this finding, we propose SelectTTRL, a representation-guided data selection method that ranks test problems using COS and applies TTRL only to a selected percentile interval. SelectTTRL leaves the standard TTRL pipeline unchanged for the selected problems, while avoiding unnecessary multi-response sampling and reinforcement learning updates on the remaining ones. Experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL. Code is available at https://github.com/heyiliyi/SelectTTRL-public.

Figures & tables

	MATH-500			GPQA-Diamond			MMLU-Pro STEM
Method	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16
Matched-TTRL	86.20	87.18	92.40	51.01	51.33	57.07	75.30	76.61	79.52
Random	86.00	86.71	91.20	49.49	50.95	54.55	75.70	75.41	79.32
A-PPL	87.40	86.84	91.00	53.54	50.69	56.06	75.10	76.58	81.53
SelectTTRL	86.20	86.55	92.60	51.52	50.73	58.08	74.10	75.77	80.12
	AIME 2024			AIME 2025			AMC

Table 1: Comparison of TTRL methods on Qwen3-8B across six reasoning benchmarks.

	MATH-500			GPQA-Diamond			MMLU-Pro STEM
Method	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16
Matched-TTRL	78.80	78.34	82.00	35.35	35.04	35.86	64.66	65.31	69.28
Random	79.80	77.68	80.80	36.67	36.81	40.40	64.46	63.45	67.27
A-PPL	78.40	78.04	82.40	32.83	35.86	37.88	64.66	62.79	66.47
SelectTTRL	79.60	78.61	83.00	36.87	35.61	38.89	62.65	64.22	68.47
	AIME 2024			AIME 2025			AMC

Table 2: Comparison of TTRL methods on Qwen2.5-7B-Instruct across six reasoning benchmarks.

Figure 1: CAA and norm-matched random steering results on Qwen3-8B. Random results are means across five directions.

Condition	Accuracy (%)	Mean COS	$\Delta$ Accuracy (pp)	$\Delta$ COS
Baseline	30.00	0.1238	—	—
Ablate 31368	20.00	0.1115	$-10.00$	$-0.0123$
Ablate 52642	26.67	0.1239	$-3.33$	$+0.0001$
Ablate 41030	23.33	0.1212	$-6.67$	$-0.0026$
Ablate 3447	23.33	0.1194	$-6.67$	$-0.0045$
Ablate 48965	20.00	0.1220	$-10.00$	$-0.0018$

Table 3: SAE feature ablation on AIME24. Numbers denote SAE feature IDs.

Direction source	Source Greedy (%)	Mean Pearson $r$	Mean Spearman $\rho$
PIQA	87.65	0.6420	0.6273
CommonsenseQA	81.08	0.6340	0.6116
MMLU-Pro	72.69	0.6260	0.6071
LogiQA 2.0	74.95	0.5942	0.5692
GPQA	44.95	0.5385	0.5126
TruthfulQA	71.97	0.5328	0.4519

Table 4: Comparison of direction-extraction datasets. Source Greedy is the model’s accuracy on the source dataset. Pearson

r

and Spearman

\rho

between COS and question-level accuracy are averaged across MATH-500, AIME24, AIME25, and AMC.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\widehat{\beta}$	95% CI	$\rho(c_{A},\widehat{p}_{B})$	$\rho(c_{B},\widehat{p}_{A})$
MATH-500	0.02271	$[0.01815,\,0.02821]$	0.296	0.338
GPQA-Diamond	0.01602	$[0.01125,\,0.02081]$	0.249	0.244
MMLU-Pro STEM	0.02131	$[0.01803,\,0.02460]$	0.434	0.464
AIME 2024	0.05077	$[0.03646,\,0.06377]$	0.536	0.610

Appendix

Table 5: Class-projection gaps and cross-half COS–accuracy correlations. Confidence intervals for

\widehat{\beta}

use 1,000 question-level bootstrap resamples, retaining all responses of each sampled question.

Dataset	$\widehat{C}_{\mathrm{total}}$	$\widehat{C}_{\mathrm{proj}}$	$\widehat{C}_{\mathrm{res}}$
MATH-500	2.158 $[1.535,\,2.834]$	1.752 $[1.276,\,2.267]$	0.407 $[0.136,\,0.673]$
GPQA-Diamond	1.884 $[0.993,\,2.842]$	1.879 $[1.302,\,2.425]$	0.005 $[-0.591,\,0.627]$
MMLU-Pro STEM	2.909 $[2.349,\,3.491]$	2.471 $[2.017,\,2.927]$	0.438 $[0.169,\,0.708]$
AIME 2024	7.533 $[2.509,\,9.894]$	6.738 $[2.290,\,8.874]$	0.796 $[-1.364,\,2.678]$

Appendix

Table 6: Cross-half covariance decomposition. All values are multiplied by

10^{3}

; brackets give 95% bootstrap confidence intervals.

Component	High COS	Low COS	Difference	Share (%)
SAE features	0.07968	0.05503	$+0.02465$	47.6
Decoder bias	0.05289	0.05538	$-0.00249$	$-4.8$
Reconstruction residual	0.04675	0.01714	$+0.02962$	57.2
Total COS	0.17933	0.12755	$+0.05178$	100.0

Appendix

Table 7: Component contributions to the high–low COS difference on AIME24. Each group contains five questions. Values are rounded.

Feature ID	Observed text or symbol	Location
32531 , 47984		Formula-ending delimiter after the final answer
3447	So ; This ; exact expression is	Connective or result-statement expression
41030	58500 ; 2000 ; 4 inches per second	Numerical results and units
31368	find the sum ; how many ; ordered quadruple	Expression of the quantity or object to be found
605	The digit 7 in \boxed{27}	Numerical token within the final answer

Appendix

Table 8: Examples of high-activation locations for COS-related SAE features in MATH-500 responses.

	Reduction (%)			Time saved
Dataset	Generated tokens	Update response tokens	Training-step time	Wall-clock (min)	GPU-hours
MATH-500	17.07	17.14	5.43	14.23	0.949
MMLU-Pro STEM	19.56	18.93	6.68	18.02	1.202
GPQA-Diamond	2.88	3.24	0.66	1.43	0.095
AIME 2024	10.74	10.73	3.73	12.96	0.864
AIME 2025	26.12	25.93	10.86	36.66	2.444

Appendix

Table 9: Training-efficiency reductions and time savings across six benchmarks.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model Reasoning Test-Time RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
TTSR: Test-Time Self-Evolving via Reflection
Feb 6, 2026Haoyang He, Zihua Rong, Yunjia Zhao +3LLM Self-Refinement Test-Time Optimization
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.
Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces
Sep 16, 2026Naveen Vakada, Mingyuan Li, Shaoxiong JiFine-Tuning Test-Time Adaptation of VLMs
University of Turku, Finland · ELLIS Institute Finland

Mar 20, 2026J/K move · Enter open · S save

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu+2

School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University

RL for Language Model Reasoning Test-Time RL Pseudo-Labeling Learning with Noisy Labels

Open paper AskPDF arXiv

Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.

Abstract

Test-Time RL

Explore similar work

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

TTSR: Test-Time Self-Evolving via Reflection

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces