SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Yi Li; Linan Yue; Weibo Gao

cs.LGOct 12, 2026

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

Authors: Yi Li, Linan Yue, Weibo Gao

Organizations: Southeast University · The Hong Kong Polytechnic University

Abstract

Test-Time Reinforcement Learning (TTRL) enables large language models to adapt directly on unlabeled test problems by constructing pseudo-labels from multiple sampled responses. However, existing TTRL methods typically apply the same expensive sampling and optimization procedure to all test problems, despite substantial differences in their training utility. In this paper, we study whether such problems can be identified before costly test-time training, without access to ground-truth answers. We uncover a stable relationship between internal response representations and problem-level empirical accuracy. Specifically, we define COS as the cosine similarity between a model's response representation and a correctness-related direction constructed from an independent question-answering dataset. COS consistently correlates with problem solvability across different models, datasets, and experimental settings, and we further provide a theoretical analysis explaining this relationship. Building on this finding, we propose SelectTTRL, a representation-guided data selection method that ranks test problems using COS and applies TTRL only to a selected percentile interval. SelectTTRL leaves the standard TTRL pipeline unchanged for the selected problems, while avoiding unnecessary multi-response sampling and reinforcement learning updates on the remaining ones. Experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL. Code is available at https://github.com/heyiliyi/SelectTTRL-public.

Figures & tables

	MATH-500			GPQA-Diamond			MMLU-Pro STEM
Method	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16
Matched-TTRL	86.20	87.18	92.40	51.01	51.33	57.07	75.30	76.61	79.52
Random	86.00	86.71	91.20	49.49	50.95	54.55	75.70	75.41	79.32
A-PPL	87.40	86.84	91.00	53.54	50.69	56.06	75.10	76.58	81.53
SelectTTRL	86.20	86.55	92.60	51.52	50.73	58.08	74.10	75.77	80.12
	AIME 2024			AIME 2025			AMC

Table 1: Comparison of TTRL methods on Qwen3-8B across six reasoning benchmarks.

	MATH-500			GPQA-Diamond			MMLU-Pro STEM
Method	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16	Greedy	Mean1	Maj.@16
Matched-TTRL	78.80	78.34	82.00	35.35	35.04	35.86	64.66	65.31	69.28
Random	79.80	77.68	80.80	36.67	36.81	40.40	64.46	63.45	67.27
A-PPL	78.40	78.04	82.40	32.83	35.86	37.88	64.66	62.79	66.47
SelectTTRL	79.60	78.61	83.00	36.87	35.61	38.89	62.65	64.22	68.47
	AIME 2024			AIME 2025			AMC

Table 2: Comparison of TTRL methods on Qwen2.5-7B-Instruct across six reasoning benchmarks.

Figure 1: CAA and norm-matched random steering results on Qwen3-8B. Random results are means across five directions.

Condition	Accuracy (%)	Mean COS	$\Delta$ Accuracy (pp)	$\Delta$ COS
Baseline	30.00	0.1238	—	—
Ablate 31368	20.00	0.1115	$-10.00$	$-0.0123$
Ablate 52642	26.67	0.1239	$-3.33$	$+0.0001$
Ablate 41030	23.33	0.1212	$-6.67$	$-0.0026$
Ablate 3447	23.33	0.1194	$-6.67$	$-0.0045$
Ablate 48965	20.00	0.1220	$-10.00$	$-0.0018$

Table 3: SAE feature ablation on AIME24. Numbers denote SAE feature IDs.

Direction source	Source Greedy (%)	Mean Pearson $r$	Mean Spearman $\rho$
PIQA	87.65	0.6420	0.6273
CommonsenseQA	81.08	0.6340	0.6116
MMLU-Pro	72.69	0.6260	0.6071
LogiQA 2.0	74.95	0.5942	0.5692
GPQA	44.95	0.5385	0.5126
TruthfulQA	71.97	0.5328	0.4519

Table 4: Comparison of direction-extraction datasets. Source Greedy is the model’s accuracy on the source dataset. Pearson

r

and Spearman

\rho

between COS and question-level accuracy are averaged across MATH-500, AIME24, AIME25, and AMC.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\widehat{\beta}$	95% CI	$\rho(c_{A},\widehat{p}_{B})$	$\rho(c_{B},\widehat{p}_{A})$
MATH-500	0.02271	$[0.01815,\,0.02821]$	0.296	0.338
GPQA-Diamond	0.01602	$[0.01125,\,0.02081]$	0.249	0.244
MMLU-Pro STEM	0.02131	$[0.01803,\,0.02460]$	0.434	0.464
AIME 2024	0.05077	$[0.03646,\,0.06377]$	0.536	0.610

Appendix

Table 5: Class-projection gaps and cross-half COS–accuracy correlations. Confidence intervals for

\widehat{\beta}

use 1,000 question-level bootstrap resamples, retaining all responses of each sampled question.

Dataset	$\widehat{C}_{\mathrm{total}}$	$\widehat{C}_{\mathrm{proj}}$	$\widehat{C}_{\mathrm{res}}$
MATH-500	2.158 $[1.535,\,2.834]$	1.752 $[1.276,\,2.267]$	0.407 $[0.136,\,0.673]$
GPQA-Diamond	1.884 $[0.993,\,2.842]$	1.879 $[1.302,\,2.425]$	0.005 $[-0.591,\,0.627]$
MMLU-Pro STEM	2.909 $[2.349,\,3.491]$	2.471 $[2.017,\,2.927]$	0.438 $[0.169,\,0.708]$
AIME 2024	7.533 $[2.509,\,9.894]$	6.738 $[2.290,\,8.874]$	0.796 $[-1.364,\,2.678]$

Appendix

Table 6: Cross-half covariance decomposition. All values are multiplied by

10^{3}

; brackets give 95% bootstrap confidence intervals.

Component	High COS	Low COS	Difference	Share (%)
SAE features	0.07968	0.05503	$+0.02465$	47.6
Decoder bias	0.05289	0.05538	$-0.00249$	$-4.8$
Reconstruction residual	0.04675	0.01714	$+0.02962$	57.2
Total COS	0.17933	0.12755	$+0.05178$	100.0

Appendix

Table 7: Component contributions to the high–low COS difference on AIME24. Each group contains five questions. Values are rounded.

Feature ID	Observed text or symbol	Location
32531 , 47984		Formula-ending delimiter after the final answer
3447	So ; This ; exact expression is	Connective or result-statement expression
41030	58500 ; 2000 ; 4 inches per second	Numerical results and units
31368	find the sum ; how many ; ordered quadruple	Expression of the quantity or object to be found
605	The digit 7 in \boxed{27}	Numerical token within the final answer

Appendix

Table 8: Examples of high-activation locations for COS-related SAE features in MATH-500 responses.

	Reduction (%)			Time saved
Dataset	Generated tokens	Update response tokens	Training-step time	Wall-clock (min)	GPU-hours
MATH-500	17.07	17.14	5.43	14.23	0.949
MMLU-Pro STEM	19.56	18.93	6.68	18.02	1.202
GPQA-Diamond	2.88	3.24	0.66	1.43	0.095
AIME 2024	10.74	10.73	3.73	12.96	0.864
AIME 2025	26.12	25.93	10.86	36.66	2.444

Appendix

Table 9: Training-efficiency reductions and time savings across six benchmarks.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

Mar 20, 2026

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.

Dong Yan, Jian Liang, Yanbo Wang +3

School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University

RL for Language Model Reasoning Test-Time RL Pseudo-Labeling Learning with Noisy Labels

Ask PDF

Feb 6, 2026

TTSR: Test-Time Self-Evolving via Reflection

Test-time training (TTT) adapts large language models (LLMs) during inference using only unlabeled test inputs. Existing methods, however, face two major bottlenecks on hard reasoning tasks: (1) lack of learnable samples, as self-generated pseudo-labels on difficult questions are often noisy and yield unstable rewards; and (2) inefficient exploration, as performance gains depend on repeatedly sampling many rollouts without explicit diagnosis of why previous attempts fail. We propose TTSR (Test-Time Self-Reflection), a self-evolving framework based on a reflect-then-synthesize paradigm. A single pretrained model alternates between a Student role and a Teacher role: the Student solves test questions and updates, while the Teacher analyzes failed trajectories and synthesizes targeted variant questions closer to the Student's capability frontier. TTSR further maintains a cross-iteration weakness memory and compiles persistent weaknesses into a lightweight strategy note prepended to subsequent Student inputs, so diagnostic knowledge can guide exploration and gradually fade as weaknesses are resolved. Experiments on challenging mathematical reasoning benchmarks show consistent test-time improvements, strong cross-backbone generalization, and transfer to general-domain reasoning tasks.

Haoyang He, Zihua Rong, Yunjia Zhao +3

Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.

LLM Self-Refinement Test-Time Optimization Test-Time Training Teacher-Student Learning Language Model Self-Improvement

Ask PDF

Sep 16, 2026

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000x fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.

Naveen Vakada, Mingyuan Li, Shaoxiong Ji

University of Turku, Finland · ELLIS Institute Finland

Fine-Tuning Test-Time Adaptation of VLMs RL for Language Model Reasoning Test-Time RL

Ask PDF

Abstract

Test-Time RL

Explore similar work

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

TTSR: Test-Time Self-Evolving via Reflection

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces