Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning

Zhanming Zhang; Vinoth Selvendran

cs.LGOct 1, 2026

Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning

Authors: Zhanming Zhang, Vinoth Selvendran

Organizations: Independent Researcher New York, NY · Independent Researcher Palo Alto, CA

Abstract

Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's \emph{entry state} consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose \textbf{entry-state sharpening}: use data-free training \emph{before} TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman $ρ=-0.90$ ; $ρ=-0.99$ after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at $3.39$ nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches $0.07$ nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a \emph{state-control problem}: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.

Figures & tables

Figure 1: Identical 15-step TTRL on unlabeled MATH problems for Base, R-Zero, and SPIRAL@400; means over seeds $\{2,3\}$ . R-Zero starts $4.5$ accuracy points ahead of Base and finishes $5.8$ points behind it, while SPIRAL@400 finishes highest despite never training on math.

Checkpoint	Objective	Entry	Entry	Entry	Final	Final	$\eta$
		entropy	mean@16	best@16	mean@16	best@16
R-Zero	uncert. curriculum	$3.391$	$0.717$	$0.799$	$0.713\pm 0.004$	$0.783\pm 0.007$	$0.911\pm 0.013$
Base	— (pretrained)	$1.186$	$0.672$	$0.749$	$0.771\pm 0.005$	$0.820\pm 0.006$	$0.941\pm 0.001$
SPIRAL@32	game play, Kuhn @32	$0.344$	$0.747$	$0.806$	$0.771\pm 0.015$	$0.823\pm 0.013$	$0.937\pm 0.003$
Self-distilled	data-free, high-dose	$0.183$	$0.710$	$0.771$	$0.767\pm 0.016$	$0.811\pm 0.018$	$0.945\pm 0.001$
SPIRAL@400	game play, multi-game	$0.073$	$0.755$	$0.796$	$\mathbf{0.787\pm 0.005}$	$0.822\pm 0.001$	$\mathbf{0.957\pm 0.006}$

Table 1: The five data-free checkpoints under the frozen 15-step TTRL protocol on MATH, ordered by entry entropy. Entry columns are seed means; final columns and

\eta

are means

\pm

sample s.d. over seeds

\{2,3\}

. Endpoint conversion efficiency

\eta

is strongly ordered by entry entropy.

Figure 2: Entry policy entropy versus endpoint conversion efficiency for the five data-free checkpoints. Filled markers denote seed means and hollow markers individual seeds $\{2,3\}$ .

Checkpoint	Objective	Entry	Entry	Entry	Final	Final	$\eta$
		entropy	mean@16	best@16	mean@16	best@16
R-Zero	uncert. curriculum	$4.644$	$0.305$	$0.731$	$0.317\pm 0.006$	$0.735\pm 0.024$	$0.432\pm 0.021$
Base	— (pretrained)	$1.791$	$0.239$	$0.736$	$0.357\pm 0.013$	$0.766\pm 0.017$	$0.466\pm 0.007$
SPIRAL@32	game play, Kuhn @32	$0.681$	$0.349$	$0.790$	$0.383\pm 0.004$	$0.783\pm 0.011$	$0.489\pm 0.012$
Self-distilled	data-free, high-dose	$0.454$	$0.272$	$0.758$	$0.369\pm 0.001$	$0.727\pm 0.005$	$0.508\pm 0.002$
SPIRAL@400	game play, multi-game	$0.130$	$0.383$	$0.780$	$\mathbf{0.406\pm 0.002}$	$0.755\pm 0.010$	$\mathbf{0.538\pm 0.005}$

Table 2: GPQA under the frozen 15-step TTRL protocol, matched seeds

\{2,3\}

, ordered by entry entropy. Entry columns are seed means; final columns and

\eta

are means

\pm

sample s.d.

Checkpoint	Objective	Entry	Entry	Entry	Final	Final	$\eta$
		entropy	mean@16	best@16	mean@16	best@16
R-Zero	uncert. curriculum	$3.452$	$0.417$	$0.710$	$0.414\pm 0.003$	$0.665\pm 0.024$	$0.623\pm 0.018$
Base	— (pretrained)	$1.041$	$0.393$	$0.739$	$0.460\pm 0.006$	$0.747\pm 0.001$	$0.615\pm 0.007$
SPIRAL@32	game play, Kuhn @32	$0.278$	$0.426$	$0.684$	$0.460\pm 0.013$	$0.725\pm 0.004$	$0.635\pm 0.021$
Self-distilled	data-free, high-dose	$0.221$	$0.403$	$0.741$	$\mathbf{0.479\pm 0.006}$	$0.700\pm 0.002$	$\mathbf{0.684\pm 0.011}$
SPIRAL@400	game play, multi-game	$0.076$	$0.436$	$0.744$	$0.476\pm 0.011$	$0.704\pm 0.009$	$0.676\pm 0.007$

Table 3: AMC under the frozen 15-step TTRL protocol, matched seeds

\{2,3\}

, ordered by entry entropy. Entry columns are seed means; final columns and

\eta

are means

\pm

sample s.d.

Explore similar work

Sep 16, 2026cs.LG

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000x fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.

Naveen Vakada, Mingyuan Li, Shaoxiong Ji

University of Turku, Finland · ELLIS Institute Finland

Stable Test-Time Adaptation Test-Time Adaptation Subspace Steering Vista

Ask PDF

May 27, 2026cs.LG

Label-Free Reinforcement Learning via Cross-Model Entropy

Post-training large language models with reinforcement learning is bottlenecked by the reward signal. Existing approaches require either ground-truth verifiable rewards, restricting training to domains with automatic correctness checks (e.g., mathematics, code execution), or human preference labels, which are expensive to collect and prone to reward hacking. Recent label-free methods replace ground-truth verifiers with self-referential signals like majority voting or token entropy over a model's own outputs, but risk reinforcing a model's own errors. In this work we propose Cross-Model Entropy (CME), the mean log-likelihood of a generator's response under a separate verifier model, as a label-free reward signal for RL post-training. CME is continuous, training-free, and grounded in the principle that responses a verifier finds unsurprising are likely correct or high quality. Because the verifier is independent of the generator, the signal cannot be gamed through self-consistency. We integrate CME into GRPO with no other changes to the training loop, extending label-free RL to open-ended instruction following -- a regime where self-referential signals are inapplicable or poorly suited. On open-ended instruction following (UltraFeedback prompts, evaluated on AlpacaEval 2.0), CME rewards beat the untrained base in head-to-head LLM-as-Judge comparisons across four model families (Qwen, Llama, Gemma, OLMo) and three training regimes (pretrained, SFT, and instruction-tuned), with tie-adjusted win rates ranging from 52.5% to 71.4%. Code will be released upon publication.

Matt Gorbett, Hossein Shirazi

Independent Researcher · San Diego State University

Reinforcement Learning Post-Training Cross-Entropy Losses Verifiable Rewards Open-Ended Responses

Ask PDF

Feb 6, 2026cs.CL

TTSR: Test-Time Self-Evolving via Reflection

Test-time training (TTT) adapts large language models (LLMs) during inference using only unlabeled test inputs. Existing methods, however, face two major bottlenecks on hard reasoning tasks: (1) \emph{lack of learnable samples}, as self-generated pseudo-labels on difficult questions are often noisy and yield unstable rewards; and (2) \emph{inefficient exploration}, as performance gains depend on repeatedly sampling many rollouts without explicit diagnosis of why previous attempts fail. We propose \textbf{TTSR} (\textbf{T}est-\textbf{T}ime \textbf{S}elf-\textbf{R}eflection), a self-evolving framework based on a \emph{reflect-then-synthesize} paradigm. A single pretrained model alternates between a \textit{Student} role and a \textit{Teacher} role: the Student solves test questions and updates, while the Teacher analyzes failed trajectories and synthesizes targeted variant questions closer to the Student's capability frontier. TTSR further maintains a cross-iteration \textit{weakness memory} and compiles persistent weaknesses into a lightweight \textit{strategy note} prepended to subsequent Student inputs, so diagnostic knowledge can guide exploration and gradually fade as weaknesses are resolved. Experiments on challenging mathematical reasoning benchmarks show consistent test-time improvements, strong cross-backbone generalization, and transfer to general-domain reasoning tasks.

Haoyang He, Zihua Rong, Yunjia Zhao +3

Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.

Test-Time Training Test Time Mathematical Reasoning Benchmarks Reasoning Skills Reflections

Ask PDF