Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

Kun Wang; Reinhard Heckel

cs.LGMar 13, 2026

Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

Authors: Kun Wang, Reinhard Heckel

Organizations: School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning

Abstract

Direct evaluation of LLMs on benchmarks can be misleading because comparatively strong performance may reflect task familiarity rather than capability. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised finetuning. However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as SFT-based train-before test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models which perform substantially better once aligned, yielding a more faithful evaluation of their capabilities. Moreover, for reasoning tasks, the performance gap between fine-tuned models and their base models largely disappears after alignment, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.

Figures & tables

Figure 1: Fine-tuning gains are largely diminished after task alignment. Left: The reported performance reveals substantial accuracy gaps between base models and their fine-tuned variants. Right: After applying TTRA to all models, the base model’s performance increases significantly, nearly matching that of the fine-tuned models, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.

Figure 2: Overview of the TTRA base train-before-test evaluation pipeline.

Figure 7: Performance comparison of Qwen2.5-Math base and fine-tuned models under TTRA evaluation. Results are shown for 7B models (top) and 1.5B models (bottom) across four benchmarks. The orange bars denote the base models, while blue bars denote various fine-tuned variants. The orange dashed line marks the base model’s performance level, highlighting that the performance advantage of fine-tuned models largely disappears after applying TTRA.

Figure 8: Evaluation of models aligned via TTRA compared to human doctors and a directly evaluated baseline. For models marked with (*), base model performance is taken from [ 13 ] . After applying TTRA, the models significantly outperform direct evaluation, indicating that the base models possess latent capabilities previously underestimated, and relative comparisons between models change.

Figure 9: Stage-wise ablation study of TTRA. Left: For Qwen2.5-Math-1.5B on the MATH500 benchmark, both 1-shot alignment and TTRL contribute positively, with the hybrid TTRA approach achieving the highest accuracy. Right: For Qwen2.5-1.5B on the MIMIC-CDM-FI benchmark, removing 1-shot alignment causes TTRL to fail, highlighting the critical role of initial alignment in unfamiliar domains.

Method	Time	Relative time
Direct Evaluation	19 min	1.0
Train-before-test (SFT)		32 $\times$
SFT	10 h
Inference	19 min
Train-before-test (TTRA)		6.6 $\times$
One-shot RL	15 min

Table 1: Evaluation time and relative computational cost of direct evaluation and train-before-test methods (SFT and TTRA) using Qwen3-0.6B on GSM8K.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: TTRA increases the ranking correlation between GSM8K and MathQA. Left: Direct evaluation shows a moderate correlation (

\rho=0.58

)1. Right: After TTRA alignment, the correlation significantly increases (

\rho=0.91

Figure 11: TTRA maintains a high correlation for already harmonized tasks. Left: Direct evaluation of the GSM8K and GSM-plus-mini benchmarks shows a high correlation. Right: TTRA alignment preserves this high correlation.

Figure 12: Sensitivity of SFT to training data selection. We evaluate the performance of the base model fine-tuned on various open-source datasets with a fixed data budget (1k samples). The results reveal significant performance variance across different data sources.

	Math500		OlympiadBench
Models	Direct	TTRA	Direct	TTRA
Llama-3.2-3B-NuminaQA
Base	$0.6^{*}$	$40.52$	$0.1^{*}$	9.59
Oat-Zero-3B [ 22 ]	$50.0^{*}$	$46.27$	$14.7^{*}$	12.53
	+49.4	+5.75	+14.6	+2.94

Appendix

Table 2: Comparison between direct evaluation and evaluation using train-before-test with TTRA. Scores marked with * are reported from the original papers. Under direct evaluation, fine-tuned variants show large improvements; after applying TTRA, these gains are substantially smaller.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

References Improve LLM Alignment in Non-Verifiable Domains
Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM Alignment LLM-as-a-Judge
Yale University · Meta · Scale AI +2
Compute Aligned Training: Optimizing for Test Time Inference
Apr 27, 2026Adam Ousherovitch, Ambuj TewariTest-Time Optimization Inference-Time Scaling
Department of Statistics University of Michigan
Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Mar 12, 2026Yixin Liu, Yue Yu, DiJia Su +7LLM-as-a-Judge LLM Alignment
Meta Superintelligence Labs · Yale Univeristy

Feb 18, 2026J/K move · Enter open · S save

References Improve LLM Alignment in Non-Verifiable Domains

Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri+2

Yale University · Meta · Scale AI+2

LLM Alignment LLM-as-a-Judge LLM Self-Refinement LLM Evaluation

Open paper AskPDF arXiv

While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.

Abstract

LLM Evaluation

Explore similar work

References Improve LLM Alignment in Non-Verifiable Domains

Compute Aligned Training: Optimizing for Test Time Inference

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training