Direct evaluation of LLMs on benchmarks can be misleading because comparatively strong performance may reflect task familiarity rather than capability. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised finetuning. However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as SFT-based train-before test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models which perform substantially better once aligned, yielding a more faithful evaluation of their capabilities. Moreover, for reasoning tasks, the performance gap between fine-tuned models and their base models largely disappears after alignment, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.
Figures & tables
Figure 1: Fine-tuning gains are largely diminished after task alignment. Left: The reported performance reveals substantial accuracy gaps between base models and their fine-tuned variants. Right: After applying TTRA to all models, the base model’s performance increases significantly, nearly matching that of the fine-tuned models, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.
Figure 2: Overview of the TTRA base train-before-test evaluation pipeline.
Figure 3Figure 4
Figure 7: Performance comparison of Qwen2.5-Math base and fine-tuned models under TTRA evaluation. Results are shown for 7B models (top) and 1.5B models (bottom) across four benchmarks. The orange bars denote the base models, while blue bars denote various fine-tuned variants. The orange dashed line marks the base model’s performance level, highlighting that the performance advantage of fine-tuned models largely disappears after applying TTRA.
Figure 8: Evaluation of models aligned via TTRA compared to human doctors and a directly evaluated baseline. For models marked with (*), base model performance is taken from [ 13 ] . After applying TTRA, the models significantly outperform direct evaluation, indicating that the base models possess latent capabilities previously underestimated, and relative comparisons between models change.
Figure 9: Stage-wise ablation study of TTRA. Left: For Qwen2.5-Math-1.5B on the MATH500 benchmark, both 1-shot alignment and TTRL contribute positively, with the hybrid TTRA approach achieving the highest accuracy. Right: For Qwen2.5-1.5B on the MIMIC-CDM-FI benchmark, removing 1-shot alignment causes TTRL to fail, highlighting the critical role of initial alignment in unfamiliar domains.
Method
Time
Relative time
Direct Evaluation
19 min
1.0
Train-before-test (SFT)
32 ×
SFT
10 h
Inference
19 min
Train-before-test (TTRA)
6.6 ×
One-shot RL
15 min
Table 1: Evaluation time and relative computational cost of direct evaluation and train-before-test methods (SFT and TTRA) using Qwen3-0.6B on GSM8K.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: TTRA increases the ranking correlation between GSM8K and MathQA. Left: Direct evaluation shows a moderate correlation ( ρ=0.58 )1. Right: After TTRA alignment, the correlation significantly increases ( ρ=0.91 ).
Figure 11: TTRA maintains a high correlation for already harmonized tasks. Left: Direct evaluation of the GSM8K and GSM-plus-mini benchmarks shows a high correlation. Right: TTRA alignment preserves this high correlation.
Figure 12: Sensitivity of SFT to training data selection. We evaluate the performance of the base model fine-tuned on various open-source datasets with a fixed data budget (1k samples). The results reveal significant performance variance across different data sources.
Math500
OlympiadBench
Models
Direct
TTRA
Direct
TTRA
Llama-3.2-3B-NuminaQA
Base
0.6∗
40.52
0.1∗
9.59
Oat-Zero-3B [ 22 ]
50.0∗
46.27
14.7∗
12.53
+49.4
+5.75
+14.6
+2.94
Appendix
Table 2: Comparison between direct evaluation and evaluation using train-before-test with TTRA. Scores marked with * are reported from the original papers. Under direct evaluation, fine-tuned variants show large improvements; after applying TTRA, these gains are substantially smaller.
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Scaling test-time compute has emerged as a powerful mechanism for enhancing Large Language Model (LLM) performance. However, standard post-training paradigms, Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), optimize the likelihood of individual samples under a base policy, creating a misalignment with test time procedures that rely on aggregated or filtered outputs. In this work, we propose Compute Aligned Training, which aligns training objectives with test-time strategies. By conceptualizing inference strategies as operators on the base policy, we derive new loss functions that maximize performance when said strategies are applied. We instantiate such loss functions for SFT and RL across common test time strategies. Finally, we provide empirical evidence that this training method substantially improves test time scaling over standard training.
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.