In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. With sufficiently capable judges and enough training, RAL matches direct RLVR. We then apply RAL to scientific abstract generation in two settings. In the first setting, RAL guides the generator to produce abstracts in the scientific domain of the reference abstracts, even though the prompt leaves that domain unspecified. In the second setting, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers.
Figures & tables
Figure 1: Reinforced Adversarial Learning (RAL). A co-trained judge learns to tell reference answers from generated ones, and its reference probability is the generator’s only reward. With no verifiers or reasoning demonstrations, reasoning z emerges from answer-level feedback. Bottom: example tasks, with illustrative reasoning.
Figure 2: RAL with 1.7B and 4B judges matches verifiable-reward training on integer multiplication. Generator accuracy, judge accuracy, relative error and judge-score correlation with relative error. Diamonds denote held-out test performance, including SFT. Y-axes are truncated.
Figure 3: Judge capacity and co-training time drive performance on Countdown-Code. Generator verifiable reward, judge accuracy, generator relative error and correlation between judge score and relative error. Diamonds denote held-out test performance at each run’s cutoff, including SFT ( ∼45% ). The 32B judge surpasses both verifiable-reward baselines after 4,000 training steps using reference answers alone. Y-axes are truncated for readability.
Figure 4: RAL with a 32B judge outperforms binary verifiable-reward training on SMILES-to-formula. Generator verifiable reward, judge accuracy, generated formula L1 distance to target and judge-score correlation with formula L1 distance. Diamonds show held-out test performance at each run’s cutoff. Y-axes are truncated.
Figure 5: Co-training recovers the hidden biomedical domain without collapsing diversity. A frozen Haiku judge yields increasing reward but collapses generation onto a narrow, out-of-domain template. RAL uses a co-trained Qwen3-14B judge. Biomedical rate is assessed by an LLM classifier; diversity is mean pairwise cosine distance between text embeddings.
Method
Accuracy
Verifiable reward
0.938
Judge + verifiable reward
0.925
SFT
0.005
RAL (0.6B judge)
0.861
RAL (4B judge)
0.949
Table 1: RAL and baseline performance across tasks.
Figure 6: RAL on NIH grant abstracts. Top left: fooling rates against human experts for our model, the base model and frontier baselines. Error bars show ±1 SE, and the dashed line marks chance. Top middle: fooling rate against LLM judges over training. The curve shows the mean across five judges, with shading indicating their min–max range, and horizontal lines mark baseline rates. Top right: co-trained judge accuracy on current and replayed generations. Bottom: verbatim excerpts for one held-out title at steps 0 and 2000, with selected strategies highlighted in the answers and reasoning traces.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training examples
20,000
Evaluation examples
800 , evaluated every 250 steps
Generator model
Qwen/Qwen3-0.6B
Judge models
Qwen/Qwen3-0.6B , 1.7B , and 4B
Reward variants
judge-only, exact-only, judge × exact; exact-only and judge × exact use the 0.6B judge setting
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.
Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic +1
University College London London, UK · University of Basel Basel, Switzerland