Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

Albert Bou; Javier Valdes-Aleman; Jon M. Laurent; Samuel G. Rodriques; Andrew D. White; Michaela Hinks; Siddharth Narayanan

cs.AIOct 28, 2026

Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

Authors: Albert Bou, Javier Valdes-Aleman, Jon M. Laurent, Samuel G. Rodriques, Andrew D. White, Michaela Hinks, Siddharth Narayanan

Organizations: Edison Scientific Inc. San Francisco, CA, USA

Abstract

In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. With sufficiently capable judges and enough training, RAL matches direct RLVR. We then apply RAL to scientific abstract generation in two settings. In the first setting, RAL guides the generator to produce abstracts in the scientific domain of the reference abstracts, even though the prompt leaves that domain unspecified. In the second setting, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers.

Figures & tables

Figure 1: Reinforced Adversarial Learning (RAL). A co-trained judge learns to tell reference answers from generated ones, and its reference probability is the generator’s only reward. With no verifiers or reasoning demonstrations, reasoning $z$ emerges from answer-level feedback. Bottom: example tasks, with illustrative reasoning.

Figure 2: RAL with 1.7B and 4B judges matches verifiable-reward training on integer multiplication. Generator accuracy, judge accuracy, relative error and judge-score correlation with relative error. Diamonds denote held-out test performance, including SFT. Y-axes are truncated.

Figure 3: Judge capacity and co-training time drive performance on Countdown-Code. Generator verifiable reward, judge accuracy, generator relative error and correlation between judge score and relative error. Diamonds denote held-out test performance at each run’s cutoff, including SFT ( $\sim 45\%$ ). The 32B judge surpasses both verifiable-reward baselines after 4,000 training steps using reference answers alone. Y-axes are truncated for readability.

Figure 4: RAL with a 32B judge outperforms binary verifiable-reward training on SMILES-to-formula. Generator verifiable reward, judge accuracy, generated formula L1 distance to target and judge-score correlation with formula L1 distance. Diamonds show held-out test performance at each run’s cutoff. Y-axes are truncated.

Figure 5: Co-training recovers the hidden biomedical domain without collapsing diversity. A frozen Haiku judge yields increasing reward but collapses generation onto a narrow, out-of-domain template. RAL uses a co-trained Qwen3-14B judge. Biomedical rate is assessed by an LLM classifier; diversity is mean pairwise cosine distance between text embeddings.

Method	Accuracy
Verifiable reward	0.938
Judge + verifiable reward	0.925
SFT	0.005
RAL (0.6B judge)	0.861
RAL (4B judge)	0.949

Table 1: RAL and baseline performance across tasks.

Figure 6: RAL on NIH grant abstracts. Top left: fooling rates against human experts for our model, the base model and frontier baselines. Error bars show ±1 SE, and the dashed line marks chance. Top middle: fooling rate against LLM judges over training. The curve shows the mean across five judges, with shading indicating their min–max range, and horizontal lines mark baseline rates. Top right: co-trained judge accuracy on current and replayed generations. Bottom: verbatim excerpts for one held-out title at steps 0 and 2000, with selected strategies highlighted in the answers and reasoning traces.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Training examples	$20{,}000$
Evaluation examples	$800$ , evaluated every $250$ steps
Generator model	Qwen/Qwen3-0.6B
Judge models	Qwen/Qwen3-0.6B , 1.7B , and 4B
Reward variants	judge-only, exact-only, judge $\times$ exact; exact-only and judge $\times$ exact use the 0.6B judge setting
Training horizon	$3{,}000$ steps

Appendix

Table 2: RAL digit-multiplication adversarial-training setup.

Setting	Value
Model	Qwen/Qwen3-0.6B (full finetuning)
Training examples	$20{,}000$
Held-out evaluation	$800$ examples
Effective batch	$\approx 3{,}200$ examples ( $131$ K tokens)
Batch composition	$64$ packed sequences of length $2048$
Optimization	AdamW, learning rate $2\times 10^{-5}$

Appendix

Table 3: Digit-multiplication SFT baseline setup.

Setting	Value
Training examples	$15{,}000$
Evaluation examples	$800$
Generator model	Qwen/Qwen3-1.7B
Judge models	Qwen/Qwen3-1.7B , 4B , 8B , 14B , and 32B
Reward variants	judge-only, exact-only, judge $\times$ exact
Training horizon	$10{,}000$ steps

Appendix

Table 4: Countdown-Code adversarial-training setup.

Setting	Value
Model	Qwen/Qwen3-1.7B (full finetuning)
Training examples	$15{,}000$
Held-out evaluation	$800$ examples
Effective batch	$\approx 730$ examples ( $524$ K tokens)
Batch composition	$128$ packed sequences of length $4096$
Optimization	AdamW, learning rate $3\times 10^{-5}$

Appendix

Table 5: Countdown SFT baseline setup.

Setting	Value
Dataset	Ether0 simple-formula SMILES-to-formula examples
Training examples	$17{,}938$
Evaluation examples	$800$ held-out examples
Generator model	Qwen/Qwen3-8B
Judge models	Qwen/Qwen3-8B , 14B , and 32B
Reward variants	judge-only, exact-only, judge $\times$ exact

Appendix

Table 6: SMILES-to-formula adversarial-training setup.

Setting	Value
Model	Qwen/Qwen3-8B (full finetuning)
Training examples	$17{,}938$
Held-out evaluation	$800$ examples
Effective batch	$\approx 2{,}250$ examples ( $131$ K tokens)
Batch composition	$64$ packed sequences of length $2048$
Optimization	AdamW, learning rate $10^{-5}$

Appendix

Table 7: SMILES-to-formula SFT baseline setup.

Setting	Value
Dataset	Medical-domain paper abstracts
Training examples	$5{,}000$
Generator model	Qwen/Qwen3-8B
Judge models	claude-haiku-4-5 pairwise judge and Qwen/Qwen3-14B learned judge
Reward variants	Pairwise Haiku judge reward and learned single-candidate judge reward
Training horizon	$2{,}000$ steps

Appendix

Table 8: Medical/biomedical paper-abstract adversarial-training setup.

Setting	Value
Dataset	NIH RePORTER FY2026 grant proposal abstracts, cleanv4 moderate split
Training examples	$20{,}000$ split, with $19{,}995$ configured for training
Evaluation examples	$128$ held-out grant titles and abstracts
Generator model	Qwen/Qwen3-32B
Judge model	Qwen/Qwen3-235B-A22B
Reward variant	Learned single-candidate judge reward

Appendix

Table 9: NIH grant-proposal abstract adversarial-training setup.

Setting	Value
Model	Qwen/Qwen3-32B (full finetuning)
Training examples	$19{,}995$ ( $5$ duplicates of held-out grants removed)
Held-out evaluation	$128$ examples
Effective batch	$\approx 255$ examples ( $147$ K tokens)
Batch composition	$72$ packed sequences of length $2048$
Optimization	AdamW, learning rate $4.5\times 10^{-5}$

Appendix

Table 10: NIH grant-abstract SFT baseline setup.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Mar 12, 2026Yixin Liu, Yue Yu, DiJia Su +7LLM-as-a-Judge LLM Alignment
Meta Superintelligence Labs · Yale Univeristy
References Improve LLM Alignment in Non-Verifiable Domains
Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM Alignment LLM-as-a-Judge
Yale University · Meta · Scale AI +2
Frontier Learning: Training LLM Reasoners at the Edge of Capability
Sep 28, 2026Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic +1RL for Language Model Reasoning LLM Post-Training
University College London London, UK · University of Basel Basel, Switzerland

Mar 12, 2026J/K move · Enter open · S save

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

Yixin Liu, Yue Yu, DiJia Su, Sid Wang+6

Meta Superintelligence Labs · Yale Univeristy

LLM-as-a-Judge LLM Alignment LLM-Guided RL Reward Hacking

Open paper AskPDF arXiv

Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.

Abstract

RL for Language Model Reasoning

Explore similar work

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

References Improve LLM Alignment in Non-Verifiable Domains

Frontier Learning: Training LLM Reasoners at the Edge of Capability