Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning

Aozhe Wang; Yuchen Yan; Nan Zhou; Zhengxi Lu; Weiming Lu; Jun Xiao; Yueting Zhuang; Yongliang Shen

cs.CLMar 16, 2026

Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning

Authors: Aozhe Wang, Yuchen Yan, Nan Zhou, Zhengxi Lu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

Organizations: Zhejiang University

Abstract

Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates. Yet high-quality test suites are scarce, existing datasets offer limited coverage, and static rewards fail to adapt as models improve. Recent self-play methods unify code and test generation in a single model, but face a inherent dilemma: white-box access leads to self-collusion where the model produces trivial tests for easy rewards, yet black-box restriction yields generic tests that miss implementation-specific bugs. We introduce Code-A1, an adversarial co-evolution framework that jointly optimizes a Code LLM and a Test LLM with opposing objectives. The Code LLM is rewarded for passing more tests, while the Test LLM is rewarded for exposing more defects. This architectural separation eliminates self-collusion risks and safely enables white-box test generation, where the Test LLM can inspect candidate code to craft targeted adversarial tests. We further introduce a Mistake Book mechanism for experience replay and a composite reward balancing test validity with adversarial difficulty. Experiments on Qwen2.5-Coder models demonstrate that Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests, while significantly improving test generation capability.

Figures & tables

Figure 1: Comparison of three training paradigms for code generation. Vanilla GRPO relies on static golden tests with rigid rewards. GRPO with Self-Play unifies code and test generation but must operate in black-box mode to prevent self-collusion. Code-A1 decouples the two tasks into models with opposing objectives, safely enabling white-box test generation and stable co-evolution via the Mistake Book.

Figure 2: Overview of the Code-A1 training framework. The Code LLM generates solutions accessible to the Test LLM for white-box testing. Generated tests are validated and merged with historical tests from the Mistake Book. The Code LLM is rewarded for passing more tests; the Test LLM is rewarded for exposing more defects, enabling curriculum-aware adversarial co-evolution.

Code LLM	Method	HumanEval +	MBPP +	BigCodeBench	Avg
Code LLM	Method	avg@32	avg@32	avg@32	Avg
Qwen2.5-Coder-1.5B-Instruct	/	63.42	60.87	29.34	51.21
	Golden Tests	71.15	63.30	34.23	56.23
	Self-Play	70.64	63.54	33.47	55.88
	Code-A1	72.69	63.33	34.82	56.95
Qwen2.5-Coder-3B-Instruct	/	77.63	63.12	41.78	60.84

Table 1: Performance comparison of Code LLMs. The best results are highlighted in bold, and the second best results are underlined.

Test LLM	Method	UnLeakedTestBench						Mul
Test LLM	Method	pass@1	pass@2	pass@5	mut@1	mut@2	mut@5	Mul
Qwen2.5-Coder-1.5B-Instruct	/	18.34	16.91	16.29	17.10	20.35	22.30	3.63
	SFT	16.01	15.16	14.76	14.53	27.57	29.45	4.35
	Self-Play	24.12	23.37	23.39	14.82	26.98	28.91	6.76
	Code-A1	27.42	25.68	27.05	17.31	23.57	26.41	7.14
Qwen2.5-Coder-3B-Instruct	/	19.77	20.50	20.93	28.35	29.99	42.55	8.91

Table 2: Performance comparison of Test LLMs. The best results are highlighted in bold, and the second best results are underlined.

Figure 3: Impact of trade-off weight

\alpha

on model capabilities using Qwen2.5-Coder-1.5B-Instruct. Left: Code generation performance (avg@32) on three benchmarks. Right: Test generation metrics on UnLeakedTestBench.

Settings	HumanEval +	MBPP +	BigCodeBench	Avg	UnLeakedTestBench		Mul
Settings	avg@32	avg@32	avg@32	Avg	pass@5	mut@5	Mul
Code-A1	72.69	63.33	34.82	56.95	27.05	26.41	7.14
- w/o predicted answer	70.69	64.47	34.60	56.59	11.18	21.72	2.43
- w/o referable code	68.66	62.99	33.34	55.00	23.77	23.52	5.59
- w/o Mistake Book	69.66	63.94	33.73	55.78	25.50	27.41	6.99

Table 3: Ablation study: removing referable code, predicted answer, or Mistake Book each degrades performance, confirming the contribution of white-box access, multi-objective optimization, and experience replay.

Panel of Figure 4 — Figure 4: Effect of Mistake Book. (a) Pass rate on newly generated tests, historical tests, and Golden Tests baseline over training. (b) Number of tests added to and removed from the Mistake Book per step.

Golden Test Source	HE +	MBPP +	BCB	Avg
Golden Test Source	avg@32	avg@32	avg@32	Avg
Human Annotation	71.15	63.30	34.23	56.23
base Test LLM	69.68	63.66	32.85	55.40
SFT Test LLM	71.49	64.24	32.93	56.22
Code-A1 Test LLM	71.67	64.42	34.17	56.75

Table 4: Performance comparison of Code LLMs on three benchmarks using Qwen2.5-Coder-1.5B-Instruct as the base model and employing Golden Tests from various sources for RLVR. The best results are highlighted in bold, and the second best results are underlined.

Figure 5: Training dynamics (Code Reward and Test Reward) over training steps for (a) Code LLM and (b) Test LLM.

Parallel TTS		HE+	MBPP +	BCB	Avg
Code LLM	Test LLM	avg@32	avg@32	avg@32	Avg
Base	/	77.63	63.12	41.78	60.84
Code-A1	/	83.52	69.07	45.85	66.15
Base	Base	82.32	70.63	42.45	65.13
Code-A1	Base	84.76	71.69	45.91	67.45
Base	Code-A1	82.93	70.90	43.63	65.82

Table 5: Performance of different combinations under parallel test-time scaling (see details in Appendix A.5 ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment Setup & Resources
Hyperparameter	Golden Tests (GT)	Self-Play (SP)	Code-A1
Test LLM Source	-	Self-Play (Shared)	Independent Model
Reward Manager	acg_gt	acg_llm	acg_llm
Reward Alpha ( $\alpha$ )	-	0.5	0.5
n_gpus_per_node	8	8	4
Trainer

Appendix

Table 6: Detailed hyperparameters for reinforcement learning experiments.

Model	Params	HumanEval +	MBPP +	BigCodeBench	Avg	UnLeakedTestBench		Mul
Model	Params	avg@32	avg@32	avg@32	Avg	pass@5	mut@5	Mul
ReasonFlux-Coder-14B	14B	78.73	72.93	52.50	68.05	32.18	53.27	17.14
Code-A1-7B	7B (2 $\times\!\downarrow$ )	85.21	74.50	52.46	70.72	37.15	53.14	19.74
ReasonFlux-Coder-7B	7B	76.62	69.88	46.49	64.33	23.65	46.28	10.95
Code-A1-3B	3B (2.3 $\times\!\downarrow$ )	83.52	69.07	45.85	66.15	30.86	49.56	15.29
ReasonFlux-Coder-4B †	4B	61.81	66.08	46.99	58.29	36.90	57.73	21.30

Appendix

Table 7: Comparison with ReasonFlux-Coder (CURE). Code-A1 achieves comparable or superior performance with significantly fewer parameters.

\dagger

: Long-CoT base model.

Figure 6: Detailed pipeline of Code-A1. Top: Code LLM training with code generation (Step 1), evaluation against combined tests (Step 4), and reward computation via group-relative advantage (Step 5). Bottom: Test LLM training with white-box test generation (Step 2-1), Mistake Book retrieval and update (Steps 2-2, 2-3, 4-2), test validation against ground truth (Step 3), and composite reward computation with TopVar selection (Step 5-1).

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 311 papers

Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs
Sep 3, 2026Jiacheng Xu, Wentao Zhang, Zhiyi Lyu +4Automated Test Generation LLM Code Generation
Nanyang Technological University, Singapore · Skywork AI
EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning
Mar 13, 2026Chi Ruan, Dongfu Jiang, Huaye Zeng +2RL for Code Generation Reinforcement Learning with Verifiable Rewards
University of Waterloo · Vector Institute · Harvard University
ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimization
Apr 17, 2026Yixu Huang, Xinglei Yu, Zhongyu WeiExecution-Guided Code Generation Language Model Self-Improvement
School of Data Science Fudan University Shanghai, China

Sep 3, 2026J/K move · Enter open · S save

Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs

Jiacheng Xu, Wentao Zhang, Zhiyi Lyu, Fuxiang Zhang+3

Nanyang Technological University, Singapore · Skywork AI

Automated Test Generation LLM Code Generation

Open paper AskPDF arXiv

Reinforcement learning (RL) has substantially advanced code generation with large language models (LLMs) through executable feedback. The feedback for coding problems mainly comes from specific test cases, where high-quality test cases are often scarce since they should be both sound and discriminative. We thus turn to study the auto-generation of test cases using the learned model. We find this is naturally an adversarial RL problem: the model is expected to generate effective test cases as counterexamples, depending on the solver's current failure modes. We propose Test Cases Scaling (TCS), a two-stage RL framework for effective test generation. Both stages train a test generator from a rolling policy-aligned buffer: Stage 1 generates tests consistent with the reference solution, and Stage 2 restricts the buffer to current failure modes and learns counterexample tests. Across TACO and LiveCodeBench, TCS improves both pass@1 and inference-time answer selection according to generated tests. We find the learned test generator also enables effective selection among other LLM outputs.

Abstract

LLM Code Generation

Explore similar work

Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs

EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning

ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimization