Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation

Ana Nunez; Peyman Najafirad

cs.AISep 18, 2026

Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generation

Authors: Ana Nunez, Peyman Najafirad

Organizations: Secure AI and Autonomy Lab University of Texas at San Antonio

Abstract

Self-play methods that co-train a single language model as both coder and test author promise to move code-generation RL beyond fixed test suites, but they suffer from two coupled pathologies: permissiveness collapse, where pass-rate rewards are maximised by trivial, non-discriminative tests, and concentration bias, where i.i.d. sampled tests cluster on modal inputs and inflate estimator variance. We introduce CoVer (Co-trained Coder and Verifier), a single-policy GRPO framework that addresses both failure modes. First, an information-gain (IG) reward scores each self-generated test by the mutual information between its pass/fail vector and a graded, ground-truth-anchored correctness signal y [0, 1] m, gated by the sign of their covariance so that only positively discriminative tests receive reward. Second, a three-stage diversity-aware selection step prunes a candidate pool to a behaviourally non-redundant suite (invalidity, input-string, execution-profile filtering), raising the effective sample size of the IG estimator at fixed execution budget. On five benchmarks (LiveBench, MBPP, LiveCodeBench, CodeContests, Code-Forces), CoVer raises one-shot pass@1 by +5.8 points at 7B and +7.1 points at 14B over the Qwen2.5-Instruct backbone, and achieves the highest macro-average among all compared methods at both scales. As a drop-in backbone inside the CodeT ranking pipeline, CoVer-7B adds +3.5 points, demonstrating the dual benefit of co-training for both generation and selection.

Figures & tables

Method family	GT only at	Trains a	MI-based	Diversity	No GT at
Method family	training	verifier	reward	selection	evaluation
Curated-test RL ( Gehring et al., 2025 ; Dou et al., 2024 ; Zeng et al., 2025 )	✓	–	–	–	✓
Inference-time filtering ( Chen et al., 2023 ; Ridnik et al., 2024 )	–	–	–	–	✓
Self-play co-training ( Lin et al., 2025 ; Wang et al., 2025 )	–/✓	✓	–	–	✓
CoVer (ours)	✓	✓	✓	✓	✓

Table 1: Positioning CoVer against four threads of prior work. CoVer channels GT through a graded MI reward, prunes self-generated tests via diversity-aware selection for variance reduction, and evaluates in zero-shot mode with no GT signal.

Figure 1: Two failure modes of standard self-play code RL ( left ) and CoVer’s solutions ( right ). (1) Permissiveness collapse : trivial, non-discriminative tests maximise pass-rate and other binary-acceptance rewards while carrying zero information. (2) Concentration bias : i.i.d. sampling clusters tests on modal inputs, inflating estimator variance. (3) CoVer’s IG reward scores each test by

I(E_{:,j};\,y)

\mathrm{Cov}(E_{:,j},y)>0

, rewarding only positively discriminative tests. (4) Three-stage diversity-aware selection (invalidity, input-string, execution-profile) selects a behaviourally diverse suite at fixed execution budget.

Figure 2: Overview of the CoVer pipeline. Given a program specification , a single shared policy $\pi_{\theta}$ acts in two roles, coder and verifier , sampling a pool of candidate solutions $\{c_{i}\}$ and candidate tests $\{t_{j}\}$ . During trajectory execution , every solution is run against every test to populate an execution table $E$ ; in parallel, each solution is run against the ground-truth tests to compute a graded correctness signal $y$ . The CoVer verifier signal then (1) applies a three-stage diversity-aware selection to prune the pool to a behaviourally non-redundant suite, (2) keeps a verifier group $S_{T}$ of distinct tests, and (3) assigns each kept test a covariance-gated information-gain reward, so that only positively discriminative tests are rewarded. Finally, in the GRPO update , the coder reward and the verifier IG reward over $S_{T}$ update the shared policy, which feeds back into the next round.

Baselines ( $\sim$ 7–8B scale)
Model	LiveBench	MBPP	LCB	CC	CF	Avg.
RLEF-8B (Llama 3.1 8B Inst) ( Gehring et al., 2025 )	–	–	–	9.7	–	–
Sol-Ver (Llama 3.1 8B) ( Lin et al., 2025 )	–	41.0	27.24	–	–	–
ReasonFlux-Coder-7B ( Wang et al., 2025 )	37.1	70.2	31.2	25.9	8.2	34.52
Qwen2.5-7B-Coder-Instruct	35.0	68.0	29.8	22.8	6.7	32.46
Qwen2.5-7B-Instruct	31.1	66.3	26.9	21.2	5.4	30.18

Table 2: CoVer achieves the highest macro-average one-shot pass@1 (%) at both 7B and 14B scales. We compare against four families of baselines: general-purpose backbones (Qwen2.5-Instruct), code-pretrained LLMs (Qwen2.5-Coder-Instruct), Curated-test RL (RLEF), and RL self-play methods for code (Sol-Ver, ReasonFlux-Coder). LCB = LiveCodeBench, CC = CodeContests, CF = CodeForces. Backbone is noted in parentheses where it differs from Qwen2.5.

Figure 3: Code accuracy (pass@1, %) versus training step for CoVer-7B and CoVer-14B. Both scales improve steadily over training, with the largest gains in the first $150$ steps.

Variant (Qwen2.5-7B-Instruct)	LiveBench	MBPP	LCB	CC	CF	Avg.
CoVer-7B (diversity selection, $k\!=\!16$ )	40.0	72.4	32.8	26.0	8.5	35.94
No pruning (all $K\!=\!32$ columns)	37.64	70.0	30.68	24.81	8.2	34.27
$\Delta$	+2.36	+2.4	+2.12	+1.19	+0.3	+1.67

Table 3: Three-stage diversity-aware selection ablation at matched sampling budget (

K\!=\!32

m\!=\!16

). Both variants execute

m\!\times\!K

code–test pairs; CoVer prunes to a diverse

k\!=\!16

columns for the IG reward. The consistent gain confirms that removing redundant columns recovers verifier signal lost to correlated test outcomes.

Variant	LB	MBPP	LCB	CC	CF	Avg.
CoVer-7B	40.0	72.4	32.8	26.0	8.5	35.94
A1: pass-rate rew.	36.32	69.25	31.25	24.94	7.8	33.91
A2: binary $y$	37.98	70.5	31.73	25.52	8.0	34.75
A3: coder only	38.03	67.7	30.19	23.92	7.7	33.51

Table 4: Reward-design ablation on Qwen2.5-7B-Instruct. A1: pass-rate reward replaces IG; A2: binary

y

replaces graded

y

; A3: no verifier role. Each ablation removes one design choice; all three degrade performance. All training runs are conducted with 350 optimization steps.

Method	LiveBench	MBPP	LCB	CC	CF	Avg.
Qwen2.5-7B-Instruct (one-shot)	31.1	66.3	26.9	21.2	5.4	30.18
CodeT ( $n\!=\!8$ )	44.53	75.95	37.76	28.03	8.56	38.97
CodeT + CoVer-7B ( $n\!=\!8$ )	50.78	79.63	40.70	29.70	11.56	42.47
$\Delta$ vs CodeT	+6.25	+3.68	+2.94	+1.67	+3.00	+3.50

Table 5: Test-time scaling with CodeT ( Chen et al., 2023 ) (

n\!=\!8

candidates). Replacing the backbone with CoVer-7B raises the macro-average by

+3.50

points, confirming that co-training produces both a stronger coder and a more discriminative verifier.

Comparison	LiveBench	MBPP	LCB	CC	CF
Comparison	$\Delta$ (std)	$\Delta$ (std)	$\Delta$ (std)	$\Delta$ (std)	$\Delta$ (std)
CoVer-7B vs Qwen2.5-7B-Inst	+8.9 (0.36)	+6.1 (0.47)	+5.9 (0.31)	+4.8 (0.33)	+3.1 (0.28)
CoVer-14B vs Qwen2.5-14B-Inst	+11.4 (0.14)	+3.1 (0.32)	+8.0 (0.27)	+7.3 (0.37)	+5.6 (0.20)
CoVer-7B vs Coder-7B	+5.0 (0.36)	+4.4 (0.47)	+3.0 (0.31)	+3.2 (0.33)	+1.8 (0.28)
CoVer-14B vs Coder-14B	+5.6 (0.14)	+6.8 (0.32)	+3.3 (0.27)	+9.6 (0.37)	+5.1 (0.20)

Table 6: Per-benchmark pass@1 improvement (

\Delta

) with cross-seed standard deviation (in parentheses) over

4

seeds. CoVer shows consistent improvements across all benchmarks, with gains exceeding the cross-seed variance on every comparison.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Train	Eval
CodeContests (diff. $\leq\!2$ )	4,500	239
LiveBench	–	128
MBPP	–	221
LiveCodeBench (v2)	–	511
CodeForces	–	467
Total	4,500	1,527

Appendix

Table 7: Training and evaluation task counts. RL training uses only the CodeContests training partition; the other four benchmarks contribute no training data and are used solely for evaluation.

Parameter	Value
Backbones	Qwen2.5-7B/14B-Instruct
	Llama-3.1-8B-Instruct
Candidate codes ( $m$ )	16
Sampled tests ( $K$ )	32
Kept tests ( $k$ )	16
Learning rate	$1\times 10^{-6}$

Appendix

Table 8: CoVer hyperparameter configuration.

Tests passed (of $k\!=\!16$ )	GT-correct
0–3	4.2%
4–7	17.4%
8–11	37.1%
12–15	53.7%
16 (all)	86.8%

Appendix

Table 9: Fraction of candidate solutions correct under the GT tests, bucketed by the number of CoVer-generated tests passed (held-out problems).

Model	LiveBench	MBPP	LCB	CC	CF	Avg.
Llama-3.1-8B-Instruct	17.38	49.97	14.56	12.47	2.50	19.38
CoVer-Llama-8B (ours)	31.34	59.04	26.29	20.10	4.80	28.31
$\Delta$	+13.96	+9.07	+11.73	+7.63	+2.30	+8.94

Appendix

Table 10: Cross-family generalization. One-shot pass@1 (%) for CoVer trained on Llama-3.1-8B-Instruct under the same 350-step protocol and hyperparameters used for the Qwen2.5 runs.

CoVer-7B
Run	LiveBench	MBPP	LCB	CC	CF	Avg.
Run 1	40.00	72.40	32.80	26.00	8.50	35.94
Run 2	39.06	70.55	32.61	25.73	8.70	35.33
CoVer-14B
Run 1	47.75	79.44	41.49	32.92	12.88	42.90
Run 2	48.29	78.73	42.31	33.79	13.14	43.25

Appendix

Table 11: Two fully independent training runs at each scale under an identical protocol. Macro-averages differ by

0.61

points at 7B and

0.35

at 14B, well below the

+5.76

and

+7.08

gains over the respective Qwen2.5-Instruct backbones.

RL for Code Generation

Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.

See all 150 papers

CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test
May 22, 2026Zhangyi Hu, Chenhui Liu, Tian Huang +6LLM Self-Refinement Execution-Guided Code Generation
1The Hong Kong University of Science and Technology (Guangzhou) · Institute of Deep Perception Technology, JITRI, Wuxi, China
Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning
Mar 16, 2026Aozhe Wang, Yuchen Yan, Nan Zhou +5LLM Code Generation RL for Code Generation
Zhejiang University
Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift
Oct 19, 2026Amarsaikhan Batjargal, Noriko TomuroRL for Code Generation Reinforcement Learning
Jarvis College of Computing and Digital Media, DePaul University, Chicago, IL, USA

May 22, 2026J/K move · Enter open · S save

CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test

Zhangyi Hu, Chenhui Liu, Tian Huang, Jindong Li+5

1The Hong Kong University of Science and Technology (Guangzhou) · Institute of Deep Perception Technology, JITRI, Wuxi, China

LLM Self-Refinement Execution-Guided Code Generation Code Generation Test-Time Scaling Automated Test Generation LLM Code Generation

Open paper AskPDF arXiv

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) and Test-Time Scaling (TTS) have advanced LLM code generation through executable verification. Yet Ground-Truth Unit Tests (GT UTs) remain a bottleneck: SOTA RLVR methods require them for costly training, while existing TTS methods lose competitiveness without them. This motivates GT-free TTS, where existing methods directly use self-generated UTs to refine and select code candidates. Yet such UTs are often noisy or spuriously coupled with wrong code, and UT quality in turn cannot be validated without reliable code. The key challenge is therefore to jointly improve both. To this end, we present CoSPlay, a GT-free, training-free framework that jointly improves codes and UTs through cooperative self-play. It first explores diverse solution ideas and identifies their potential failure modes to produce discriminative UT ideas. It then uses bidirectional pass-count signals from the Code-UT execution matrix to iteratively prune or fix weak codes and refresh or replace unreliable UTs, letting the two pools co-evolve. Finally, when multiple codes remain tied at the highest pass count, it picks the final code from the largest output-consensus cluster, since correct codes agree on the same inputs while wrong codes diverge. Experiments on four challenging benchmarks show that CoSPlay on Qwen2.5-7B-Instruct improves average BoN from 22.1% to 33.2% and UT accuracy from 14.6% to 78.3%, matching or surpassing the RLVR model CURE-7B. When applied to CURE-7B, it further improves BoN by 5.7%. CoSPlay also generalizes across diverse backbones and outperforms GT-free TTS baselines under comparable token budgets, with continued gains as the budget scales up. These results suggest a scalable inference strategy for competitive code generation without any GT data.

Abstract

RL for Code Generation

Explore similar work

CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test

Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning

Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift