GASP: Guided Asymmetric Self-Play For Coding LLMs

Swadesh Jana; Cansu Sancaktar; Tomáš Daniš; Georg Martius; Antonio Orvieto; Pavel Kolev

cs.LGMar 16, 2026

GASP: Guided Asymmetric Self-Play For Coding LLMs

Authors: Swadesh Jana, Cansu Sancaktar, Tomáš Daniš, Georg Martius, Antonio Orvieto, Pavel Kolev

Organizations: University of Tübingen · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center

Abstract

Asymmetric self-play has emerged as a promising paradigm for post-training large language models, where a teacher continually generates questions for a student to solve at the edge of the student's learnability. Although these methods promise open-ended data generation bootstrapped from no human data, they suffer from one major problem: not all problems that are hard to solve are interesting or informative to improve the overall capabilities of the model. Current asymmetric self-play methods are goal-agnostic with no real grounding. We propose Guided Asymmetric Self-Play (GASP), where grounding is provided by real-data goalpost questions that are identified to pose a hard exploration challenge to the model. During self-play, the teacher first generates an easier variant of a hard question, and then a harder variant of that easier question, with the goal of gradually closing the gap to the goalpost throughout training. Doing so, we improve pass@20 on LiveCodeBench (LCB) by 2.5% over unguided asymmetric self-play, and through the curriculum constructed by the teacher, we manage to solve hard goalpost questions that remain out of reach for all baselines.

Figures & tables

Figure 1: Overview of GASP . Self-play is guided by hard real-data coding questions that a standard RLVR run fails to solve. We refer to this subset as our goalpost questions. The teacher first generates an easy variant ( lemma ) of this goalpost and then a harder variant ( lift ), producing progressively more challenging questions that push the model’s knowledge boundary.

Panel of Figure 2 — Figure 2: Illustration of GASP . Iterative training on generated lemma and lift questions expands the student’s knowledge boundary, while the generated questions move closer to the goalpost $h$ .

Figure 3: Teacher rewards for lemma and lift proposals. We show the learnability curves according to Eq. 4 and Eq. 5 , where the vertical lines mark the corresponding valid bands.

Figure 4: Pass@k performance on the LCB eval benchmark (LCB v5 ). We compare GASP , GASP + Real-data RL, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we perform single-checkpoint model selection by choosing the checkpoint that maximizes pass@20 on the LCB v5 evaluation split, and report mean

\pm

std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting their checkpoint based on best performance on their evaluation benchmarks (including LCB) over training from a single-seed run. We additionally report pass@1-based checkpoint selection in the appendix (Figure 8 ), which shows the same qualitative trends.

Panel of Figure 5 — Figure 5: Goalpost questions solved with GASP and GASP + Real-data RL. Visualization over training checkpoints (x-axis: global step) and goalpost question IDs (y-axis). We include only goalpost questions that are solved at least once during training. A cell is marked as solved (green) if pass@ $100>0$ (i.e., at least one of 100 samples passes); otherwise it is unsolved (red). Results are aggregated across three seeds, where a cell is marked solved if any seed solves the question at that checkpoint (union across seeds).

Panel of Figure 6 — Figure 6: Goalpost $\rightarrow$ lemma $\rightarrow$ lift examples from GASP run. Functional-axis ( $f$ ) selected for difficulty adjustment on hard question with id abc351_d . (shared motif: connectivity/reachability)

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Pass@20 performance on the LCB v5 benchmark throughout RL training for GASP . Evaluation performance continues to improve throughout training and shows no plateau by the final iteration, where the runs were truncated due to compute constraints.

		Training on Real-data	Real-data Guidance	selected by pass@1		selected by pass@20
		Training on Real-data	Real-data Guidance	pass@1	pass@20	pass@1	pass@20
	GASP	✗	✓	19.10 ±0.92	33.63 ±0.35	18.26 ±0.68	33.69 ±0.28
$\mathopen{}\mathclose{{\left.\vbox{\vrule height=0.0pt,width=0.0pt}\textnormal{{Ablations}}}}\right\{$	GASP no i/o axis	✗	✓	19.36 ±1.35	33.31 ±1.34	18.94 ±1.69	33.47 ±1.14
	GASP w/o rejection sampling	✗	✓	17.97 ±0.79	32.72 ±1.81	17.64 ±1.33	33.03 ±1.50
	GASP one-step medium	✗	✓	17.10 ±0.33	31.41 ±0.03	15.73 ±1.61	32.36 ±0.03
	GASP one-step hard	✗	✓	17.76 ±1.14	32.20 ±0.91	17.13 ±0.68	33.37 ±0.12

Appendix

Table 2: Performance on the LCB v5 benchmark. We compare GASP pass@1 and pass@20, as well as ablations of our method, with standard RLVR training on all LCB questions in our training time split (Real-data RL), as well as performance of the AZR checkpoint open-sourced by Zhao et al. (2025) , and the base model Qwen2.5-Coder-7B . We show performance across 3 seeds of RL training respectively for two selection criteria: in each seed run we choose the model checkpoint with the best pass@1 or pass@20 performance. GASP outperforms AZR (unguided self-play) across all selection criteria. We use bold for global best , blue underline for best self-play and red for best with real-data training .

Figure 8: Pass@k performance on the LCB eval benchmark (LCB v5 ) with checkpoint selected for best pass@1. We compare GASP , GASP + Real-data RL, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@1 within that run, and report mean

\pm

std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint over training from a single-seed run.

Method	LCB ${}^{\text{v5}}$	HumanEval +	MBPP +
GASP	19.75 ±2.33	79.67 ±0.35	70.63 ±0.70
AZR	17.13 ${}_{\phantom{\pm 0.00}}$	83.54 ${}_{\phantom{\pm 0.00}}$	69.58 ${}_{\phantom{\pm 0.00}}$
Real-data RL	18.21 ±1.41	78.66 ±1.61	69.31 ±0.95
GASP + Real-data RL	20.52 ±3.60	80.49 ±1.61	71.60 ±1.00
Qwen2.5-Coder-7B	13.89 ${}_{\phantom{\pm 0.00}}$	79.88 ${}_{\phantom{\pm 0.00}}$	69.58 ${}_{\phantom{\pm 0.00}}$

Appendix

Table 3: Greedy pass@1 performance on LCB v5 , HumanEval + and MBPP + . Checkpoint selected by best LCB pass@20. Mean over seeds;

\pm

shows standard deviation. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint based on performance on their evaluation benchmarks, including LCB v1-5 , HumanEval + , and MBPP + over training from a single-seed run. Shaded area indicates our target distribution, i.e. LCB v5 . We use bold for global best , blue underline for best self-play and red for best with real-data training .

Figure 9: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP without rejection sampling. We compare GASP without rejection-sampling, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean

\pm

Figure 10: Dissimilarity of proposed lemma and lift questions with respect to their corresponding buffers for GASP with and without rejection sampling. We observe that GASP without rejection sampling suffers from diversity collapse as RL training progresses, with proposed questions becoming increasingly similar to those already in the buffer. Results shown for three seeds. (see Appendix. B for more on the buffer dissimilarity computation.)

Figure 11: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP without input-output (i/o) difficulty axis for the teacher proposals. We compare GASP without the i/o difficulty axis (only relying on functional difficulty axis) with Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean

\pm

Figure 12: Teacher rewards for one-step medium and one-step hard proposals.

Figure 13: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP with one-step instead of the two-step lemma and lift generation. We compare GASP with a one-step curriculum (intermediate difficulty: one-step medium, hard: one-step hard), Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean

\pm

	Method	Per-seed	Unique (union)
	GASP	6 / 3 / 3	11
$\mathopen{}\mathclose{{\left.\vbox{\vrule height=0.0pt,width=0.0pt}\textnormal{{Ablations}}}}\right\{$	GASP no i/o axis	2 / 2 / 2	4
	GASP w/o rejection sampling	5 / 8 / 5	14
	GASP one-step medium	6 / 4 / 4	11
	GASP one-step hard	0 / 1 / 1	2
	AZR	—	0 †

Appendix

Table 4: Unique goalpost questions solved across training checkpoints (out of 146 goalposts). A question is counted if it is solved at any checkpoint within a run (pass@100

>0

). Per-seed counts and union across three seeds are reported. Bold : global best.

\dagger

Zero by construction (goalposts filtered to be unsolvable by these methods).

Figure 14: Goalpost questions solved by GASP without rejection sampling. Visualization over training checkpoints (x-axis: global step) and goalpost question IDs (y-axis). We include only goalpost questions that are solved at least once during training. A cell is marked as solved (green) if the corresponding question is solved at that checkpoint; otherwise it is unsolved (red). Results are aggregated across three seeds, where a cell is marked solved if any seed solves the question at that checkpoint (union across seeds).

Panel of Figure 15 — Figure 15: Goalpost $\rightarrow$ lemma $\rightarrow$ lift examples from GASP run (with rejection sampling). Input–output-axis (io) selected for difficulty adjustment on hard questions with id abc364_e (top) and abc358_e (bottom). Shared motifs: for abc364_e maximizing count under cumulative resource constraints (1D $\rightarrow$ 2D feasibility); for abc358_e counting strings under per-letter usage caps (mod / DP).

Panel of Figure 16 — Figure 16: Goalpost $\rightarrow$ lemma $\rightarrow$ lift examples from GASP run (with rejection sampling). Function-axis (f) selected for difficulty adjustment on hard question with id 3414 . Shared motif: counting paths / number of ways in a stateful process with constrained moves; DP on position + auxiliary state.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Scaling Self-Play with Self-Guidance
Apr 22, 2026Luke Bailey, Kaiyue Wen, Kefan Dong +2Self-Play RL Automated Theorem Proving
Stanford University
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
May 29, 2026Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang +1Open-Ended Generation Self-Play RL
University of Edinburgh · Imperial College London
CompassPlay: Rewarding the Proposer for Where It Moves the Solver
Sep 26, 2026Sophia Xiao Pu, Ximeng Sun, Jiang Liu +4Self-Play RL Reward Design for RL
University of California, Santa Barbara · Advanced Micro Devices

Apr 22, 2026J/K move · Enter open · S save

Scaling Self-Play with Self-Guidance

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto+1

Stanford University

Self-Play RL Automated Theorem Proving LLM-Guided RL Language Model Self-Improvement

Open paper AskPDF arXiv

LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute, instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the language model itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4.

Abstract

Language Model Self-Improvement

Explore similar work

Scaling Self-Play with Self-Guidance

SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

CompassPlay: Rewarding the Proposer for Where It Moves the Solver