Asymmetric self-play has emerged as a promising paradigm for post-training large language models, where a teacher continually generates questions for a student to solve at the edge of the student's learnability. Although these methods promise open-ended data generation bootstrapped from no human data, they suffer from one major problem: not all problems that are hard to solve are interesting or informative to improve the overall capabilities of the model. Current asymmetric self-play methods are goal-agnostic with no real grounding. We propose Guided Asymmetric Self-Play (GASP), where grounding is provided by real-data goalpost questions that are identified to pose a hard exploration challenge to the model. During self-play, the teacher first generates an easier variant of a hard question, and then a harder variant of that easier question, with the goal of gradually closing the gap to the goalpost throughout training. Doing so, we improve pass@20 on LiveCodeBench (LCB) by 2.5% over unguided asymmetric self-play, and through the curriculum constructed by the teacher, we manage to solve hard goalpost questions that remain out of reach for all baselines.
Figures & tables
Figure 1: Overview of GASP . Self-play is guided by hard real-data coding questions that a standard RLVR run fails to solve. We refer to this subset as our goalpost questions. The teacher first generates an easy variant ( lemma ) of this goalpost and then a harder variant ( lift ), producing progressively more challenging questions that push the model’s knowledge boundary.
Figure 2: Illustration of GASP . Iterative training on generated lemma and lift questions expands the student’s knowledge boundary, while the generated questions move closer to the goalpost h .
Figure 3: Teacher rewards for lemma and lift proposals. We show the learnability curves according to Eq. 4 and Eq. 5 , where the vertical lines mark the corresponding valid bands.
Figure 4: Pass@k performance on the LCB eval benchmark (LCB v5 ). We compare GASP , GASP + Real-data RL, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we perform single-checkpoint model selection by choosing the checkpoint that maximizes pass@20 on the LCB v5 evaluation split, and report mean ± std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting their checkpoint based on best performance on their evaluation benchmarks (including LCB) over training from a single-seed run. We additionally report pass@1-based checkpoint selection in the appendix (Figure 8 ), which shows the same qualitative trends.
Figure 5: Goalpost questions solved with GASP and GASP + Real-data RL. Visualization over training checkpoints (x-axis: global step) and goalpost question IDs (y-axis). We include only goalpost questions that are solved at least once during training. A cell is marked as solved (green) if pass@ 100>0 (i.e., at least one of 100 samples passes); otherwise it is unsolved (red). Results are aggregated across three seeds, where a cell is marked solved if any seed solves the question at that checkpoint (union across seeds).
Figure 6: Goalpost → lemma → lift examples from GASP run. Functional-axis ( f ) selected for difficulty adjustment on hard question with id abc351_d . (shared motif: connectivity/reachability)
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Pass@20 performance on the LCB v5 benchmark throughout RL training for GASP . Evaluation performance continues to improve throughout training and shows no plateau by the final iteration, where the runs were truncated due to compute constraints.
Table 2: Performance on the LCB v5 benchmark. We compare GASP pass@1 and pass@20, as well as ablations of our method, with standard RLVR training on all LCB questions in our training time split (Real-data RL), as well as performance of the AZR checkpoint open-sourced by Zhao et al. (2025) , and the base model Qwen2.5-Coder-7B . We show performance across 3 seeds of RL training respectively for two selection criteria: in each seed run we choose the model checkpoint with the best pass@1 or pass@20 performance. GASP outperforms AZR (unguided self-play) across all selection criteria. We use bold for global best , blue underline for best self-play and red for best with real-data training .
Figure 8: Pass@k performance on the LCB eval benchmark (LCB v5 ) with checkpoint selected for best pass@1. We compare GASP , GASP + Real-data RL, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@1 within that run, and report mean ± std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint over training from a single-seed run.
Method
LCB v5
HumanEval +
MBPP +
GASP
19.75 ±2.33
79.67 ±0.35
70.63 ±0.70
AZR
17.13 ±0.00
83.54 ±0.00
69.58 ±0.00
Real-data RL
18.21 ±1.41
78.66 ±1.61
69.31 ±0.95
GASP + Real-data RL
20.52 ±3.60
80.49 ±1.61
71.60 ±1.00
Qwen2.5-Coder-7B
13.89 ±0.00
79.88 ±0.00
69.58 ±0.00
Appendix
Table 3: Greedy pass@1 performance on LCB v5 , HumanEval + and MBPP + . Checkpoint selected by best LCB pass@20. Mean over seeds; ± shows standard deviation. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint based on performance on their evaluation benchmarks, including LCB v1-5 , HumanEval + , and MBPP + over training from a single-seed run. Shaded area indicates our target distribution, i.e. LCB v5 . We use bold for global best , blue underline for best self-play and red for best with real-data training .
Figure 9: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP without rejection sampling. We compare GASP without rejection-sampling, Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean ± std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint over training from a single-seed run.
Figure 10: Dissimilarity of proposed lemma and lift questions with respect to their corresponding buffers for GASP with and without rejection sampling. We observe that GASP without rejection sampling suffers from diversity collapse as RL training progresses, with proposed questions becoming increasingly similar to those already in the buffer. Results shown for three seeds. (see Appendix. B for more on the buffer dissimilarity computation.)
Figure 11: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP without input-output (i/o) difficulty axis for the teacher proposals. We compare GASP without the i/o difficulty axis (only relying on functional difficulty axis) with Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean ± std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint over training from a single-seed run.
Figure 12: Teacher rewards for one-step medium and one-step hard proposals.
Figure 13: Pass@k performance on the LCB eval benchmark (LCB v5 ) for GASP with one-step instead of the two-step lemma and lift generation. We compare GASP with a one-step curriculum (intermediate difficulty: one-step medium, hard: one-step hard), Real-data RL, AZR , and Qwen2.5-Coder-7B . All RL-based results are repeated over three seeds. For each seed of our RL runs, we select the checkpoint with the best pass@20 within that run, and report mean ± std across seeds. For AZR , we evaluate the authors’ publicly released checkpoint (trained from Qwen2.5-Coder-7B ); the authors report selecting the best checkpoint over training from a single-seed run.
Table 4: Unique goalpost questions solved across training checkpoints (out of 146 goalposts). A question is counted if it is solved at any checkpoint within a run (pass@100 >0 ). Per-seed counts and union across three seeds are reported. Bold : global best. † Zero by construction (goalposts filtered to be unsolvable by these methods).
Figure 14: Goalpost questions solved by GASP without rejection sampling. Visualization over training checkpoints (x-axis: global step) and goalpost question IDs (y-axis). We include only goalpost questions that are solved at least once during training. A cell is marked as solved (green) if the corresponding question is solved at that checkpoint; otherwise it is unsolved (red). Results are aggregated across three seeds, where a cell is marked solved if any seed solves the question at that checkpoint (union across seeds).
Figure 15: Goalpost → lemma → lift examples from GASP run (with rejection sampling). Input–output-axis (io) selected for difficulty adjustment on hard questions with id abc364_e (top) and abc358_e (bottom). Shared motifs: for abc364_e maximizing count under cumulative resource constraints (1D → 2D feasibility); for abc358_e counting strings under per-letter usage caps (mod / DP).
Figure 16: Goalpost → lemma → lift examples from GASP run (with rejection sampling). Function-axis (f) selected for difficulty adjustment on hard question with id 3414 . Shared motif: counting paths / number of ways in a stateful process with constrained moves; DP on position + auxiliary state.
LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute, instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the language model itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4.
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.
Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang +1
In self-play, a proposer generates verifiable tasks to train a solver. Proposer rewards often depend on the solver's success rate, but equally difficult tasks can differ in their training value. We introduce CompassPlay, a self-play method that rewards the proposer through gradient alignment. The reward favors tasks whose solver loss gradients align with those of reference tasks representing the target capabilities. It draws on a first-order approximation to learning progress and scores each eligible task without additional solver training. Our experiments show gains in performance and training efficiency. In coding self-play with Qwen2.5-Coder-7B, a small external reference set guides task generation. CompassPlay improves average accuracy over AZR's difficulty reward by 1.5 percentage points on in-domain coding and 2.7 on out-of-domain mathematics. In Lean4 theorem proving, CompassPlay matches the difficulty baseline's 150-iteration cumulative coverage with 40% fewer GPU-hours.
Sophia Xiao Pu, Ximeng Sun, Jiang Liu +4
University of California, Santa Barbara · Advanced Micro Devices