CompassPlay: Rewarding the Proposer for Where It Moves the Solver

Sophia Xiao Pu; Ximeng Sun; Jiang Liu; Jialian Wu; Emad Barsoum; Zicheng Liu; William Yang Wang

cs.LGSep 26, 2026

CompassPlay: Rewarding the Proposer for Where It Moves the Solver

Authors: Sophia Xiao Pu, Ximeng Sun, Jiang Liu, Jialian Wu, Emad Barsoum, Zicheng Liu, William Yang Wang

Organizations: University of California, Santa Barbara · Advanced Micro Devices

Abstract

In self-play, a proposer generates verifiable tasks to train a solver. Proposer rewards often depend on the solver's success rate, but equally difficult tasks can differ in their training value. We introduce CompassPlay, a self-play method that rewards the proposer through gradient alignment. The reward favors tasks whose solver loss gradients align with those of reference tasks representing the target capabilities. It draws on a first-order approximation to learning progress and scores each eligible task without additional solver training. Our experiments show gains in performance and training efficiency. In coding self-play with Qwen2.5-Coder-7B, a small external reference set guides task generation. CompassPlay improves average accuracy over AZR's difficulty reward by 1.5 percentage points on in-domain coding and 2.7 on out-of-domain mathematics. In Lean4 theorem proving, CompassPlay matches the difficulty baseline's 150-iteration cumulative coverage with 40% fewer GPU-hours.

Figures & tables

Figure 1: (a) CompassPlay rewards alignment between the solver’s loss gradients on verified candidate solutions and on reference solutions. (b) Mean accuracy on five out-of-domain mathematics benchmarks during coding self-play. Curves show the mean and

\pm 1

s.d. across three seeds, averaged over three evaluation checkpoints. (c) Lean self-play with novelty weighting: cumulative coverage of 3,323 target theorems from iteration 10, with one run per method.

Method	LiveCodeBench	CRUXEval-I	CRUXEval-O	Mean
Initial model	40.02	48.98	50.89	46.63
solve-only	46.78	58.05	60.20	55.01
AZR	49.87 $\pm$ 1.62	59.63 $\pm$ 0.12	61.23 $\pm$ 0.47	56.91 $\pm$ 0.72
CompassPlay ( cos )	52.60 $\pm$ 0.17	60.08 $\pm$ 0.34	62.56 $\pm$ 0.44	58.41 $\pm$ 0.18
Gain over AZR (pp)	+2.73	+0.45	+1.33	+1.50

Table 1: In-domain coding accuracy (%). Trained methods are averaged over steps 150–200. CompassPlay and AZR report mean

\pm

sample s.d. across three seeds; solve-only has one run. Initial model is Qwen2.5-Coder-7B before self-play. Mean averages benchmarks equally. Gain over AZR is the CompassPlay improvement in percentage points (pp).

Method	MATH500	AMC23	GSM8K	Minerva	OlympiadBench	Mean
Initial model	53.40	45.00	67.00	18.00	19.30	40.54
solve-only	66.11	37.73	78.25	26.84	31.22	48.03
AZR	60.73 $\pm$ 2.63	43.94 $\pm$ 0.92	76.98 $\pm$ 1.83	26.54 $\pm$ 1.40	30.40 $\pm$ 2.04	47.72 $\pm$ 1.07
CompassPlay ( cos )	63.34 $\pm$ 3.77	47.12 $\pm$ 6.56	78.72 $\pm$ 1.95	29.14 $\pm$ 2.34	33.53 $\pm$ 1.58	50.37 $\pm$ 2.87
Gain over AZR (pp)	+2.61	+3.18	+1.74	+2.60	+3.14	+2.65

Table 2: Out-of-domain mathematics accuracy (%), using the same averaging as Table 1 . Initial model reports accuracy before self-play. Gain over AZR is the CompassPlay improvement in percentage points (pp).

Figure 2: Accuracy over training on three coding and five mathematics benchmarks. The figure labels AZR as difficulty . These two methods report means over three seeds with

\pm 1

sample s.d. error bars; solve-only is a single run. The shaded band marks steps 150–200, the averaging window in Tables 1 and 2 .

Figure 3: Mean coding and mathematics accuracy against cumulative GPU-hours. CompassPlay ( cos ) reports mean accuracy and cost across three seeds; SOAR is our single-run reimplementation. Each point is an evaluation checkpoint. GPU-hours are training wall-clock time multiplied by the number of GPUs, excluding benchmark evaluation. Appendix B gives the comparison with AZR at matched GPU-hours.

Figure 4: Copy rate: the fraction of eligible conjectures with token-Jaccard similarity to the seed above 0.8. The cos and cos

\times

novelty curves denote the two CompassPlay variants.

Panel of Figure 5 — Figure 5: Cumulative coverage of CompassPlay ( cos $\times$ novelty ) and baselines at available checkpoints through iteration 150. The dotted line marks difficulty ’s coverage at iteration 150. GPU-hours are computed as training wall-clock time times 8 GPUs.

Method	Alignment	Novelty	Copy rate $\downarrow$	Gain (targets) $\uparrow$
CompassPlay ( cos )	$\checkmark$	$\times$	0.712	+3.1
novelty-only	$\times$	$\checkmark$	0.000	$-7.9$
CompassPlay ( cos $\times$ novelty )	$\checkmark$	$\checkmark$	0.000	+13.6

Table 3: Ablation of CompassPlay ( cos

\times

novelty ). Copy rate is measured at iteration 115. Gain is the mean number of additional targets solved over difficulty , using available paired checkpoints in iterations 15–115.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

reward	proposer reward (s)	optimizer update (s)	remainder (s)	step total (s)
difficulty	72.1 (44%)	13.7 (8%)	77.1 (47%)	162.9
cos (ours)	128.8 (58%)	13.4 (6%)	80.2 (36%)	222.4

Appendix

Table 4: Per-phase wall-clock of one training step (seconds; share of the step in parentheses), one run of each method on 8 GPUs, mean over the run.

	in-domain			OOD
budget (GPU-h)	cos	difficulty	SOAR	cos	difficulty	SOAR
20	49.07	48.94	46.78	37.24	39.33	38.36
40	53.20	54.98	43.94	43.15	43.65	34.42
60	57.23	57.38	45.75	47.09	46.85	32.40
69	57.86	56.90	45.75	48.77	48.10	32.40
endpoint	59.20 (95)	57.60 (71)	42.15 (109)	50.56 (95)	47.93 (71)	33.16 (109)

Appendix

Table 5: Mean accuracy (%) at matched GPU-hours. For CompassPlay ( cos ) and AZR ( difficulty ), accuracy and cumulative cost are averaged across three seeds at each evaluation step; SOAR is our single-run reimplementation. Each budget uses the last step whose mean cumulative cost is within budget. The endpoint row gives final-checkpoint accuracy and total GPU-hours in parentheses; these are distinct from the final-quarter averages in Tables 1 and 2 .

Figure 6: Reference-gradient ablation: CE reference (the main method) against REINFORCE reference gradients, accuracy against cumulative GPU-hours. Single runs with a shared seed.

Figure 7: Cosine against dot product as the alignment score, accuracy against cumulative GPU-hours, with REINFORCE gradients on both sides (Appendix C.1 ). The two runs share the seed and differ in one flag.

Method	Iteration	Pure-rename %	Near-copy %	Jaccard $>0.8$ %
CompassPlay ( cos )	39	77.50	92.20	91.06
CompassPlay ( cos $\times$ novelty )	40	1.01	4.48	1.46
difficulty	39	0.27	2.45	0.54

Appendix

Table 6: Pure-rename and token-overlap rates at each method’s audited iteration (percent).

	coding self-play	theorem-proving self-play
base loop	AZR ( Zhao et al., 2025 )	adapted SGS ( Bailey et al., 2026 )
proposer / solver weights	shared, one model	separate copies of one base model
environment $v$	Python execution	Lean 4 + mathlib4
tasks	program-based deduction, abduction, induction	Lean statements
proposer conditioning $z$	past self-generated tasks	a seed statement the solver has proved
verified solution $y^{\star}(\tilde{x})$	oracle answer in the solver’s format	up to $m=4$ verified proofs, averaged

Appendix

Table 7: The two instantiations of the reward.

Self-Play RL

Self-play reinforcement learning trains agents through repeated interaction with copies, opponents, or co-evolving task generators.

See all 83 papers

Scaling Self-Play with Self-Guidance
Apr 22, 2026Luke Bailey, Kaiyue Wen, Kefan Dong +2Self-Play RL Automated Theorem Proving
Stanford University
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
May 29, 2026Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang +1Open-Ended Generation Self-Play RL
University of Edinburgh · Imperial College London
GASP: Guided Asymmetric Self-Play For Coding LLMs
Mar 16, 2026Swadesh Jana, Cansu Sancaktar, Tomáš Daniš +3Language Model Self-Improvement LLM Code Generation
University of Tübingen · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen +1

Apr 22, 2026J/K move · Enter open · S save

Scaling Self-Play with Self-Guidance

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto+1

Stanford University

Self-Play RL Automated Theorem Proving LLM-Guided RL Language Model Self-Improvement

Open paper AskPDF arXiv

LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute, instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the language model itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4.

Abstract

Self-Play RL

Explore similar work

Scaling Self-Play with Self-Guidance

SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

GASP: Guided Asymmetric Self-Play For Coding LLMs