Test-Time Self-Play: Scaling without a Verifier

Akira Yoshiyama; Akmal Ashirmatov; Jonas Hübotter; Ido Hakimi; Andreas Krause

cs.LGNov 5, 2026

Test-Time Self-Play: Scaling without a Verifier

Authors: Akira Yoshiyama, Akmal Ashirmatov, Jonas Hübotter, Ido Hakimi, Andreas Krause

Organizations: ETH Zurich & Tufa Labs · ETH Zurich

Abstract

Test-time scaling methods spend additional compute at deployment to improve a large language model's performance on a hard target problem. Most methods rely on a verifier that scores candidate solutions for correctness, using it to select among or refine the model's outputs. In many deployment regimes, however, a strong verifier on the target problem is unavailable. We introduce Test-Time Self-Play (TTSP), an approach to test-time scaling using asymmetric self-play without reliance on a verifier. On a difficulty-calibrated subset of LiveCodeBench v6, TTSP is the best-performing method at every verifier strength we evaluate, and the gap to the strongest baseline widens as the verifier weakens; in the no-verifier limit, TTSP yields a 35% absolute improvement in mean@8. The gains hold across a range of verifier strengths, except when the base policy has near-zero coverage of the correct answer.

Figures & tables

Figure 1: TTSP outperforms all baseline test-time scaling methods in the weak verifier regime. TTSP and ACE are evaluated at converged checkpoints; FMV uses public and private input test cases.

Figure 2: Overview of Test-Time Self-Play. Given a target problem

x^{\star}

and previously generated synthetic problems, the proposer policy

\pi_{\phi}

generates target-conditioned synthetic problems

\mathcal{D}

consisting of a problem statement, tests, and a reference solution. The solver policy is rewarded for correctly solving the synthetic problems, while the proposer is rewarded for generating synthetic problems near the solver’s current learning frontier (both via GRPO; see the red arrows). The synthetic problem history provides context for the proposer and fills a replay buffer

\mathcal{B}

. After a sufficient number of iterations, the test-time trained solver can then attempt the target problem.

Target Problem $x^{\star}$	Concept
For each $A_{i}$ , maximize the GCD of a size- $K$ subset containing $A_{i}$ .	Largest feasible divisor of $A_{i}$
Synthetic Problems
Find the largest integer $d$ that divides at least $K$ elements of the array.	Feasibility predicate: $\mathrm{cnt}[d]\geq K$
For every divisor $d$ , compute $\mathrm{cnt}[d]=\|\{j:d\mid A_{j}\}\|$ .	Multiples-sieve divisor counts
For each $A_{i}$ , output the largest divisor $d\mid A_{i}$ with $\mathrm{cnt}[d]\geq K$ .	Per-element feasible divisor search

Table 1: Example synthetic curriculum. The synthetic problems are generated by the proposer, teaching the solver concepts which are useful to recover the final strategy needed to solve the target problem.

Method	Verifier Access
Rejection sampling	Inputs + Outputs
Context optimization	Inputs + Outputs
Majority voting	Inputs Only
TTSP (ours)	None

Table 2: Required verifier access per method. Each method requires test case inputs, test case outputs, neither, or both, from which a learning signal can be derived.

Figure 3: Baselines performance on HardLCBv6 as compute scales. Each line reports the mean@8 with the indicated evaluation-time verifier applied to the validation problem submissions. Left and middle: Qwen 3 8B and FMV scale on the number of sampled rollouts k. Right: ACE scales on the number of generation–reflection–curation steps.

Figure 6: Unfiltered mean@8 of TTSP and FMV binned by rank of the correct functional cluster in the base policy. Bars group target problems by the rank of the correct functional cluster among base-policy rollouts.

Figure 7: Difference in mean@8 between TTSP and Qwen3-8B. The gap is large and positive on MidHardLCBv6 as the verifier weakens. On VeryHardLCBv6, the gap is near zero: only marginally positive under the weakest verifiers.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Target Problem	mean@512	Correct Solution Rank
1035	0.686	1
1050	0.645	1
976	0.633	1
920	0.596	1
1047	0.555	1
962	0.523	1

Appendix

Table 3: MidHardLCBv6 target problems. LiveCodeBench v6 target problem identifiers and base model mean@512 of Qwen 3 8B for each selected target problem in the MidHardLCBv6 subset (

0.1<\mathrm{mean}@512<0.7

Target Problem	mean@512	Correct Solution Rank
941	0.057	3
1017	0.045	2
946	0.043	3
919	0.018	3
1027	0.012	2
947	0.010	11

Appendix

Table 4: VeryHardLCBv6 target problems. LiveCodeBench v6 target problem identifiers and base model mean@512 of Qwen 3 8B for each selected target problem in the VeryHardLCBv6 subset (

0<\mathrm{mean}@512\leq 0.1

Verifier	Mean tests used
	Generated	Public	Private	Total
$v_{0}$	0.00	0.00	0.00	0.00
$v_{25}$	0.36	0.07	0.86	1.29
$v_{50}$	0.39	0.07	1.00	1.46
$v_{75}$	0.57	0.18	0.79	1.54
$v_{\mathrm{pub}}$	0.00	2.86	0.00	2.86

Appendix

Table 5: Tests used by each evaluation-time verifier on HardLCBv6. Values are averages over the 28 target problems and are grouped by test source.

Setting	Value
Model
Backbone	Qwen3-8B
Trainable parameters	Separate proposer and solver LoRA adapters
LoRA rank / $\alpha$	$32/32$
LoRA target modules	All linear layers
Optimization

Appendix

Table 6: Core TTSP training hyperparameters. All values are fixed across target problems. The model is adapted per target through separate proposer and solver LoRA adapters, while the Qwen 3 8B backbone remains frozen.

Setting	Value
Generation
Inference engine	vLLM
Sampling temperature	$1.0$
Max prompt / response length	$8096/16384$ tokens
Proposer contexts per step	$2$
Proposer rollouts per context	$8$

Appendix

Table 7: Curriculum, rollout, and replay hyperparameters. Fresh synthetic problems are generated by the proposer; replayed synthetic problems are sampled from history and re-evaluated under the current solver policy when replayed.

Figure 8: FMV under partial test-input access on HardLCBv6. The final evaluation-time filter is removed, and all submitted candidates are scored directly against the public+private evaluator $v^{\star}$ . The x-axis denotes the test inputs available to FMV for functional clustering, not a post-hoc verifier. TTSP and Qwen 3 8B are shown as unfiltered baselines independent of FMV’s input-access condition.

Figure 9: Synthetic problem rejection diagnostics. Pre-training proposer generations rejected before entering solver training, averaged over target problems and grouped by failure type. Rejection counts decrease over training; invalid generated output is the largest rejection category.

Figure 10: Average and top- $4$ proposer reward during TTSP training. The Proposer curve is the mean proposer reward over generated synthetic problems, while Proposer (top-4) averages the four highest proposer rewards at each step. The gap measures within-batch reward concentration. Solver reward and unfiltered target validation are shown for reference. Shaded bands denote $\pm 1$ SEM across target problems.

Figure 11: Frozen-base and adapted-solver performance on generated synthetic problems. Example trajectory for one target problem. Frozen Qwen 3 8B is the initial model evaluated on the synthetic problems generated at each step; TTSP solver is the adapted solver evaluated on the same synthetic problems; Target validation is the adapted solver’s unfiltered mean@8 on the original target problem under $v_{0}$ .

Figure 12: Policy entropy during TTSP training. Mean proposer-policy and solver-policy entropy across GRPO steps on HardLCBv6, averaged over target problems. The proposer entropy increases over training, while solver entropy remains stable. Shaded regions show variation across target problems. Shaded bands denote $\pm 1$ SEM across target problems.

Panel of Figure 13 — Figure 13: Curriculum novelty and target relatedness. Left: cumulative normalized Vendi gain computed from embeddings of extracted Python reference programs. Right: text-level cosine similarity between generated synthetic problem statements and the original target problem statement. Shaded bands denote $\pm 1$ SEM across target problems.

Figure 14: Per-target TTSP training curves on MidHardLCBv6. Mean@8 over GRPO steps for each individual target problem, with one panel per target and one line per evaluation-time verifier strength. Many targets show the rapid rise within $10$ – $20$ steps and stable plateau seen in the aggregate curve of Figure 5 .

Figure 15: Functional output clusters for MidHardLCBv6. Qwen 3 8B functional output cluster distribution for each target problem in MidHardLCBv6. Each panel shows, for one target, the fraction of $512$ base rollouts whose execution traces fall into each functional cluster, with the correct cluster highlighted in green.

Figure 16: Per-target TTSP training curves on VeryHardLCBv6. Mean@8 over GRPO steps for each individual target problem, with one panel per target and one line per evaluation-time verifier strength. The flat or noisy curves illustrate the discovery-side failure mode discussed in Section 5.4 when base coverage is below $\mathrm{mean}@512\approx 0.1$ .

Figure 17: Functional output clusters for VeryHardLCBv6. Qwen 3 8B functional output cluster distribution for each target problem in VeryHardLCBv6. Each panel shows, for one target, the fraction of $512$ base rollouts whose execution traces fall into each functional cluster, with the correct cluster highlighted in green. The correct cluster typically carries negligible mass, matching the discovery-regime characterization in Figure 7 .

Self-Play for Language Models

See all 8 papers

Aug 19, 2026

Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck

Test-time scaling (TTS) improves language model outputs by spending additional inference compute - generating multiple candidates, searching over partial sequences, or iteratively refining drafts. These techniques yield large gains on mathematics and code, but have been developed and stress-tested almost exclusively on tasks where verification is straightforward. We conduct the first compute-normalised comparison of five TTS families across five open-ended generation benchmarks spanning medicine, law, finance, general chat, and creative writing - grounded in a unified framework that decomposes the effectiveness of each method's token budget into exploration and exploitation. The answer depends on which side of that decomposition you examine. Scaling exploration works: the best candidate in the pool improves steadily with compute across all settings. What breaks is exploitation - the step that converts a rich candidate pool into a final output. With state-of-the-art generators, reward models correlate at only

ρ_v \approx 0.12

with true quality, rendering selection near-random regardless of budget. Tree search amplifies this failure through diversity collapse. Refinement helps on one of five benchmarks; its apparent gains elsewhere are confounded. Only synthesis across candidates (Fusion) consistently improves over single-sample baselines, yet still recovers only ~40% of available quality. The candidate pool is not the bottleneck - choosing from it is.

Davide Romano, Kanak Raj, Jerrod Parker +1

Thomson Reuters Foundational Research, 5 Canada Square, London

Test-Time Scaling Language Model Decoding Open-Ended Generation Language Model Generation Evaluation Reward Modeling

Ask PDF

Oct 1, 2026

How Much Can Language Models Gain from Test-Time Computation?

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.

Bangji Yang, Jingyuan Li, Jiajun Fan +8

University of Illinois at Urbana-Champaign · University of Washington · Tsinghua University

LLM Evaluation Test-Time Scaling

Ask PDF

Nov 6, 2026

Test-Time Compute Scales Better with Parallel Agent Teams

Test-time scaling improves the performance of a fixed model by spending more computation at inference. There are two common ways to spend it on agents when a verifier can score every attempt. Best-of-N sampling runs many independent agents and keeps the best result but saturates quickly. One large team of collaborating agents can score higher but stops improving as it grows. Can we get the best of both? Our method, parallel agent teams, does so by sampling many small teams in parallel and keeping the best team's result. We test it on 12 long-horizon tasks from GPU kernels and formal proofs to exploit development. With a 27B model and a budget of 100 agents, parallel teams of ten beat parallel agents on 10 of 12 tasks and beat one team of 100 on all four tasks where we ran such large teams. At equal dollar cost the same parallel teams beat a frontier model on half the tasks. A simple statistical model shows that individual agents lose because their many samples have a low mean and that one large team loses because a single sample gets no gain from variance. Sampling many small teams in parallel balances a high mean with the gain from variance. Parallel agent teams thus offer a more effective way to scale test-time compute.

Junlin Wang, Haozhe Chen, Ang Li +2

Duke University · Modal Labs · Princeton University +1

Test-Time Scaling Multi-Agent Collaboration

Ask PDF

Abstract

Self-Play for Language Models

Explore similar work

Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck

How Much Can Language Models Gain from Test-Time Computation?

Test-Time Compute Scales Better with Parallel Agent Teams