The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning
Organizations: College of Computer and Information Engineering, Xi’an University of Architecture and Technology · Australian Artificial Intelligence Institute, University of Technology Sydney
Abstract
Reinforcement learning with verifiable rewards has become the dominant recipe for improving large language model reasoning, yet it presumes large human-curated task collections. Zero-data self-play removes this dependency, but existing methods vet learnability only by probing candidates and rejecting post hoc, never learning where along an environment's difficulty axis to place a task, and credit the solver with sparse terminal rewards alone. We recast zero-data self-play as a pursuit-evasion game: in LURE, an LLM evader positions tasks along each environment's difficulty axis to stay one step ahead of a planner-executor pursuer that hunts it down through verifiable interaction. The evader is trained on a capture-frontier reward that peaks when the solver captures it on exactly half of its rollouts, turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band. The pursuer earns capture-anchored dense process credit, in which monotone verifier progress is group-normalized jointly with the terminal capture under a round-anchored KL that keeps the co-evolution stable. Across three verifiable reasoning environments and three backbone families, LURE outperforms advanced baselines under unified/specialist settings, while the unified model attains stronger aggregate OOD zero-shot accuracy than all trained baselines across nine held-out benchmarks from three task families.
Figures & tables
| Unified model | Per-environment specialists | |||||||
| PhantomWiki | IFEval | ZebraLogic | PhantomWiki | IFEval | ZebraLogic | |||
| Method | SUCCESS | Prompt Acc. | Puzzle Acc. | Cell Acc. | SUCCESS | Prompt Acc. | Puzzle Acc. | Cell Acc. |
| Qwen2.5-7B-Instruct ( 18 ) | ||||||||
| Base | ||||||||
| GRPO ( 20 ) | ||||||||
| GRPO-Zero | ||||||||
| Configuration | PhantomWiki | IFEval | ZebraLogic | Mean | |
| Full | — | ||||
| Curriculum | |||||
| w/o learned challenger | |||||
| w/o capture-frontier reward | |||||
| w/o repetition penalty | |||||
| Pursuer credit | |||||
| Gold Kept | Gold Removed | ||
| (a) ZebraLogic ( Subset), R-Zero Round-5 Checkpoint | |||
| Old Grammar | |||
| New Grammar | |||
| (b) PhantomWiki, Same Leak, Every Method | |||
| Base | |||
| GRPO-Zero | |||
| Rollout budget per round | R-Zero | LURE (Ours) |
|---|---|---|
| Selecting the curriculum | ||
| Candidates emitted | ||
| Probe rollouts | ||
| Candidates discarded | ||
| Training on it | ||
| Tasks entering training | ||
| Benchmark | Base | GRPO | GRPO-Zero | R-Zero | LURE | |
|---|---|---|---|---|---|---|
| MATH500 | ||||||
| AIME24 | ||||||
| AIME25 | ||||||
| OlympiadBench | ||||||
| KOR-Bench | ||||||
| LiveBench-Reasoning |
Self-Play RL
Self-play reinforcement learning trains agents through repeated interaction with copies, opponents, or co-evolving task generators.
See all 77 papers