WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement

Fangyuan Li; Pengfei Li; Shijie Wang; Junqi Gao; Jianxing Liu; Biqing Qi; Yuqiang Li

cs.LGMar 22, 2026

WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement

Authors: Fangyuan Li, Pengfei Li, Shijie Wang, Junqi Gao, Jianxing Liu, Biqing Qi, Yuqiang Li

Organizations: Harbin Institute of Technology · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) offers a practical path to self-improvement of language models, but existing methods face a key trade-off: endogenous self-play can drift over iterations, while corpus-grounded approaches rely on curated data environments. We present WIST, a Web-grounded Iterative Self-play Tree framework for domain-targeted reasoning improvement that learns directly from the open web without requiring any pre-arranged domain corpus. WIST incrementally expands a domain tree for exploration, and retrieves and cleans path-consistent web corpus to construct a controllable training environment. It then performs Challenger--Solver self-play with verifiable rewards, and feeds learnability signals back to update node posteriors and guide subsequent exploration through an adaptive curriculum. Across four backbones, WIST consistently improves over the base models and typically outperforms both purely endogenous self-evolution and corpus-grounded self-play baselines, with the Overall gains reaching +9.8 (Qwen3-4B-Base) and +9.7 (OctoThinker-8B). WIST is also domain-steerable, improving Qwen3-8B-Base by +14.79 in medicine and Qwen3-4B-Base by +5.28 on PhyBench. Ablations further confirm the importance of WIST's key components for stable open-web learning. Our Code is available at https://github.com/lfy-123/WIST.

Figures & tables

Figure 1: Comparison of R-Zero, SPICE, and our WIST.

Figure 2: Overview of our proposed WIST, a web-grounded iterative self-play Tree framework for domain-targeted reasoning improvement.

	Mathematical Reasoning							General Reasoning
Method	AMC	Minerva	MATH 500	GSM8K	Olymp.	AIME 24	AIME 25	Super- GPQA	GPQA- Diamond	MMLU- Pro	BBEH	Overall
Qwen3-4B-Base
Base Model	41.4	35.7	57.0	75.9	30.2	9.5	6.4	18.0	32.8	51.5	8.2	33.3
+ R-Zero	53.5	44.1	77.0	91.1	39.5	10.3	7.1	26.7	33.4	53.7	10.4	40.6
+ SPICE	50.1	47.8	76.2	92.5	41.0	12.0	10.9	27.8	35.1	54.3	11.8	41.8
+ WIST (ours)	60.0	47.8	78.2	92.9	40.0	11.6	9.7	29.6	37.2	55.7	11.8	43.1

Table 1: Main results on mathematical and general reasoning benchmarks across four backbones. Best and second-best results within each backbone block are marked in bold and underline , respectively.

Panel of Figure 3 — Figure 3: Training performance of our method WIST in the medical domain using three Qwen3 models of different sizes.

Panel of Figure 4 — Figure 4: Ablation results of WIST , including domain transfer, hyperparameter sensitivity, tree scaling, and exploration strategy.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	WIST	SPICE	R-Zero
Data Source
Corpus documents	$\sim 15,000$	20,000	–
Question source	Web-grounded	Document-grounded	Self-generated
External grounding	$\checkmark$	$\checkmark$	$\times$
Training Details
Challenger training	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 2: Training configurations for all compared methods.

Hyperparameter	Value
Rollout batch size (paths per iteration) $B$	128
Training batch size $B_{\mathrm{train}}$	512
Tree maximum depth $\mathcal{L}$	4
Beta prior $(\alpha_{0},\beta_{0})$	$(1,1)$
Learnability threshold (variance) $\tau_{\mathrm{var}}$	0.20
Sliding window size $\mu$	5

Appendix

Table 3: Main hyperparameters for WIST.

	Mathematical Reasoning							General Reasoning
Method	AMC	Minerva	MATH 500	GSM8K	Olymp.	AIME 24	AIME 25	Super- GPQA	GPQA- Diamond	MMLU- Pro	BBEH	Overall
Qwen3-14B-Base
Base Model	62.5	50.0	80.2	93.7	43.1	13.3	11.4	35.2	40.4	63.9	14.0	46.2
+ R-Zero	60.3	52.9	81.4	94.4	45.2	14.3	11.8	37.1	44.4	65.3	14.9	47.5
+ SPICE	60.2	56.3	83.8	94.5	46.7	15.2	15.1	37.1	45.5	65.4	15.2	48.6
+ WIST (ours)	67.5	56.6	83.4	94.7	46.8	15.4	15.8	37.4	44.4	66.0	15.3	49.4

Appendix

Table 4: Additional results on Qwen3-14B-Base across mathematical and general reasoning benchmarks. Best and second-best results within this backbone block are marked in bold and underline , respectively.

	Mathematical Reasoning							General Reasoning
Variant	AMC	Minerva	MATH 500	GSM8K	Olymp.	AIME 24	AIME 25	Super- GPQA	GPQA- Diamond	MMLU- Pro	BBEH	Overall
Qwen3-4B-Base
Base Model	41.4	35.7	57.0	75.9	30.2	9.5	6.4	18.0	32.8	51.5	8.2	33.3
+ WIST (Random)	51.1	46.0	78.0	92.8	40.1	11.9	9.8	28.0	37.9	53.5	11.7	41.9
+ WIST (Reward)	60.0	47.8	78.2	92.9	40.0	11.6	9.7	29.6	37.2	55.7	11.8	43.1

Appendix

Table 5: Full-benchmark comparison between random and reward-guided tree expansion under the same training budget.

Method	Medbullets	MedMCQA	MedQA	Avg.
Qwen3-4B-Base
Base Model	24.03	32.27	48.52	34.94
+ R-Zero	24.03	32.27	51.50	35.93
+ SPICE	24.35	32.39	52.69	36.48
+ WIST (Random)	24.19	32.30	52.76	36.42
+ WIST (Reward)	24.35	32.37	53.42	36.71

Appendix

Table 6: Medical-domain results after steering the target domain to medicine. We report benchmark-level accuracy on Medbullets, MedMCQA, and MedQA, together with their average. Best and second-best results within each backbone block can be marked in bold and underline , respectively.

	Mathematical Reasoning							General Reasoning
Variant	AMC	Minerva	MATH 500	GSM8K	Olymp.	AIME 24	AIME 25	Super- GPQA	GPQA- Diamond	MMLU- Pro	BBEH	Avg.
Base Model	41.4	35.7	57.0	75.9	30.2	9.5	6.4	18.0	32.8	51.5	8.2	33.3
+ WIST (Strong-Tree)	53.2	49.3	78.2	92.5	39.7	11.5	9.6	28.5	37.4	55.4	11.7	42.5
+ WIST (Self-Tree)	60.0	47.8	78.2	92.9	40.0	11.6	9.7	29.6	37.2	55.7	11.8	43.1

Appendix

Table 7: Tree transfer from a stronger builder model ( Qwen3-14B ) to a smaller learner ( Qwen3-4B-Base ) in mathematics.

	Mathematical Reasoning							General Reasoning
Method	AMC	Minerva	MATH 500	GSM8K	Olymp.	AIME 24	AIME 25	Super- GPQA	GPQA- Diamond	MMLU- Pro	BBEH	Overall
Qwen3-4B-Base
Base Model	41.4	35.7	57.0	75.9	30.2	9.5	6.4	18.0	32.8	51.5	8.2	33.3
+ WIST (HTML-only)	56.6	48.6	77.0	91.8	40.0	11.3	9.5	28.6	31.8	55.1	12.1	42.0
+ WIST (full filtering)	60.0	47.8	78.2	92.9	40.0	11.6	9.7	29.6	37.2	55.7	11.8	43.1
Qwen3-8B-Base

Appendix

Table 8: Ablation on web data processing. We report the performance over all mathematical and general reasoning benchmarks.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 172 papers

H $^2$ SD: Hybrid Hindsight Self-Distillation
Jul 21, 2026Qiye Cai, Yichuan Ma, Weitao Ma +6RL for Language Model Reasoning Language Model Distillation
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +1
OpenSIR: Open-Ended Self-Improving Reasoner
Nov 1, 2025Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3Self-Play RL RL for Language Model Reasoning
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning
Apr 22, 2026Wenjie Liao, Like Wu, Liangjie Zhao +2Fine-Tuning LLM Fine-Tuning
Graduate School of Information, Production and Systems, Waseda University, Fukuoka, Japan · Institute of Computing Technology, University of the Chinese Academy of Sciences

Jul 21, 2026J/K move · Enter open · S save

H $^2$ SD: Hybrid Hindsight Self-Distillation

Qiye Cai, Yichuan Ma, Weitao Ma, Peiji Li+5

Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University+1

RL for Language Model Reasoning Language Model Distillation Reinforcement Learning with Verifiable Rewards Self-Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but scalar trajectory rewards offer limited guidance for token-level credit assignment. Existing self-distillation methods incorporate privileged teacher information through fixed strategies: direct distribution matching may destabilize successful behavior, while magnitude modulation provides limited corrective guidance after failure. We propose Hybrid Hindsight Self-Distillation (H

^2

SD), which adapts both teacher context and update strategy to trajectory correctness. For successful trajectories, the teacher conditions on the verified student response and a rephrasing instruction to rescore the original tokens. These scores modulate reward-derived updates while preserving advantage signs. For failed trajectories, a verified reference hint guides corrective supervision through reverse KL distillation. Across Sudoku, Calcudoku, Arrow Maze, and mathematical reasoning with two Qwen3 models, H

^2

SD achieves an overall average accuracy of 43.33%, outperforming the strongest baseline by 11.43 percentage points while maintaining stable optimization.

Abstract

Language Model Self-Improvement

Explore similar work

H2^22SD: Hybrid Hindsight Self-Distillation

OpenSIR: Open-Ended Self-Improving Reasoner

IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning

H $^2$ SD: Hybrid Hindsight Self-Distillation