More: Mode-Optimized Reasoning with Self-Adaptive Effort for Self-Evolving Agents

Dianxing Shi; Dingjie Fu; Tianyu Lei; Wangjie Qiu

cs.AIOct 24, 2026

More: Mode-Optimized Reasoning with Self-Adaptive Effort for Self-Evolving Agents

Authors: Dianxing Shi, Dingjie Fu, Tianyu Lei, Wangjie Qiu

Organizations: Beihang University · Huazhong University of Technology

Abstract

Self-evolving training improves the performance of agents on difficult tasks through verifiable feedback, but it often applies costly deliberation uniformly across tasks. To address this problem, we propose Mode-Optimized Reasoning with Self-Adaptive Effort (MORE), a self-evolving training framework that adaptively selects both reasoning form and computational effort for each task. Specifically, MORE follows a two-stage training paradigm: (i) Stage 1 establishes a compact prior over direct answering, short and long chain-of-thought, and code-based reasoning, and (ii) Stage 2 self-evolves this prior with Mode-Adaptive and Test-time Effort PPO (MATE-PPO) on generated, verifiable task streams without curated data. By learning mode selection and token expenditure jointly, MORE allocates concise reasoning to tasks that permit it while retaining extended deliberation where it is useful. Across coding, mathematics, and general-reasoning benchmarks, MORE achieves consistent performance gains on Base and Coder backbones at model scales from 3B to 14B. During inference, MORE demonstrates higher accuracy and better efficiency, with markedly shorter response lengths and reduced inference time.

Figures & tables

Figure 1: Left : MORE uses Two-Stage self-evolving training with a mode-difficulty aware budget to select the most efficient reasoning mode for each task while maintaining appreciable accuracy. Right : Across coding, mathematics, and general-reasoning tasks, MORE markedly improves average accuracy and efficiency over both its underlying backbone and Absolute-Zero Reasoner (AZR).

Figure 2: Overview of the MORE framework. Mode-format supervision initializes four reasoning contracts. MATE-PPO then couples self-evolving verifiable rollouts with online difficulty estimation, mode–budget control, a joint terminal return, and a clipped policy update. The diagram summarizes correctness gating: mode-fit and joint credit require a correct response, while the budget-deviation term regulates overlength for both correct and incorrect responses.

Model	HE +	MBPP +	LCB	CAvg	A24	A25	AMC	M500	Min.	Oly.	MAvg	GSM8K	M-Pro	AVG
Base Models
Qwen2.5-7B-Base [ 19 ]	73.2	65.3	17.5	52.0	6.7	3.3	37.5	64.8	25.0	27.7	27.5	80.4	47.6	51.9
Qwen2.5-Coder-7B [ 9 ]	80.5	69.3	19.9	56.6	6.7	3.3	40.0	54.0	17.3	21.9	23.9	72.2	42.5	48.8
Qwen2.5-Math-7B [ 28 ]	61.0	57.9	16.2	45.0	10.0	16.7	42.5	64.2	15.4	28.0	29.5	73.5	39.8	47.0
Zero-Style Reasoners: Curated Coding Data
AceCoder-RM [ 29 ]	78.0	66.4	27.5	57.3	13.3	3.3	27.5	62.6	29.4	29.0	27.5	83.7	43.2	52.9

Table 1: Performance of reasoning models based on Qwen2.5-7B variants. CAvg is the mean of HumanEval + , MBPP + , and the LiveCodeBench v1–v5 average; MAvg is the mean of the six mathematical reasoning benchmarks. AVG is the mean of CAvg, MAvg, GSM8K, and MMLU-Pro. A24/A25 denote AIME 2024/2025, M500 denotes MATH-500, Min. denotes Minerva Math, and Oly. denotes OlympiadBench. The best and second-best results are shown in bold and underlined , respectively. Appendix A.2 provides more implementation, training-data, and initialization details.

Figure 3: In-distribution and out-of-distribution reasoning task performance. (a) Average scores on CruxEval-I, CruxEval-O, and LiveCodeBench-Execution, corresponding to abduction, deduction, and deduction task types, respectively, evaluate in-distribution abilities of AZR and MORE throughout training across Coder 3B, 7B, and 14B backbones. (b) Aggregate out-of-distribution scores for Coder-3B, Base-14B, and Coder-14B variants. Complete results appear in Appendix H .

Figure 4: Long-horizon out-of-distribution trajectories for matched Base-7B and Coder-7B models. Columns report Code Avg, Math Avg, and MMLU-Pro (as general reasoning).

Figure 5: Training-time metrics. (a) Rollouts’ mode proportions for MORE 7B models. (b) Mean prompt-plus-response tokens per rollout for matched MORE and AZR runs.

Metric	Model	Code Avg	Math Avg	GSM8K	MMLU-Pro
Response Length Avg	Qwen2.5-7B-Base	394.2	3975.7	499.2	553.3
	AZR-7B	603.7	4468.2	412.6	485.4
	MORE -7B	291.5	3173	195.9	351.7
	Qwen2.5-7B-Coder	366.7	4225.1	418.5	501.0
	AZR-Coder-7B	537.4	5769.3	322.5	537.9
	MORE -Coder-7B	191.73	3015.9	105.1	303.4

Table 2: Out-of-distribution evaluation efficiency. Response Length Avg is the sample-mean number of generated response tokens, and Wall Time is the total elapsed time for the corresponding benchmark suite. The best results within each Base or Coder model family are highlighted in bold .

Method	HE +	MBPP +	LCB	CAvg	A24	A25	AMC	M500	Min.	Oly.	MAvg	GSM8K	M-Pro	AVG
Qwen2.5-Coder-7B (Base)	80.5	69.3	19.9	56.6	6.7	3.3	40.0	54.0	17.3	21.9	23.9	72.2	42.5	48.8
Base +10K	80.9	69.7	21.4	57.3	6.7	3.3	42.4	55.2	17.1	22.2	24.5	71.9	42.8	49.1
Base + Stage 1	77.3	65.7	13.7	52.2	3.3	0.0	34.8	50.1	11.5	19.6	19.9	70.0	39.1	45.3
Base + Stage 2	81.1	67.4	20.4	56.3	3.3	3.3	39.4	52.7	17.8	19.5	22.7	71.6	40.7	47.8
AZR-Coder-7B	83.5	69.6	31.7	61.6	20.0	10.0	57.5	72.6	36.4	38.2	39.1	72.5	44.6	54.4
AZR +10K	84.0	68.3	32.6	61.6	20.0	13.3	58.0	72.9	36.8	38.3	39.9	72.7	44.5	54.7

Table 3: Ablation results on Qwen2.5-Coder-7B. +10K increases the maximum response length to 10,240 tokens during training process. The best result in each column is shown in bold .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Shared runtime
Accelerators	$4\times$ NVIDIA A100 (80 GB)
Numerical precision	BF16 mixed precision
Stage-2 rollout tensor parallelism	TP $=2$
Stage 1: mode-format SFT
SFT method	All-target LoRA

Appendix

Table 4: Training hyperparameters for the reported main runs. The six Stage-2 streams comprise proposer and solver rollouts for each executable task family.

Model	Data Curation	Base Model
Qwen2.5-7B-Base [ 19 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-7B-Base
Qwen2.5-Coder-7B [ 9 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-Coder-7B
Qwen2.5-Math-7B [ 28 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-Math-7B
Qwen2.5-Coder-3B [ 9 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-Coder-3B
Qwen2.5-14B-Base [ 19 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-14B-Base
Qwen2.5-Coder-14B [ 9 ]	N/A (off-the-shelf base checkpoint)	Qwen2.5-Coder-14B

Appendix

Table 5: Training-data source and backbone provenance for the models in Table 1 and Figure 3 . For MORE rows, Stage-2 entries state whether self-evolution uses curated data.

Figure 6: A shared Stage-1 question expressed through the four response contracts. Direct, Short CoT, Long CoT, and Code share the same target answer while instantiating distinct reasoning forms.

Family	Proposed task state	Solver target	Execution check
Input recovery (abduction)	Executable function $f$ and an input $x$ ; execution supplies $y=f(x)$	Recover $\hat{x}$ from $(f,y)$	Check $f(\hat{x})=y$
Output prediction (deduction)	Executable function $f$ paired with input $x$	Predict $\hat{y}$ from $(f,x)$	Compare $\hat{y}$ with $f(x)$
Function induction	Inputs for a latent executable function; execution supplies visible and hidden I/O pairs	Synthesize $\hat{f}$ from visible pairs	Run $\hat{f}$ on every hidden pair

Appendix

Table 6: Three executable task families, each instantiated as a proposer and a solver stream.

Group	Implemented coordinates
Code structure	Python AST depth; branch, loop, nested-loop, call, and function counts; recursion flag; expression depth; and list/dict/set/tuple counts. Non-Python tasks are represented by the remaining structural and semantic coordinates.
Prompt artifacts	Code-block, test/assert, stub, I/O-example, bug, and constraint cues; word and nonempty-line counts; input, output, and pattern cues.
Task semantics	One-hot family and subtype; family-specific deduction-path, abduction-cause, or induction-pattern indicators; input-contract or output-state indicators; reference-text presence.
Verifier shape	One-hot input, output, error, and function targets together with the corresponding contract, trace, error-surface, and function-check burden indicators.
General/behavior	General semantic coordinates: text length, code presence, and constraint count. For every UID: rolling success mean, success variance, response-length mean, and a scalar structural score complementing the supplied coordinates.

Appendix

Table 7: Feature groups used by the rollout-clustering implementation.

Difficulty	direct	short	long	code
easy	$+0.2$	$+0.1$	$-0.1$	$-0.1$
medium	$0.0$	$+0.2$	$+0.1$	$0.0$
hard	$-0.3$	$0.0$	$+0.2$	$+0.3$

Appendix

Table 8: Mode utility

U(d,m)

used by MATE-PPO.

Difficulty	direct	short	long	code
easy	16–128	96–384	384–1536	128–768
medium	32–256	160–768	768–3072	256–1536
hard	64–512	256–1024	1024–4096	512–3072

Appendix

Table 9: Initial reward bands

B^{-}_{d,m}

–

B^{+}_{d,m}

in tokens, computed from the 6,144-token controller scale.

Model	Variant	HE +	MBPP +	LCB	CAvg	A24	A25	AMC	M500	Min.	Oly.	MAvg	GSM8K	M-Pro	AVG
Qwen2.5-Coder-3B [ 9 ]	–	67.1	65.9	20.0	51.0	3.3	3.3	20.0	51.0	18.4	16.6	18.8	69.4	38.3	44.4
AZR-Coder-3B [ 31 ]	+AZR	71.3	69.0	24.4	54.9	3.3	3.3	37.5	62.0	26.1	27.0	26.5	71.3	40.3	48.3
MORE -Coder-3B	+MORE (Ours)	72.9	68.8	25.1	55.6	3.3	6.7	38.6	63.1	27.6	28.7	28.0	73.3	42.5	49.9
Qwen2.5-14B-Base [ 19 ]	–	78.0	66.7	21.7	55.5	6.7	3.3	35.0	66.2	28.3	32.4	28.7	84.0	40.1	52.1
AZR-14B [ 31 ]	+AZR	70.7	68.8	35.2	58.2	10.0	20.0	62.5	76.2	40.4	42.5	41.9	85.2	41.9	56.8
MORE -14B	+MORE (Ours)	75.4	68.5	37.3	60.4	10.0	16.7	65.7	75.1	42.8	42.9	42.2	87.6	42.3	58.1

Appendix

Table 10: Detailed out-of-distribution benchmark results for 3B and 14B scaling experiments. CAvg is the mean of HumanEval + , MBPP + , and LiveCodeBench v1–v5; MAvg is the mean of the six mathematical reasoning benchmarks; AVG is the mean of CAvg, MAvg, GSM8K, and M-Pro.

Figure 7: Training-time rollout difficulty-prior proportions. Easy, medium, and hard denote the online difficulty tiers assigned before rollout generation. Allocation is logged every 10 updates through step 50 and at selected 50-step checkpoints thereafter.

Figure 8: Mean prompt-plus-response token count on the in-distribution validation stream. Base-7B traces are dashed and Coder-7B traces are solid.

Figure 9: GPU-normalized total-token rate, total_tok_per_gpu_s , for matched Base-7B and Coder-7B runs. The upper row reports training measurements with the piecewise step scale used in Figure 5 ; the lower row reports equally spaced validation measurements.

Figure 10: Test-time mode distributions across held-out benchmarks. Each bar gives the percentage of generated mode tokens for one evaluation suite; A24/A25 denote AIME 2024/2025, M500 denotes MATH-500, Min. denotes Minerva Math, Oly. denotes OlympiadBench, and M-Pro denotes MMLU-Pro.

Figure 11: CruxEval-O (sample 8): Four-mode response contracts. A shared program-output task is expressed as an answer-only response, a compact condition check, an expanded execution trace, and an executable evaluation.

Figure 12: MATH-500 (prealgebra problem 1622): Four-mode response contracts. A shared geometry task is expressed through an answer-only response, a compact numerical derivation, an expanded symbolic derivation, and an executable calculation.

Policy	Code Avg		Math Avg		GSM8K		MMLU-Pro
	Acc. $\uparrow$	RLA $\downarrow$	Acc. $\uparrow$	RLA $\downarrow$	Acc. $\uparrow$	RLA $\downarrow$	Acc. $\uparrow$	RLA $\downarrow$
Fixed direct	49.4	77.4	33.7	537.5	60.7	43.8	37.5	153.3
Fixed short_cot	51.2	103.7	37.4	1,216.8	67.4	87.7	43.2	254.6
Fixed long_cot	61.6	405.8	43.1	4,762.3	79.1	122.4	46.9	435.7
Fixed code	55.3	156.5	39.2	2,066.1	70.9	91.6	41.3	269.0
Adaptive MORE -Coder-7B	62.8	191.7	42.8	3,015.9	77.8	105.1	46.0	303.4

Appendix

Table 11: Fixed-mode and adaptive evaluation of MORE -Coder-7B. Fixed policies apply the indicated reasoning contract to every instance. Accuracy is reported in percentage points; Response Length Avg (RLA) is the mean number of generated response tokens, where lower is more efficient. The best and second-best accuracy in each suite are shown in bold and underlined , respectively.

Figure 13: Terminal-return telemetry for the full MORE -Coder-7B trajectory. Values at steps 50–400 are means across the six proposer–solver streams. (a) Contributions of $R_{\mathrm{acc}}$ , $R_{\mathrm{BD}}$ , $R_{\mathrm{MF}}$ , and $R_{\mathrm{MB}}$ . (b) Raw and clipped budget-deviation telemetry; their overlap denotes values within the active clipping interval.

Figure 14: Task-construction proxies for the full MORE -Coder-7B trajectory. Structural complexity and mean edit distance are reported for the input and output generators; answer diversity additionally reports the function stream’s input- and output-answer diversity.

Figure 15: Mode-conditioned budget calibration for the full MORE -Coder-7B trajectory. (a) Dynamic $B^{-}$ (dashed) and $B^{+}$ (solid) bands for each reasoning mode. (b) Fraction of rollouts within the active band. (c) Mode-wise under-band and over-band deviation ratios. All traces are means across the six proposer–solver streams at steps 50–400.

Figure 16: Mode allocation and verifier success by proposer–solver stream for MORE -Coder-7B. Each cell is the mean value over checkpoints 50–400. The left matrix gives the allocation ratio of each reasoning mode; the right matrix gives the corresponding verifier success rate.

Adaptive Reasoning in Language Models

See all 31 papers

INFUSER: Influence-Guided Self-Evolution Improves Reasoning
Jun 8, 2026Siyu Chen, Miao Lu, Beining Wu +7RL for Language Model Reasoning Group Relative Policy Optimization
Yale University · Stanford University · University of Chicago +2
Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training
Sep 28, 2026Yuyang Deng, Yu Wang, Jiayun WangSelf-Training for Language Models Large Language Model-Guided Optimization
Accenture, Center for Advanced AI · Georgia Institute of Technology
e1: Learning Adaptive Control of Reasoning Effort
Oct 30, 2025Michael Kleinman, Matthew Trager, Alessandro Achille +2Efficient Language Model Reasoning Cost-Aware Inference
AWS Agentic AI

Jun 8, 2026J/K move · Enter open · S save

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen+6

Yale University · Stanford University · University of Chicago+2

RL for Language Model Reasoning Group Relative Policy Optimization Curriculum Learning Language Model Self-Improvement

Open paper AskPDF arXiv

Self-evolution offers a scalable path to stronger reasoning: a pretrained language model improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

Abstract

Adaptive Reasoning in Language Models

Explore similar work

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

e1: Learning Adaptive Control of Reasoning Effort