ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

Yu Li; Rui Miao; Zhengling Qi; Tian Lan

cs.AIMar 17, 2026

ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

Authors: Yu Li, Rui Miao, Zhengling Qi, Tian Lan

Organizations: Department of Electrical and Computer Engineering, George Washington University · Department of Mathematical Sciences, University of Texas at Dallas · School of Business, George Washington University

Abstract

The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at https://github.com/Skylanding/ARISE.

Figures & tables

Figure 1: Overview of ARISE on Qwen3-4B, showing performance across seven benchmarks, training reward dynamics on DeepScaleR, skill utilization over training, and accuracy gain versus token overhead relative to GRPO.

Figure 2: Overview of ARISE. The shared policy $\pi_{\theta}$ operates as both Skills Manager and Worker. Before each rollout, the manager scores cache entries via conditional log-probability and injects the selected skill into the prompt ( Download ). After reward computation, an additional rollout $O_{G+1}$ distills successful solutions into a structured skill document ( Upload ). The two-tier library consists of a compact cache (active pool for selection) and a larger reservoir (archive for future promotion), maintained by five operations: Add , Update , Evict , Load , and Delete .

	In-Distribution Competition			Out-of-Distribution Omni-MATH
Method	AMC 23	AIME 24	AIME 25	Algebra	Num.Thy	Combin.	Geometry	Avg. †
Qwen3-4B-Instruct-2507
Baseline	69.6	53.6	45.6	30.1	21.2	13.0	20.5	21.2
GRPO Family (Vanilla)
GRPO ( Guo et al., 2025 )	72.9	54.1	46.5	33.5	24.3	15.1	22.8	23.9
Dr.GRPO ( Liu et al., 2025 )	72.8	54.3	46.6	33.8	24.1	15.3	22.6	23.9

Table 1: Main results on mathematical reasoning benchmarks. We report average Pass@1 accuracy (%) over 32 independent runs. All methods use

G{=}8

and are trained on the DeepScaleR dataset. Bold indicates the best per column within each base model. Shaded rows represent our proposed method, with subscripts indicating absolute change relative to the GRPO baseline.

Figure 4

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Skill Selection (Manager)
Hyperparameter	Symbol	Value
Softmax temperature	$\sigma$	1.0
$\varepsilon$ -greedy exploration rate	$\varepsilon$	0.1
Confidence gate threshold	$\delta$	0.35
Max skill tokens for scoring	—	128
Skill Library

Appendix

Table 3: ARISE-specific hyperparameters. Parameters above the double rule are introduced by our method; those below are inherited from the GRPO training infrastructure.

Field	Max Length	Description
skill_name	40 chars	Snake_case identifier; must be non-empty
problem_type	—	One of: algebra, geometry, combinatorics, number_theory, calculus, general
key_insight	160 chars	Core mathematical idea explaining why the approach works
method	3 steps	2–3 procedural steps (each $\leq$ 100 chars)
check	100 chars	Verification procedure; defaults to “Substitute back to verify”
Total document		Hard-capped at 220 characters

Appendix

Table 4: Skill document JSON schema with field-level constraints.

Panel of Figure 4 — Figure 4: Example of a generated skill document following the uniform schema.

Skill Name	Problem Type	Key Insight
equation_setup	Algebra	Translate word-problem quantities into variables and equations before solving
modular_arithmetic_check	Number Theory	Reduce expressions modulo small primes to constrain or verify integer solutions
case_enumeration	General	Systematically split into exhaustive cases and verify each independently
symmetry_exploitation	General	Identify and leverage algebraic or geometric symmetry to simplify the problem
extremal_principle	General	Consider boundary or extremal configurations to establish bounds or find optima

Appendix

Table 5: Seed skills used to initialize the cache at the beginning of training.

Panel of Figure 5 — Figure 5: Prompt template used for the skill generation rollout $O_{G+1}$ . Up to two successful traces are included, each truncated to 400 characters. The generation uses temperature 0.7, top- $p$ 0.95, and a maximum of 192 new tokens.

Skill Used	Task Correct	$r^{\mathrm{task}}$	$r^{\mathrm{skill}}$	$R$
No	No	0	0	0 (incorrect)
No	Yes	1	0	1 (correct, unaided)
Yes	No	0	0	0 (incorrect despite skill)
Yes	Yes	1	1	2 (correct with skill)

Appendix

Table 6: Reward assignment by skill usage and task correctness. The bonus

r^{\mathrm{skill}}

is granted only when the agent both uses a selected skill and solves the problem correctly, preventing the policy from being reinforced for skill usage that does not improve reasoning.

Confidence gate $\delta$	0.15	0.25	0.35	0.45	0.55
Accuracy	55.2	55.9	56.4	56.1	55.0
Skill Use (%)	89.3	81.6	73.0	58.2	41.7
Exploration $\varepsilon$	0.0	0.05	0.10	0.20	0.30
Accuracy	55.7	56.0	56.4	55.8	54.9
Cache size $C_{c}$	5	10	20	50	—
Accuracy	55.5	56.4	56.1	55.3	—

Appendix

Table 7: Hyperparameter sensitivity on AIME 2024 (Qwen3-4B, Pass@1 %). Default values are underlined . Each row varies a single parameter while holding the rest at default.

Stage	Skill Name	Type	Key Insight
Seed	equation_setup	Algebra	Translate word-problem quantities into variables and equations before solving
Seed	case_enumeration	General	Systematically split into exhaustive cases and verify each independently
Step 1k	modular_residue_analysis	Num. Theory	Reduce expressions modulo $n$ to constrain integer solutions before full computation
Step 1k	generating_function_setup	Combin.	Encode a counting sequence as polynomial coefficients and extract the target term
Step 2k	vieta_root_reconstruction	Algebra	Use Vieta’s formulas to express symmetric functions of roots without solving the polynomial
Step 2k	angle_chasing_with_inscribed	Geometry	In cyclic quadrilaterals, the inscribed angle theorem converts arc relations to angle equations

Appendix

Table 8: Representative skill documents at different training stages, illustrating the progression from generic heuristics to domain-specific techniques.

Method	Prompt	Response	Total	Overhead
GRPO	312	2,847	3,159	—
DAPO	312	2,903	3,215	+1.8%
GSPO	312	2,871	3,183	+0.8%
EvolveR + GRPO	489	2,921	3,410	+7.9%
ARISE + GRPO	378	2,876	3,254	+3.0%

Appendix

Table 9: Token overhead comparison on AIME 2024 (Qwen3-4B). Overhead is computed as

(\text{total tokens}_{\text{method}}-\text{total tokens}_{\text{GRPO}})\,/\,\text{total tokens}_{\text{GRPO}}\times 100\%

, where total tokens = prompt + response.

Component	GRPO	ARISE
Solution rollouts ( $O_{1},\ldots,O_{G}$ )	38.2s	39.1s
Reward computation	0.4s	0.5s
Advantage estimation & gradient update	12.6s	12.8s
Skill scoring & selection (Phase II)	—	2.4s
Skill generation ( $O_{G+1}$ )	—	4.8s
Library maintenance	—	0.1s

Appendix

Table 10: Per-step wall-clock time breakdown on 8

\times

A100-80GB (Qwen3-4B,

G{=}8

). Percentages are relative to GRPO baseline.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning
May 18, 2026Li Wang, Xiaohan Wang, Xiaodong Lu +5RL for Language Model Reasoning Hierarchical RL
Meituan · Tsinghua University
Learn Hard Problems During RL with Reference Guided Fine-tuning
Mar 1, 2026Yangzhen Wu, Shanda Li, Zixin Wen +5LLM Mathematical Reasoning RL for Language Model Reasoning
UC Berkeley · Carnegie Mellon University · ByteDance Seed
SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
Mar 30, 2026Yifan Wang, Bolian Li, David Cho +3RL for Language Model Reasoning Reinforcement Learning
Purdue University · Texas Instruments

May 18, 2026J/K move · Enter open · S save

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

Li Wang, Xiaohan Wang, Xiaodong Lu, Zipeng Zhang+4

Meituan · Tsinghua University

RL for Language Model Reasoning Hierarchical RL LLM Tool Use LLM Mathematical Reasoning RL for Tool Use

Open paper AskPDF arXiv

Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87%, 2.16%, and 2.53% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.

Abstract

RL for Language Model Reasoning

Explore similar work

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

Learn Hard Problems During RL with Reference Guided Fine-tuning

SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology