ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
Authors: Yu Li, Rui Miao, Zhengling Qi, Tian Lan
Organizations: Department of Electrical and Computer Engineering, George Washington University · Department of Mathematical Sciences, University of Texas at Dallas · School of Business, George Washington University
The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at https://github.com/Skylanding/ARISE.
Figures & tables
Figure 1: Overview of ARISE on Qwen3-4B, showing performance across seven benchmarks, training reward dynamics on DeepScaleR, skill utilization over training, and accuracy gain versus token overhead relative to GRPO.
Figure 2: Overview of ARISE. The shared policy πθ operates as both Skills Manager and Worker. Before each rollout, the manager scores cache entries via conditional log-probability and injects the selected skill into the prompt ( Download ). After reward computation, an additional rollout OG+1 distills successful solutions into a structured skill document ( Upload ). The two-tier library consists of a compact cache (active pool for selection) and a larger reservoir (archive for future promotion), maintained by five operations: Add , Update , Evict , Load , and Delete .
In-Distribution Competition
Out-of-Distribution Omni-MATH
Method
AMC 23
AIME 24
AIME 25
Algebra
Num.Thy
Combin.
Geometry
Avg. †
Qwen3-4B-Instruct-2507
Baseline
69.6
53.6
45.6
30.1
21.2
13.0
20.5
21.2
GRPO Family (Vanilla)
GRPO ( Guo et al., 2025 )
72.9
54.1
46.5
33.5
24.3
15.1
22.8
23.9
Dr.GRPO ( Liu et al., 2025 )
72.8
54.3
46.6
33.8
24.1
15.3
22.6
23.9
Table 1: Main results on mathematical reasoning benchmarks. We report average Pass@1 accuracy (%) over 32 independent runs. All methods use G=8 and are trained on the DeepScaleR dataset. Bold indicates the best per column within each base model. Shaded rows represent our proposed method, with subscripts indicating absolute change relative to the GRPO baseline.
Figure 4
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Symbol
Value
Skill Selection (Manager)
Softmax temperature
σ
1.0
ε -greedy exploration rate
ε
0.1
Confidence gate threshold
δ
0.35
Max skill tokens for scoring
—
128
Skill Library
Appendix
Table 3: ARISE-specific hyperparameters. Parameters above the double rule are introduced by our method; those below are inherited from the GRPO training infrastructure.
Field
Max Length
Description
skill_name
40 chars
Snake_case identifier; must be non-empty
problem_type
—
One of: algebra, geometry, combinatorics, number_theory, calculus, general
key_insight
160 chars
Core mathematical idea explaining why the approach works
method
3 steps
2–3 procedural steps (each ≤ 100 chars)
check
100 chars
Verification procedure; defaults to “Substitute back to verify”
Total document
Hard-capped at 220 characters
Appendix
Table 4: Skill document JSON schema with field-level constraints.
Figure 4: Example of a generated skill document following the uniform schema.
Skill Name
Problem Type
Key Insight
equation_setup
Algebra
Translate word-problem quantities into variables and equations before solving
modular_arithmetic_check
Number Theory
Reduce expressions modulo small primes to constrain or verify integer solutions
case_enumeration
General
Systematically split into exhaustive cases and verify each independently
symmetry_exploitation
General
Identify and leverage algebraic or geometric symmetry to simplify the problem
extremal_principle
General
Consider boundary or extremal configurations to establish bounds or find optima
Appendix
Table 5: Seed skills used to initialize the cache at the beginning of training.
Figure 5: Prompt template used for the skill generation rollout OG+1 . Up to two successful traces are included, each truncated to 400 characters. The generation uses temperature 0.7, top- p 0.95, and a maximum of 192 new tokens.
Skill Used
Task Correct
rtask
rskill
R
No
No
0
0
0 (incorrect)
No
Yes
1
0
1 (correct, unaided)
Yes
No
0
0
0 (incorrect despite skill)
Yes
Yes
1
1
2 (correct with skill)
Appendix
Table 6: Reward assignment by skill usage and task correctness. The bonus rskill is granted only when the agent both uses a selected skill and solves the problem correctly, preventing the policy from being reinforced for skill usage that does not improve reasoning.
Confidence gate δ
0.15
0.25
0.35
0.45
0.55
Accuracy
55.2
55.9
56.4
56.1
55.0
Skill Use (%)
89.3
81.6
73.0
58.2
41.7
Exploration ε
0.0
0.05
0.10
0.20
0.30
Accuracy
55.7
56.0
56.4
55.8
54.9
Cache size Cc
5
10
20
50
—
Accuracy
55.5
56.4
56.1
55.3
—
Appendix
Table 7: Hyperparameter sensitivity on AIME 2024 (Qwen3-4B, Pass@1 %). Default values are underlined . Each row varies a single parameter while holding the rest at default.
Stage
Skill Name
Type
Key Insight
Seed
equation_setup
Algebra
Translate word-problem quantities into variables and equations before solving
Seed
case_enumeration
General
Systematically split into exhaustive cases and verify each independently
Step 1k
modular_residue_analysis
Num. Theory
Reduce expressions modulo n to constrain integer solutions before full computation
Step 1k
generating_function_setup
Combin.
Encode a counting sequence as polynomial coefficients and extract the target term
Step 2k
vieta_root_reconstruction
Algebra
Use Vieta’s formulas to express symmetric functions of roots without solving the polynomial
Step 2k
angle_chasing_with_inscribed
Geometry
In cyclic quadrilaterals, the inscribed angle theorem converts arc relations to angle equations
Appendix
Table 8: Representative skill documents at different training stages, illustrating the progression from generic heuristics to domain-specific techniques.
Method
Prompt
Response
Total
Overhead
GRPO
312
2,847
3,159
—
DAPO
312
2,903
3,215
+1.8%
GSPO
312
2,871
3,183
+0.8%
EvolveR + GRPO
489
2,921
3,410
+7.9%
ARISE + GRPO
378
2,876
3,254
+3.0%
Appendix
Table 9: Token overhead comparison on AIME 2024 (Qwen3-4B). Overhead is computed as (total tokensmethod−total tokensGRPO)/total tokensGRPO×100% , where total tokens = prompt + response.
Component
GRPO
ARISE
Solution rollouts ( O1,…,OG )
38.2s
39.1s
Reward computation
0.4s
0.5s
Advantage estimation & gradient update
12.6s
12.8s
Skill scoring & selection (Phase II)
—
2.4s
Skill generation ( OG+1 )
—
4.8s
Library maintenance
—
0.1s
Appendix
Table 10: Per-step wall-clock time breakdown on 8 × A100-80GB (Qwen3-4B, G=8 ). Percentages are relative to GRPO baseline.
Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87%, 2.16%, and 2.53% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.
Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
Yangzhen Wu, Shanda Li, Zixin Wen +5
UC Berkeley · Carnegie Mellon University · ByteDance Seed
Reinforcement learning is critical to improving large reasoning models, but its success relies heavily on verifiable rewards (RLVR), making it hard to use in open-ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimizing solely toward the final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning), and we extend traditional RLVR to open-ended settings. We introduce Structure-Aware Reinforcement Learning (SARL), a label-free framework that constructs per-response reasoning maps from intermediate thinking steps and rewards their reasoning topology. SARL shifts supervision from destination to path, encouraging reasoning trajectories that are both locally coherent and globally efficient. On verifiable math tasks, SARL outperforms prior label-free RL baselines and even exceeds RL methods with ground truth supervision, with average gains of +9.1% under PPO and +11.6% under GRPO across four math benchmarks, with particularly large improvements on AIME25 (+35.5% with PPO and +44.7% with GRPO). On non-verifiable open-ended tasks, SARL achieves average gains of +34.6% under PPO and +30.4% under GRPO on WildBench across five task categories, outperforming prior label-free RL methods and DPO, which relies on additional preference labels. Beyond strong performance, SARL exhibits substantially lower KL divergence and higher policy entropy, indicating more stable and exploratory training dynamics. Code and data are available at Code Link.