HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents

Hongbo Jin; Rongpeng Zhu; Jiayu Ding; Guibo Luo; Ge Li

cs.AIMar 1, 2026

HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents

Authors: Hongbo Jin, Rongpeng Zhu, Jiayu Ding, Guibo Luo, Ge Li

Organizations: School of Electronic and Computer Engineering, Peking University

Abstract

Large language model (LLM) agents have recently demonstrated strong capabilities in interactive decision-making, yet they remain fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to inefficient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adaptation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms strong prompting and reinforcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and visually grounded environments. Our results show that introducing structured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence.

Figures & tables

Figure 1: A typical case comparsion in WebShop environment and overall performance. (a) Flat-policy architectures easily get lost in irrelevant observations (Context Drift). (b) HiMAC addresses this exploration inefficiency through a bi-level decoupled architecture (Planner and Executor sharing the same parameters $\pi_{\theta}$ ), mapping structured blueprints to executable atomic actions. (c) HiMAC yields consistent state-of-the-art performance against advanced RL reasoning baselines across multiple environments.

Figure 2: Overall pipeline of HiMAC. Phase A (left) optimizes the macro-policy for blueprint generation using GRPO while freezing the micro-policy. Phase B (right) optimizes the micro-policy for atomic action execution conditioned on a fixed blueprint $z^{*}$ . Both phases update the same underlying LLM parameters $\theta$ .

Type	Method	ALFWorld							WebShop
Type	Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-Source Model
PE	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
PE	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
PE	GPT-5	-	-	-	-	-	-	65.4	-	33.7
PE	Claude Sonnet 4.5	-	-	-	-	-	-	82.5	-	38.6

Table 1: Performance on ALFWorld and WebShop. Results are averaged over 3 random seeds. PE refers to prompt engineering. GiGPO

{}_{\text{w/ std}}

denotes using

F_{\text{norm}}=\text{std}

, while GiGPO

{}_{\text{w/o std}}

uses

F_{\text{norm}}=1

Method	Qwen2.5-VL-3B		Qwen2.5-VL-7B		Qwen3-VL-8B
Method	Succ.	Score	Succ.	Score	Succ.	Score
Prompt	14.1	-0.88	25.0	-0.26	21.9	-0.95
GRPO	76.2	4.59	83.9	5.39	82.9	5.24
GiGPO	81.1	4.86	82.8	5.27	78.1	4.67
HiMAC	83.8	5.31	87.5	6.70	90.6	7.32

Table 2: Performance on the Sokoban benchmark. ’Succ.’ and ’Score’ refers to success rate and test score respectively.

Variant	ALFWorld (%)	WebShop Score	WebShop Succ.(%)
HiMAC (Full)	92.1	93.8	84.1
w/o hierarchy (Flat GRPO)	77.6	79.3	66.1
w/o Iterative Co-Evolution	85.3	86.7	74.8
w/o <sub_done>	88.2	90.1	79.8
Random Blueprint	89.7	91.6	81.5

Table 3: Ablation study on ALFWorld and Webshop (Qwen2.5-7B-Instruct).

Benchmark (Target)	Iters to Target $\downarrow$			Final Succ. (%) $\uparrow$
Benchmark (Target)	HiMAC	GiGPO	GRPO	HiMAC	GiGPO	GRPO
ALFWorld (75%)	$\sim$ 110	$\sim$ 115	$\sim$ 150	92.1	90.2	77.6
WebShop (65%)	$\sim$ 220	$\sim$ 230	$\sim$ 380	84.1	72.8	65.7
Sokoban (80%)	$\sim$ 180	$\sim$ 195	$\sim$ 210	87.5	82.8	83.9

Table 4: Comprehensive sample efficiency analysis across benchmarks (7B backbone). We report both the convergence speed (approximate iterations required to reach a target success threshold) and the final converged success rate.

Figure 3: Evolution of HiMAC’s planning and emergent behaviors. For the task "put a candle in toilet": (Left) Early training shows blind exploration due to lacking spatial priors. (Right) Post-convergence, the planner efficiently navigates to the target and develops emergent self-verification.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 429 papers

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning
Jun 9, 2026Juncheng Diao, Zhicong Lu, Peiguang Li +6Hierarchical RL Long-Horizon LLM Agents
1Meituan · University of Chinese Academy of Sciences
A Subgoal-driven Framework for Improving Long-Horizon LLM Agents
Mar 20, 2026Taiyi Wang, Sian Gooding, Florian Hartmann +2Long-Horizon LLM Agents Web Agents
Google DeepMind
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2

Jun 9, 2026J/K move · Enter open · S save

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

Juncheng Diao, Zhicong Lu, Peiguang Li, Yongwei Zhou+5

1Meituan · University of Chinese Academy of Sciences

Hierarchical RL Long-Horizon LLM Agents LLM Context Management LLM Agent Planning RL for LLM Agents

Open paper AskPDF arXiv

While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-horizon agentic tasks. Existing methods have made progress through fine-grained credit assignment to alleviate long-horizon sparse rewards and hierarchical reinforcement learning to decompose tasks and reduce long-term dependency. However, these methods still do not directly address long-context interference, in which continuously growing histories weaken the agent's ability to track the global task state and impair subsequent reasoning and decision-making. Inspired by the way humans handle complex tasks through subgoal decomposition and completed progress summarization, we propose Hierarchical Planning and Information Folding (HIPIF) for long-horizon LLM agent learning. HIPIF trains the agent end-to-end to organize long-horizon execution around explicit subgoals while folding completed subgoal histories to reduce long-context interference. Furthermore, to stabilize subgoal-based planning and execution, HIPIF combines hierarchical reflection and subgoal-oriented process rewards to guide subgoal generation, transition, and execution, without relying on costly auxiliary models or task-specific expert trajectories. Extensive experiments on three publicly available agentic benchmarks demonstrate the validity of our method.

Abstract

Long-Horizon LLM Agents

Explore similar work

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

A Subgoal-driven Framework for Improving Long-Horizon LLM Agents

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents