ADM-Planner: LLM-Guided Long-Horizon Planning for Mobile Manipulators with Attention-Enhanced Dynamic Memory

Jiaping Xiao; Pingyuan Ji; Mir Feroskhan

cs.ROSep 24, 2026

ADM-Planner: LLM-Guided Long-Horizon Planning for Mobile Manipulators with Attention-Enhanced Dynamic Memory

Authors: Jiaping Xiao, Pingyuan Ji, Mir Feroskhan

Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore

Abstract

Large language models can decompose mobile-manipulation goals into long action sequences, but the resulting plans remain reliable only while their world context is current. A fixed scene description becomes stale when objects are discovered, moved, or completed while retaining every observation instead produces a growing history with redundant and conflicting state. To resolve this tension, we present an LLM-guided planning framework ADM-Planner with attention-enhanced dynamic memory (ADM). Persistent workspace knowledge is separated from object-centric state, asynchronous observations and action outcomes update that state, and a bounded retriever exposes only the entries that can affect the next decision. The LLM replans when an update invalidates the remaining plan. Across 1,500 task-simulator episodes, the proposed ADM achieved 100% full-task success in the 14-container noisy dynamic setting, compared with 62% for static memory and 97% for unfiltered dynamic memory, while reducing the context-size proxy by 95.8% relative to the latter. In a six-episode live GPT-5 Mini planner, both dynamic memory variants completed every mission, while ADM reduced provider-reported input tokens by 14.4% and mean planner calls from 7.0 to 6.0. A separate 60-trial PyBullet study retained 100% success for ADM, compared with 50% for static memory. Finally, the mobile manipulator with ADM-Planner completed various missions in indoor and outdoor physical experiments while incorporating targets revealed after execution began. The results show that selective state maintenance with ADM, rather than prompt history alone, is a practical basis for long-horizon planning in changing environments. Project page: https://xjp99v5.github.io/ADM-Planner

Figures & tables

Fig. 1: Dynamic aerial-to-ground memory update during a physical recycling mission. Left: the mobile manipulator operates among containers, collection stations, and obstacles in a previously unseen workspace. Upper right: the aerial robot detects the task-relevant can3 during execution. The accepted observation updates mobile manipulator’s memory status and LLM subsequently conducts replanning to deliver can2 to Station A accordingly. Lower right: the map incorporates the updated planning information and generates a new trajectory.

Method	S-6 $\uparrow$	D-10 $\uparrow$	DN-14 $\uparrow$	Context $\downarrow$
No memory [ 4 ]	63	17	11	5,978
Static memory [ 1 ]	100	84	62	3,876
Unfiltered dynamic [ 16 ]	100	96	97	53,049
ADM (ours)	100	98	100	2,212
Oracle state	100	98	100	2,576

TABLE I: Task-level success over 100 paired seeds per scenario. Context is the mean proxy for Dynamic-noisy-14.

Fig. 4: Rigid-body validation over 20 paired seeds per method: (a) full-task success, (b) completion, (c) successful-trial time, (d) successful-trial path length, (e) stale-target visits, and (f) context-size proxy. Error bars in (a) are Wilson 95% confidence intervals. (g)–(l) Time-sequential views of a representative attention-memory rollout (seed 6), progressing from initialization to six completed deliveries over 166.8 s of recorded video.

Method	Success (%) $\uparrow$	Completion (%) $\uparrow$	Context $\downarrow$
Static memory	50	87.5	1,828
Unfiltered dynamic	100	100.0	3,976
ADM (ours)	100	100.0	1,035

TABLE II: PyBullet validation over 20 paired seeds per method.

Fig. 5: Physical recycling experiments. From top to bottom, the rows show the narrow-gate scenario, cluttered scenario, unstructured daytime scenario, and nighttime scenario. Every run delivered five containers to two stations with dynamic memory updates.

Explore similar work

Sep 11, 2026cs.RO

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.

Sizhe Zhao, Haozhe Xie, Weiyu Zhao +5

Harbin Institute of Technology, China · Nanyang Technological University, Singapore · Shandong University, China +1

Efficient World-Action Model Robot Policies Classical Planning Episodic Memory Planning World Models+1

Ask PDF

Sep 29, 2026cs.RO

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback

Long-horizon robotic tasks are vulnerable to unexpected environmental changes that can render planned actions ineffective or unsafe. To address this, robots must detect such changes as they occur, interpret their impact, and adjust their actions accordingly. Traditional rule-based decision-making pipelines are brittle in open-world conditions, as they are hand-tuned for specific scenarios and lack generalization. Vision-Language Models (VLMs) offer a promising alternative as they combine broad world knowledge with unified visual--text reasoning, enabling them to generalize across diverse scenarios and generate accurate, grounded task plans. However, for effective deployment in dynamic real-world settings, VLMs must be embedded into frameworks capable of handling uncertainty and environmental changes. Existing frameworks broadly address this reactively, triggering replanning only after execution failures or post-task checks, risking failed actions. Some methods verify conditions before actions, but these discrete checks miss changes occurring during execution. To address this, we present ProAct-VLM, an adaptive, physically grounded task planning framework that integrates VLMs within a real-time perception--feedback loop. ProAct-VLM continuously monitors the environment and re-plans as soon as relevant changes are detected, enabling adaptation before failure occurs. Evaluations against multiple baselines and across different VLM backbones show that our framework improves both success rates and efficiency in dynamic, long-horizon manipulation tasks. Project page: https://github.com/moured/ProAct-VLM

Ahmed Nader Ahmed, Omar Moured, Mughni Irfan Mohammed Abdul +2

Khalifa University, Abu Dhabi, United Arab Emirates. · Sereact GmbH, Stuttgart, Germany.

Long-Horizon Manipulation Perception Feedback

Ask PDF

Apr 23, 2026cs.RO

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip

Isabella Liu, An-Chieh Cheng, Rui Yan +7

University of California, San Diego · NVIDIA

Long-Horizon Manipulation Robot State Long-Horizon Task Planning Planning Keypoint Detection

Ask PDF