Adaptive Aerial Loco-Manipulation Policy Learning via LLM-Assisted Proactive Task Generation
Authors: Jingyu Ru, Xiangyuan Xie, Lixuan Jin, Jian An, Xingming Zhang, Binghao Li, Siyu Wang, Xiang Zhou, Lei Yuan, Hongli Xu, Yang Yu
Organizations: Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China · National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · College of Information Science and Engineering, Northeastern University, Shenyang, China
Loco-manipulation extends the workspace of conventional manipulation by combining mobility with physical interaction, while aerial loco-manipulation further expands this capability into three-dimensional space, enabling robots to reach environments inaccessible to ground-based systems. Moving to an aerial platform, however, gives rise to a demanding control problem, since the vehicle must track trajectories accurately in three dimensions while the motion of the arm and its interaction with the environment continuously shift the center of mass, alter the inertia, and impose force and moment disturbances on the body. Conventional controllers address these effects through explicit models and estimation, whereas learning-based controllers infer dynamics from experience, yet control rewards provide no direct target for the response information retained from history, and fixed task sampling can leave persistent tracking difficulties insufficiently explored. We present ALMA, an aerial loco-manipulation framework for adaptive flight control under external arm commands. ALMA employs a contextual flight policy that adapts from recent measurements and actions, using action-conditioned prediction of future measurement representations to train its context to retain response information relevant to flight control. To expand the experience available for this adaptation, an evolutionary process uses a large language model (LLM) to analyze observed control difficulties and propose new structures of arm motion for subsequent training. In simulation, ALMA reduces position root-mean-square error (RMSE) by up to 24.7% relative to the strongest baseline, while the same policy transfers zero-shot to the aerial manipulator and reduces hover position RMSE by 79.6% during periodic arm motion relative to the real-world baseline.
Figures & tables
Figure 1: Overview of ALMA. Predictive supervision and LLM-assisted task generation train the contextual flight policy, while only the policy is retained for deployment.
Figure 2: Representative flight-reference trajectories in the main benchmark. From left to right: Figure-eight, Polynomial, Pentagram, and Zigzag. The benchmark also includes Hover (not shown) and evaluates the Figure-eight and Pentagram trajectories at multiple speeds.
Static Arm Commands
Periodic Arm Commands
Method
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
SR (%) ↑
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
SR (%) ↑
PID Meier and The PX4 Contributors ()
4.514±0.180
0.299±0.178
100
42.479±15.746
0.577±0.178
100
PID–INDI Smeur et al. (2016)
4.230±0.036
0.031±0.006
100
6.770±2.041
0.037±0.016
100
PID+ L1 Wu et al. (2025)
3.893±0.162
0.263±0.179
100
24.654±6.896
0.426±0.166
100
NMPC Sun et al. (2022)
3.082±0.618
0.180±0.078
100
3.315±0.089
0.176±0.096
100
Zhang et al. Zhang et al. (2025)
6.434±2.473
1.279±1.279
83.33
10.794±2.689
2.459±1.221
100
Table 1: Overall Flight Tracking Performance
Variant
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
Feedforward
3.982±0.196
0.798±0.036
GRU Only
4.020±0.153
0.647±0.092
Pred-1
3.989±0.185
0.474±0.033
Pred-3
3.872±0.319
0.538±0.101
Pred-5 (Ours)
3.576±0.167
0.485±0.045
Pred-10
3.824±0.157
0.545±0.028
Table 2: Ablation Study of Predictive Context Learning
OOD Arm Motions
Reserved Family Variants
Base Benchmark
Method
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
Position RMSE (cm) ↓
Yaw RMSE ( ∘ ) ↓
ALMA-Base
2.937±0.313
1.048±0.291
5.839±3.949
6.322±3.523
2.326±0.303
0.641±0.066
Random Evolution
2.807±0.425
1.211±0.439
5.190±2.486
5.543±2.587
2.354±0.205
0.586±0.048
ALMA (Ours)
2.490±0.343
0.935±0.246
3.803±1.439
5.726±2.692
2.148±0.332
0.477±0.049
Table 3: Ablation of Proactive Task Generation
Figure 3: Zero-shot trajectory tracking in real-world experiments with the arm in (a) the home configuration and (b) the fully extended forward configuration.
Figure 4: Real-world hover stabilization under periodic arm commands. Panels (a)–(c) show PID and panels (d)–(f) show ALMA at corresponding stages of the same arm motion. Position RMSE is 3.17 cm for ALMA and 15.55 cm for PID.
Aerial robotics studies the design, perception, control, and coordination of flying robots for navigation, inspection, transport, and physical interaction.
The rapid progress of vision--language models (VLMs) has sparked growing interest in robotic control, where natural language can express the operation goals while visual feedback links perception to action. However, directly deploying VLM-driven policies on aerial manipulators remains unsafe and unreliable since the generated actions are often inconsistent, hallucination-prone, and dynamically infeasible for flight. In this work, we present AERMANI-VLM, the first framework to adapt pretrained VLMs for aerial manipulation by separating high-level reasoning from low-level control, without any task-specific fine-tuning. Our framework encodes natural language instructions, task context, and safety constraints into a structured prompt that guides the model to generate a step-by-step reasoning trace in natural language. This reasoning output is used to select from a predefined library of discrete, flight-safe skills, ensuring interpretable and temporally consistent execution. By decoupling symbolic reasoning from physical action, AERMANI-VLM mitigates hallucinated commands and prevents unsafe behavior, enabling robust task completion. We validate the framework in both simulation and hardware on diverse multi-step pick-and-place tasks, demonstrating strong generalization to previously unseen commands, objects, and environments.
Sarthak Mishra, Rishabh Dev Yadav, Avirup Das +3
Robotics Research Center, IIIT Hyderabad, India · Department of Computer Science, University of Manchester, UK · Newcastle University, UK
Vision-Language-Action (VLA) models such as π0 have demonstrated remarkable generalization across diverse fixed-base manipulators. However, transferring these foundation models to aerial platforms remains an open challenge due to the fundamental mismatch between the quasi-static dynamics of fixed-base arms and the underactuated, highly dynamic nature of flight. In this work, we introduce AirVLA, a system that investigates the transferability of manipulation-pretrained VLAs to aerial pick-and-place tasks. We find that while visual representations transfer effectively, the specific control dynamics required for flight do not. To bridge this "dynamics gap" without retraining the foundation model, we introduce a Payload-Aware Guidance mechanism that injects payload constraints directly into the policy's flow-matching sampling process. To overcome data scarcity, we further utilize a Gaussian Splatting pipeline to synthesize navigation training data. We evaluate our method through a cumulative 460 real-world experiments which demonstrate that this synthetic data is a key enabler of performance, unlocking 100% success in navigation tasks where directly fine-tuning on teleoperation data alone attains 81% success. Our inference-time intervention, Payload-Aware Guidance, increases real-world pick-and-place task success from 23% to 50%. Finally, we evaluate the model on a long-horizon compositional task, achieving a 62% overall success rate. These results suggest that pre-trained manipulation VLAs, with appropriate data augmentation and physics-informed guidance, can transfer to aerial manipulation and navigation, as well as the composition of these tasks.
Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.
Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy