LLM-guided Planar Loco-Manipulation for Quadruped Robots

Georges Jetti; Michael Khayyat; Jean-Pierre Sleiman; Stefano Arrigoni; Francesco Braghin

cs.ROOct 29, 2026

LLM-guided Planar Loco-Manipulation for Quadruped Robots

Authors: Georges Jetti, Michael Khayyat, Jean-Pierre Sleiman, Stefano Arrigoni, Francesco Braghin

Organizations: Department of Mechanical Engineering, Politecnico di Milano, 20156 Milan, Italy. · Robotics and AI Institute, USA · Flexion Robotics

Abstract

Robotic navigation typically assumes a collision-free route to the goal exists; in cluttered environments with movable obstacles, however, reaching the goal may require physical interaction. Classical solutions encode these behaviors through geometric rules and task-specific logic, while learning-based approaches often constrain the decision space to low-dimensional motion commands or predefined action sets. We instead consider a broader decision space, jointly selecting the skill, relevant object, and geometric target. Our language-conditioned task planner, fine-tuned with LoRA on synthetic data, maps natural-language tasks and a structured scene representation to variable-length sequences of parameterized actions, executed by low-level reinforcement-learning policies. The planner learns the geometric relations underlying interactive navigation and outperforms a classical planner and a prompted frontier model on interactive navigation tasks. Comparisons with models trained from scratch indicate that composition does not require pretraining when the representation is structured, whereas pretraining yields more reliable geometric targets. At the low level, we introduce an end-to-end whole-body planar pushing policy that achieves a 93.3% success rate in simulation, versus 83.5% for a hierarchical baseline, while reducing completion time by more than 60%. On hardware, the hierarchy yields the correct skill composition in 44/45 trials and end-to-end success in 34/45. Project website: https://anonymousprojects-dev.github.io/LLM-guided-LocoManipulation-Quadrupeds/

Figures & tables

Fig. 1 : Deployment pipeline of the proposed framework on the quadruped robot. The planner receives observation $o^{\prime}_{t}$ consisting of a task instruction specified by the user, scene states such as robot and object pose, along with object parameters, and the previous skill that was applied on the environment. The planner, trained on predicting the remaining action sequence to achieve the task objective, applies only the first action in the sequence, and is queried again once that action is complete. Thus, our task planner is an event-driven task planner . Blue robot instances signify that a manipulation policy was used in that stage, whereas Red signifies a navigation policy. Dashed lines represent data flow, whereas solid lines represent decision flow.

Quantity	Description	Dim	Noise
$\mathbf{o_{t}}$
${}^{R}\!\mathbf{p}_{RG}$	Goal-Robot relative position w.r.t. Robot	3	–
$\!\mathbf{\psi}_{RG}$	Goal-Robot relative heading	1	–
$\Delta\mathbf{q}_{j}$	Joint position offsets from default	12	$\mathcal{U}(-0.01,0.01)$
$\Delta\mathbf{\dot{q}}_{j}$	Joint velocities from default	12	$\mathcal{U}(-1.0,1.0)$
$\mathbf{a}_{t-1}$	Previous actions	12	–

TABLE I : Observations for the actor (

o_{m}

) and critic (

o_{m},o_{m}^{\text{pr}}

) for the manipulation policy.

Parameters	Training Range
Mass $(\mathrm{kg})$	$\mathcal{U}(1,\,6)$
$\Delta$ CoM position (cm)	$\mathcal{U}(-10.0,\,10.0)$
Static Friction	$\mathcal{U}(0.5,\,1.2)$
Dynamic Friction	$\mathcal{U}(0.6,\,1.0)$
Object type	[Box, Cylinder]
Object diameter $(\mathrm{m})$	$\mathcal{U}(0.3,\,0.6)$

TABLE II : Training parameters in simulation.

Panel of Figure 4 — Fig. 2 : Emergent Gait Cycle for (a) HRL , and (b) E2E Manipulation Policies. Results were obtained from simulation.

Panel of Figure 5 — Fig. 3 : Comparison of body orientation, (a) Roll , and (b) Pitch for a run where the robot is distant from the object. Blue and orange dashed lines denote the first instant of contact with the object for E2E and HRL, respectively. The manipulate object is a box with an edge length of 40 [cm].

Method	Success Rate (%)	Episode Time (s)	Fall Over (%)	Energy (J)
E2E loco-manipulation	84.50	4.15 $\pm$ 1.46	3.5	1737.55 $\pm$ 704.01
HRL manipulation	83.50	10.80 $\pm$ 4.02	0.3	1813.61 $\pm$ 1060.31
Ours	93.30	4.09 $\pm$ 1.31	0.8	1680.80 $\pm$ 678.12

TABLE III : Comparison with Baselines. Compared to the HRL baseline, our method improves Success Rate and Episode Time. Compared to the E2E loco-manipulation baseline, our policy improves success rate while retaining low Episode Time.

Fig. 4 : Hardware comparison of the manipulation policies.

Object	Controller	Position Err. [cm]	Yaw Err. [deg]	Time [s]
Box	HRL	7.53 $\pm$ 3.12	14.20 $\pm$ 7.47	4.57 $\pm$ 0.39
Box	Ours	5.34 $\pm$ 1.53	13.12 $\pm$ 5.10	3.08 $\pm$ 0.18
Cylinder	HRL	13.43 $\pm$ 5.22	19.18 $\pm$ 5.92	3.06 $\pm$ 0.13
Cylinder	Ours	10.37 $\pm$ 2.73	4.99 $\pm$ 2.20	1.88 $\pm$ 0.14
Prism	HRL	19.44 $\pm$ 9.61	12.64 $\pm$ 5.87	11.18 $\pm$ 2.09
Prism	Ours	10.27 $\pm$ 2.68	5.76 $\pm$ 2.31	2.52 $\pm$ 0.44

TABLE IV : Hardware evaluation of the manipulation policies. Values are reported as mean

\pm

standard deviation over five trials per object.

Task Family	Ours	ICL (Qwen2.5-1.5B)	ICL (GPT-5.5)	Transformer (scratch)	Classical planner
Direct navigation	38/40	40/40 §	36/40	35/40	40/40
Object manipulation	10/10	0/10	10/10	3/10	10/10
Interactive navigation	109/120	0/120	94/120	101/120	86/120
Long-horizon interactive navigation	29/30	0/30	18/30	22/30	22/30
Interactive manipulation (shifted)	12/30	0/30 ‡	2/30	0/30	20/30 †
Per-action Latency (ms)	$472.4\pm 15.5$	$953.5\pm 97.1$	$7011\pm 3400^{\lx@paragraphsign}$	$\mathbf{26.9\pm 6.2}$	$318.4\pm 598.0^{*}$

TABLE V : Planner baseline results, represented as (# of successes)/(# of trajectories) for each task family. Results of our planner and the task-specific Transformer are from a single training run; results of additional runs with different training seeds are reported in Appendix F-E .

Family	Case	Evaluation focus	Expected composition	Trials	Correct skill composition	E2E success
Direct navigation	N-1	Simple Navigation	$\mathrm{N}$	5	5/5	5/5
Object manipulation	OM-1	Simple Manipulation	$\mathrm{N}\rightarrow\mathrm{M}$	5	5/5	5/5
Interactive navigation	IN-1	Task composition	$\mathrm{M}\rightarrow\mathrm{N}$	5	4/5	4/5
	IN-2	Object selection	$\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	5/5
	IN-3	Object selection	$\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	5/5
Long-horizon interactive navigation	LH-1	Long-horizon composition	$\mathrm{N}\rightarrow\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	3/5

TABLE VI : Hardware evaluation of the complete loco-manipulation pipeline. Each scenario is repeated five times. Correct composition indicates whether the planner selects the appropriate ordered sequence of skills. End-to-end success requires successful completion of the full physical task.

Panel of Figure 11 — Fig. 5 : Qualitative hardware examples evaluating object selection. In both scenes, the planner must identify and manipulate the appropriate blocking object before navigating toward the goal.

Fig. 6 : Long horizon task execution for LH-1. The robot appropriately navigates to the only movable object in the scene, manipulates it to create enough space to then navigate to the goal.

Panel of Figure 13 — Fig. 7 : Qualitative hardware examples of skill composition. Scene IN-1 requires $\mathrm{M}\rightarrow\mathrm{N}$ , whereas Scene OM-1 requires $\mathrm{N}\rightarrow\mathrm{M}$ . In Scene IM-2, the planner composes these into the sequence $\mathrm{M}\rightarrow\mathrm{N}\rightarrow\mathrm{M}$ , which is excluded from training.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Reward term	Description	Weight
$rew_{\mathrm{kp}}$	Object–goal keypoint tracking	$20.0$
$rew_{v}$	Object velocity tracking	$15.0$
$rew_{v_{R,x}}$	Positive robot forward velocity	$0.5$
$\lVert\ddot{\mathbf{q}}\rVert^{2}$	Joint acceleration penalty	$-2.5*10^{-7}$
$\lVert\boldsymbol{\tau}\rVert^{2}$	Joint torque penalty	$-2*10^{-4}$
$\lVert\Delta\mathbf{a}_{t-1,t}\rVert^{2}$	Action-rate penalty	$-0.4$

Appendix

TABLE VII : Reward terms used for training the manipulation policy.

Parameter	Value
Rollout steps per update	24
Action clipping	5.0
Actor hidden layers	[512, 256, 128]
Critic hidden layers	[512, 256, 128]
Activation function	ELU
Initial action noise std.	1.0

Appendix

TABLE VIII : PPO and network hyperparameters used for training the low-level policies with RSL-RL.

Panel of Figure 16 — Fig. 8 : End-to-end loco-manipulation policy learning dynamics. Manipulation reward was activated at iteration 0 for the orange curve, iteration 400 for the cyan curve, and iteration 1000 for the red curve.

Panel of Figure 17 — Fig. 9 : Comparison of Energy Consumption between HRL and E2E Loco-manipulation policies. (a) represents the case in which the object distant from the robot , and (b) represents the case in which the object close to the robot . Blue and Orange dashed lines represent the first instant in which the robot is in contact with the object for E2E and HRL, respectively. In these two runs, the object is a cube with an edge length of 40 [cm], weight and CoM position are inside the training distribution. We provide a video showing the behavioral difference between both methods for the first case.

	Run 1		Run 2
Method	Locomotion	Manipulation	Locomotion	Manipulation
E2E	359.56	684.28	136.7	903.3
HRL	218.63	337.42	133.9	984.0

Appendix

TABLE IX : Energy in [J] for locomotion and manipulation phases, for both cases.

Method	Success Rate (%)	Episode Time (s)	Fall Over (%)	Energy (J)
w/o Positive Velocity	20.70	5.86 $\pm$ 1.99	6.0	2446.19 $\pm$ 687.44
w/o $r_{v}$	87.30	3.77 $\pm$ 1.09	7.5	2028.28 $\pm$ 789.67
Ours	93.30	4.09 $\pm$ 1.31	0.8	1680.80 $\pm$ 678.12

Appendix

TABLE X : Ablation Results.

Parameter	Value
Dataset size	16K I/O pairs (8096 trajectories)
Validation dataset size	5 $\%$ of total dataset
LoRA rank	128
LoRA alpha	256
LoRA dropout	0.05
Learning rate	2e-5

Appendix

TABLE XI : LLM fine-tuning details.

Task family	Distribution	Task composition
Direct navigation	ID	$\mathrm{N}$
Object manipulation	ID	$\mathrm{N}\rightarrow\mathrm{M}$
Interactive navigation	ID	$\mathrm{M}\rightarrow\mathrm{N}$
Long-horizon interactive navigation	ID	$\mathrm{N}\rightarrow\mathrm{M}\rightarrow\mathrm{N}$
Interactive manipulation	Shifted	$\mathrm{M}\rightarrow\mathrm{N}\rightarrow\mathrm{M}$

Appendix

TABLE XII : Planner evaluation task families.

\mathrm{(N)}

and

\mathrm{(M)}

denote navigation and manipulation actions, respectively. ID denotes task structures represented in the training distribution, while Shifted denotes the task composition used to evaluate the planner’s ability to compose a novel sequence of actions.

Task Family	LoRA Rank ( $r$ )					No AC
Task Family	$r=32$	$r=64$	$r=128$	$r=256$	$r=512$	$r=512$	$r=128$
Direct navigation	37/40	37/40	38/40	38/40	39/40	37/40	39/40
Object manipulation	10/10	10/10	10/10	10/10	10/10	10/10	10/10
Interactive navigation	98/120	107/120	109/120	109/120	112/120	107/120	105/120
Long-horizon interactive navigation	25/30	29/30	29/30	29/30	29/30	28/30	27/30
Interactive manipulation (shifted)	0/30	0/30	14/30	6/30	3/30	3/30	1/30

Appendix

TABLE XIII : Planner ablation and model-selection results, reported as (# of successes)/(# of trajectories). The shifted row reports results on the 30 validation scenes used for rank selection.

	Ours	Qwen (scratch)	Transformer (scratch)
Pretrained	✓	–	–
Input/output	Text	Text	Structured
Direct navigation	38/40	40/40	35/40
Object manipulation	10/10	8/10	3/10
Interactive navigation	109/120	63/120	101/120
Long-horizon interactive nav.	29/30	23/30	22/30

Appendix

TABLE XIV : Comparison of our planner with the two models trained from scratch, represented as (# of successes)/(# of trajectories). For the shifted task, the last row reports the number of trajectories with the correct skill composition.

	Ours			Transformer (scratch)
Task Family	Run 1	Run 2	Run 3	Run 1	Run 2	Run 3
Direct navigation	38/40	40/40	40/40	35/40	34/40	33/40
Object manipulation	10/10	10/10	10/10	3/10	4/10	5/10
Interactive navigation	109/120	104/120	103/120	101/120	97/120	91/120
Long-horizon interactive nav.	29/30	30/30	29/30	22/30	23/30	22/30
Interactive manipulation (shifted)	12/30	9/30	10/30	0/30	0/30	2/30

Appendix

TABLE XV : Planner results per training run, represented as (# of successes)/(# of trajectories). Runs differ only in the training seed; Run 1 corresponds to the models reported in Table V . For the shifted task, the last row reports the number of trajectories with the correct skill composition.

Robotic Loco-Manipulation

See all 32 papers

May 8, 2026

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.

Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2

IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1

Large Language Model-Based Robot Planning LLM Prompting Human-Robot Interaction Robot Motion Planning

Ask PDF

Oct 8, 2026

SuperNav: An Agentic Navigation System for Any Task in Any Scene

General-purpose service robots need navigation systems that can handle diverse human requests in unfamiliar environments, combining task generality with scene generality. Some existing methods fine-tune multimodal large language models (MLLMs) to predict navigation actions, making their behavior dependent on the coverage of navigation training data and potentially limiting generalization to new requests and environments. Our key insight is to let the MLLM focus on interpreting requests, understanding scenes, and making decisions while preserving its general-purpose capabilities and delegating motion execution to navigation tools. To realize this idea, we introduce SuperNav, which equips a pretrained MLLM with a specialized agent harness without navigation-specific fine-tuning of the MLLM. Our harness supports these decisions with Navigation Skills, agent-oriented Tools for physical interaction, and task-progress and context management. A unified visual-point interface connects decision-making to motion by allowing the model to specify destinations directly in images and revise its decisions from execution feedback. Together, these components support sustained navigation across different task requirements and environments. SuperNav outperforms four evaluated baselines on instance-level, multi-object, and demand-driven tasks. Category-level evaluation on HM3D and deployment on a real quadruped robot further demonstrate its applicability across environments. Project Page: https://zju3dv.github.io/SuperNav/

Jinkai Zhang, Jingyi Xu, Yuanhong Yu +5

Zhejiang University · Shenzhen University · Causa Robotics

Robot Navigation Visual Robot Navigation Indoor Robot Navigation Agentic AI for Robotics Vision-Language Navigation Service Robotics+2

Ask PDF

Oct 17, 2026

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation

Humanoid loco-manipulation is particularly challenging because it requires coordinating locomotion, balance, and object interaction, making it difficult to learn from limited demonstrations or from scratch through Reinforcement Learning (RL). We introduce DiPLO (self-distilled Diffusion Planning for Loco-manipulation), a framework that combines a self-distilled diffusion planner with a low-level whole-body tracking policy. Our planner is pretrained via supervised learning on motions derived from human-to-humanoid retargeting, while the tracker is trained via RL to execute them. DiPLO centers around two complementary post-training stages. First, we introduce Self-Distillation Diffusion Policy Optimisation (SD:DPO) to remove the planner's dependence on strong task conditioning, which we identify as a crucial ingredient for effective pretraining, as downstream fine-tuning does not recover comparable performance when it is omitted.. We then fine-tune this distilled model with Diffusion-Augmented Proximal Policy Optimization (DA:PPO), a maximum-entropy RL method for diffusion models, to improve task success and align generated plans with the frozen tracker's execution capabilities. Together, these stages enable motion generation through state-history and goal conditioning alone, enabling 99% task success in pick-carry-place tasks, compared to just 45% for the pretrained model. The resulting planner also generates motions for unseen object goals, with further evaluations on kicking and pushing/pulling tasks. Finally, we deploy our pipeline on the hardware, using our DA:PPO planner-tracker on a Unitree G1, achieving repeated successful executions without any real-world fine-tuning.

Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand +4

Technical University of Munich (TUM), Munich, Germany. · Institute for Advanced Study (IAS), Technical University of Munich, Munich, Germany. · McGill University, Montreal, Canada. +1

Robotic Loco-Manipulation Humanoid Loco-Manipulation Loco-Manipulation Diffusion-Based Planning RL for Robotics Self-Distillation

Ask PDF

Abstract

Robotic Loco-Manipulation

Explore similar work

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

SuperNav: An Agentic Navigation System for Any Task in Any Scene

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation