LLM-guided Planar Loco-Manipulation for Quadruped Robots

Georges Jetti; Michael Khayyat; Jean-Pierre Sleiman; Stefano Arrigoni; Francesco Braghin

cs.ROOct 29, 2026

LLM-guided Planar Loco-Manipulation for Quadruped Robots

Authors: Georges Jetti, Michael Khayyat, Jean-Pierre Sleiman, Stefano Arrigoni, Francesco Braghin

Organizations: Department of Mechanical Engineering, Politecnico di Milano, 20156 Milan, Italy. · Robotics and AI Institute, USA · Flexion Robotics

Abstract

Robotic navigation typically assumes a collision-free route to the goal exists; in cluttered environments with movable obstacles, however, reaching the goal may require physical interaction. Classical solutions encode these behaviors through geometric rules and task-specific logic, while learning-based approaches often constrain the decision space to low-dimensional motion commands or predefined action sets. We instead consider a broader decision space, jointly selecting the skill, relevant object, and geometric target. Our language-conditioned task planner, fine-tuned with LoRA on synthetic data, maps natural-language tasks and a structured scene representation to variable-length sequences of parameterized actions, executed by low-level reinforcement-learning policies. The planner learns the geometric relations underlying interactive navigation and outperforms a classical planner and a prompted frontier model on interactive navigation tasks. Comparisons with models trained from scratch indicate that composition does not require pretraining when the representation is structured, whereas pretraining yields more reliable geometric targets. At the low level, we introduce an end-to-end whole-body planar pushing policy that achieves a 93.3% success rate in simulation, versus 83.5% for a hierarchical baseline, while reducing completion time by more than 60%. On hardware, the hierarchy yields the correct skill composition in 44/45 trials and end-to-end success in 34/45. Project website: https://anonymousprojects-dev.github.io/LLM-guided-LocoManipulation-Quadrupeds/

Figures & tables

Fig. 1 : Deployment pipeline of the proposed framework on the quadruped robot. The planner receives observation $o^{\prime}_{t}$ consisting of a task instruction specified by the user, scene states such as robot and object pose, along with object parameters, and the previous skill that was applied on the environment. The planner, trained on predicting the remaining action sequence to achieve the task objective, applies only the first action in the sequence, and is queried again once that action is complete. Thus, our task planner is an event-driven task planner . Blue robot instances signify that a manipulation policy was used in that stage, whereas Red signifies a navigation policy. Dashed lines represent data flow, whereas solid lines represent decision flow.

Quantity	Description	Dim	Noise
$\mathbf{o_{t}}$
${}^{R}\!\mathbf{p}_{RG}$	Goal-Robot relative position w.r.t. Robot	3	–
$\!\mathbf{\psi}_{RG}$	Goal-Robot relative heading	1	–
$\Delta\mathbf{q}_{j}$	Joint position offsets from default	12	$\mathcal{U}(-0.01,0.01)$
$\Delta\mathbf{\dot{q}}_{j}$	Joint velocities from default	12	$\mathcal{U}(-1.0,1.0)$
$\mathbf{a}_{t-1}$	Previous actions	12	–

TABLE I : Observations for the actor (

o_{m}

) and critic (

o_{m},o_{m}^{\text{pr}}

) for the manipulation policy.

Parameters	Training Range
Mass $(\mathrm{kg})$	$\mathcal{U}(1,\,6)$
$\Delta$ CoM position (cm)	$\mathcal{U}(-10.0,\,10.0)$
Static Friction	$\mathcal{U}(0.5,\,1.2)$
Dynamic Friction	$\mathcal{U}(0.6,\,1.0)$
Object type	[Box, Cylinder]
Object diameter $(\mathrm{m})$	$\mathcal{U}(0.3,\,0.6)$

TABLE II : Training parameters in simulation.

Panel of Figure 4 — Fig. 2 : Emergent Gait Cycle for (a) HRL , and (b) E2E Manipulation Policies. Results were obtained from simulation.

Panel of Figure 5 — Fig. 3 : Comparison of body orientation, (a) Roll , and (b) Pitch for a run where the robot is distant from the object. Blue and orange dashed lines denote the first instant of contact with the object for E2E and HRL, respectively. The manipulate object is a box with an edge length of 40 [cm].

Method	Success Rate (%)	Episode Time (s)	Fall Over (%)	Energy (J)
E2E loco-manipulation	84.50	4.15 $\pm$ 1.46	3.5	1737.55 $\pm$ 704.01
HRL manipulation	83.50	10.80 $\pm$ 4.02	0.3	1813.61 $\pm$ 1060.31
Ours	93.30	4.09 $\pm$ 1.31	0.8	1680.80 $\pm$ 678.12

TABLE III : Comparison with Baselines. Compared to the HRL baseline, our method improves Success Rate and Episode Time. Compared to the E2E loco-manipulation baseline, our policy improves success rate while retaining low Episode Time.

Fig. 4 : Hardware comparison of the manipulation policies.

Object	Controller	Position Err. [cm]	Yaw Err. [deg]	Time [s]
Box	HRL	7.53 $\pm$ 3.12	14.20 $\pm$ 7.47	4.57 $\pm$ 0.39
Box	Ours	5.34 $\pm$ 1.53	13.12 $\pm$ 5.10	3.08 $\pm$ 0.18
Cylinder	HRL	13.43 $\pm$ 5.22	19.18 $\pm$ 5.92	3.06 $\pm$ 0.13
Cylinder	Ours	10.37 $\pm$ 2.73	4.99 $\pm$ 2.20	1.88 $\pm$ 0.14
Prism	HRL	19.44 $\pm$ 9.61	12.64 $\pm$ 5.87	11.18 $\pm$ 2.09
Prism	Ours	10.27 $\pm$ 2.68	5.76 $\pm$ 2.31	2.52 $\pm$ 0.44

TABLE IV : Hardware evaluation of the manipulation policies. Values are reported as mean

\pm

standard deviation over five trials per object.

Task Family	Ours	ICL (Qwen2.5-1.5B)	ICL (GPT-5.5)	Transformer (scratch)	Classical planner
Direct navigation	38/40	40/40 §	36/40	35/40	40/40
Object manipulation	10/10	0/10	10/10	3/10	10/10
Interactive navigation	109/120	0/120	94/120	101/120	86/120
Long-horizon interactive navigation	29/30	0/30	18/30	22/30	22/30
Interactive manipulation (shifted)	12/30	0/30 ‡	2/30	0/30	20/30 †
Per-action Latency (ms)	$472.4\pm 15.5$	$953.5\pm 97.1$	$7011\pm 3400^{\lx@paragraphsign}$	$\mathbf{26.9\pm 6.2}$	$318.4\pm 598.0^{*}$

TABLE V : Planner baseline results, represented as (# of successes)/(# of trajectories) for each task family. Results of our planner and the task-specific Transformer are from a single training run; results of additional runs with different training seeds are reported in Appendix F-E .

Family	Case	Evaluation focus	Expected composition	Trials	Correct skill composition	E2E success
Direct navigation	N-1	Simple Navigation	$\mathrm{N}$	5	5/5	5/5
Object manipulation	OM-1	Simple Manipulation	$\mathrm{N}\rightarrow\mathrm{M}$	5	5/5	5/5
Interactive navigation	IN-1	Task composition	$\mathrm{M}\rightarrow\mathrm{N}$	5	4/5	4/5
	IN-2	Object selection	$\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	5/5
	IN-3	Object selection	$\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	5/5
Long-horizon interactive navigation	LH-1	Long-horizon composition	$\mathrm{N}\rightarrow\mathrm{M}\rightarrow\mathrm{N}$	5	5/5	3/5

TABLE VI : Hardware evaluation of the complete loco-manipulation pipeline. Each scenario is repeated five times. Correct composition indicates whether the planner selects the appropriate ordered sequence of skills. End-to-end success requires successful completion of the full physical task.

Panel of Figure 11 — Fig. 5 : Qualitative hardware examples evaluating object selection. In both scenes, the planner must identify and manipulate the appropriate blocking object before navigating toward the goal.

Fig. 6 : Long horizon task execution for LH-1. The robot appropriately navigates to the only movable object in the scene, manipulates it to create enough space to then navigate to the goal.

Panel of Figure 13 — Fig. 7 : Qualitative hardware examples of skill composition. Scene IN-1 requires $\mathrm{M}\rightarrow\mathrm{N}$ , whereas Scene OM-1 requires $\mathrm{N}\rightarrow\mathrm{M}$ . In Scene IM-2, the planner composes these into the sequence $\mathrm{M}\rightarrow\mathrm{N}\rightarrow\mathrm{M}$ , which is excluded from training.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Reward term	Description	Weight
$rew_{\mathrm{kp}}$	Object–goal keypoint tracking	$20.0$
$rew_{v}$	Object velocity tracking	$15.0$
$rew_{v_{R,x}}$	Positive robot forward velocity	$0.5$
$\lVert\ddot{\mathbf{q}}\rVert^{2}$	Joint acceleration penalty	$-2.5*10^{-7}$
$\lVert\boldsymbol{\tau}\rVert^{2}$	Joint torque penalty	$-2*10^{-4}$
$\lVert\Delta\mathbf{a}_{t-1,t}\rVert^{2}$	Action-rate penalty	$-0.4$

Appendix

TABLE VII : Reward terms used for training the manipulation policy.

Parameter	Value
Rollout steps per update	24
Action clipping	5.0
Actor hidden layers	[512, 256, 128]
Critic hidden layers	[512, 256, 128]
Activation function	ELU
Initial action noise std.	1.0

Appendix

TABLE VIII : PPO and network hyperparameters used for training the low-level policies with RSL-RL.

Panel of Figure 16 — Fig. 8 : End-to-end loco-manipulation policy learning dynamics. Manipulation reward was activated at iteration 0 for the orange curve, iteration 400 for the cyan curve, and iteration 1000 for the red curve.

Panel of Figure 17 — Fig. 9 : Comparison of Energy Consumption between HRL and E2E Loco-manipulation policies. (a) represents the case in which the object distant from the robot , and (b) represents the case in which the object close to the robot . Blue and Orange dashed lines represent the first instant in which the robot is in contact with the object for E2E and HRL, respectively. In these two runs, the object is a cube with an edge length of 40 [cm], weight and CoM position are inside the training distribution. We provide a video showing the behavioral difference between both methods for the first case.

	Run 1		Run 2
Method	Locomotion	Manipulation	Locomotion	Manipulation
E2E	359.56	684.28	136.7	903.3
HRL	218.63	337.42	133.9	984.0

Appendix

TABLE IX : Energy in [J] for locomotion and manipulation phases, for both cases.

Method	Success Rate (%)	Episode Time (s)	Fall Over (%)	Energy (J)
w/o Positive Velocity	20.70	5.86 $\pm$ 1.99	6.0	2446.19 $\pm$ 687.44
w/o $r_{v}$	87.30	3.77 $\pm$ 1.09	7.5	2028.28 $\pm$ 789.67
Ours	93.30	4.09 $\pm$ 1.31	0.8	1680.80 $\pm$ 678.12

Appendix

TABLE X : Ablation Results.

Parameter	Value
Dataset size	16K I/O pairs (8096 trajectories)
Validation dataset size	5 $\%$ of total dataset
LoRA rank	128
LoRA alpha	256
LoRA dropout	0.05
Learning rate	2e-5

Appendix

TABLE XI : LLM fine-tuning details.

Task family	Distribution	Task composition
Direct navigation	ID	$\mathrm{N}$
Object manipulation	ID	$\mathrm{N}\rightarrow\mathrm{M}$
Interactive navigation	ID	$\mathrm{M}\rightarrow\mathrm{N}$
Long-horizon interactive navigation	ID	$\mathrm{N}\rightarrow\mathrm{M}\rightarrow\mathrm{N}$
Interactive manipulation	Shifted	$\mathrm{M}\rightarrow\mathrm{N}\rightarrow\mathrm{M}$

Appendix

TABLE XII : Planner evaluation task families.

\mathrm{(N)}

and

\mathrm{(M)}

denote navigation and manipulation actions, respectively. ID denotes task structures represented in the training distribution, while Shifted denotes the task composition used to evaluate the planner’s ability to compose a novel sequence of actions.

Task Family	LoRA Rank ( $r$ )					No AC
Task Family	$r=32$	$r=64$	$r=128$	$r=256$	$r=512$	$r=512$	$r=128$
Direct navigation	37/40	37/40	38/40	38/40	39/40	37/40	39/40
Object manipulation	10/10	10/10	10/10	10/10	10/10	10/10	10/10
Interactive navigation	98/120	107/120	109/120	109/120	112/120	107/120	105/120
Long-horizon interactive navigation	25/30	29/30	29/30	29/30	29/30	28/30	27/30
Interactive manipulation (shifted)	0/30	0/30	14/30	6/30	3/30	3/30	1/30

Appendix

TABLE XIII : Planner ablation and model-selection results, reported as (# of successes)/(# of trajectories). The shifted row reports results on the 30 validation scenes used for rank selection.

	Ours	Qwen (scratch)	Transformer (scratch)
Pretrained	✓	–	–
Input/output	Text	Text	Structured
Direct navigation	38/40	40/40	35/40
Object manipulation	10/10	8/10	3/10
Interactive navigation	109/120	63/120	101/120
Long-horizon interactive nav.	29/30	23/30	22/30

Appendix

TABLE XIV : Comparison of our planner with the two models trained from scratch, represented as (# of successes)/(# of trajectories). For the shifted task, the last row reports the number of trajectories with the correct skill composition.

	Ours			Transformer (scratch)
Task Family	Run 1	Run 2	Run 3	Run 1	Run 2	Run 3
Direct navigation	38/40	40/40	40/40	35/40	34/40	33/40
Object manipulation	10/10	10/10	10/10	3/10	4/10	5/10
Interactive navigation	109/120	104/120	103/120	101/120	97/120	91/120
Long-horizon interactive nav.	29/30	30/30	29/30	22/30	23/30	22/30
Interactive manipulation (shifted)	12/30	9/30	10/30	0/30	0/30	2/30

Appendix

TABLE XV : Planner results per training run, represented as (# of successes)/(# of trajectories). Runs differ only in the training seed; Run 1 corresponds to the models reported in Table V . For the shifted task, the last row reports the number of trajectories with the correct skill composition.

Robotic Loco-Manipulation

See all 32 papers

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning
May 8, 2026Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2Large Language Model-Based Robot Planning LLM Prompting
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1
SuperNav: An Agentic Navigation System for Any Task in Any Scene
Oct 8, 2026Jinkai Zhang, Jingyi Xu, Yuanhong Yu +5Robot Navigation Visual Robot Navigation
Zhejiang University · Shenzhen University · Causa Robotics
DiPLO: Self-distilled Diffusion Planning for Loco-manipulation
Oct 17, 2026Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand +4Robotic Loco-Manipulation Humanoid Loco-Manipulation
Technical University of Munich (TUM), Munich, Germany. · Institute for Advanced Study (IAS), Technical University of Munich, Munich, Germany. · McGill University, Montreal, Canada. +1

May 8, 2026J/K move · Enter open · S save

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

Karolina Źróbek, Tessa Pulli, Paweł Gajewski, Antonio Galiza Cerdeira Gonzalez+1

IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland+1

Large Language Model-Based Robot Planning LLM Prompting Human-Robot Interaction Robot Motion Planning

Open paper AskPDF arXiv

We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.

Abstract

Robotic Loco-Manipulation

Explore similar work

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

SuperNav: An Agentic Navigation System for Any Task in Any Scene

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation