Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation

Yutang Lin; Guanya Shi; C. Karen Liu

cs.ROOct 17, 2026

Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation

Authors: Yutang Lin, Guanya Shi, C. Karen Liu

Organizations: Amazon FAR. · Carnegie Mellon University. · Stanford University.

Abstract

Learning a single perceptive policy for diverse humanoid loco-manipulation requires unifying high-level task decisions without losing the state-dependent physical responses that make each interaction succeed. This makes the representation between high-level planning and whole-body control critical: kinematic or contact targets provide compact abstractions but do not directly encode interaction-specific control responses, while directly predicting low-level actions requires the planner to model high-frequency closed-loop feedback. We propose FIBER, a framework that uses the executable latent space of a frozen behavioral foundation model (BFM) as a shared planning representation for heterogeneous interaction skills. Our latent policy projection (LPP) maps independently acquired task-specific policies into this representation by identifying latent commands whose induced actions match their local control responses, enabling lower-rate planning while retaining the BFM's high-frequency state feedback. After projection, we aggregate closed-loop latent rollouts from all skills and train a single perceptive flow planner entirely offline. We further introduce AH-BFM, an adaptive-horizon BFM trained with multi-gamma forward-backward (FB) learning to support both precise control and temporally extended whole-body behavior. We evaluate FIBER on box pickup, door opening, and object grasping in simulation and on a real Unitree G1 humanoid. Across 128 diverse variations per task in simulation, FIBER achieves 42.0% mean success, more than tripling projected latent replay (11.6%) and substantially outperforming kinematic planning (3.5%). Real-world experiments further demonstrate sim-to-real transfer, skill chaining, and conditional steering through the same unified planner.

Figures & tables

Fig. 1 : Overview of Fiber . (a) Heterogeneous interaction experts are acquired through motion imitation, reward shaping, or successor-feature-guided behavior reuse. (b) Action-aligned latent policy projection maps their short-horizon control responses into a frozen adaptive-horizon bfm ; closed-loop rollouts produce executable latent trajectories. (c) These trajectories train a unified perceptive planner entirely offline. Language commands or waypoint conditions are encoded as steering tokens, and a DiT planner predicts future latent trajectories. During deployment, the planner replans from onboard observations while the frozen bfm provides high-frequency feedback.

Method	Pickup Box		Open Door		Grasp Object	Mean
Method	w/ appr.	w/o appr.	w/ appr.	w/o appr.	Grasp Object	Mean
Task-specific policies
Expert Policy †	100.0 $\pm$ 0.0	90.8 $\pm$ 0.5	89.3 $\pm$ 0.2	97.9 $\pm$ 0.8	97.0 $\pm$ 0.8	95.4
lpp w/ SONIC [ 34 ] †	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	83.1 $\pm$ 0.7	88.1 $\pm$ 0.6	5.1 $\pm$ 0.7	29.4
LPP w/ Fixed- $\gamma$ BFM †	75.4 $\pm$ 0.2	80.2 $\pm$ 1.3	61.8 $\pm$ 1.9	72.4 $\pm$ 0.7	69.3 $\pm$ 1.2	68.8
lpp w/ AH-BFM †	81.9 $\pm$ 0.9	87.4 $\pm$ 2.4	83.5 $\pm$ 1.8	97.3 $\pm$ 0.4	52.9 $\pm$ 2.1	72.8

TABLE I : Simulation success rates (%) across heterogeneous loco-manipulation tasks. “w/ appr.” includes the approaching phase, whereas “w/o appr.” initializes the humanoid near the interaction state. † denotes a task-specific policy, and ∗ denotes access to privileged object states instead of egocentric observations. Mean is computed using the “w/ appr.” results.

Fig. 2 : Skill chaining with Fiber . In a single rollout, the policy sequentially opens the door in (i), picks up the box in (ii), and carries it back through the doorway in (iii), demonstrating closed-loop chaining of multiple interaction skills under changing high-level commands.

Fig. 3 : Qualitative real-world behaviors of Fiber . (a) During box carrying, the robot reacts to an unstable grasp by briefly lifting the box and repositioning its hands underneath for more secure support, illustrating high-frequency closed-loop adjustment. (b) After picking up the box, the same policy follows waypoint commands while carrying it; after stumbling, the robot stabilizes on its knees and recovers before continuing, demonstrating interaction-preserving skill chaining, steering, and recovery. (c) For a heavy door, the robot leans its whole body into the interaction to generate sufficient pushing force, showing adaptive whole-body realization of contact-rich tasks.

Method	Pickup Box	Open Door	Grasp Object	Mean
Fiber	6/10	4/10	2/10	40%

TABLE II : Real-world success rates of the unified Fiber planner on the Unitree G1.

Method	Joint-MAE $\downarrow$	BE-EMD $\downarrow$
Method	Joint-MAE $\downarrow$	0.2 s	0.5 s	1 s	2 s
SONIC [ 34 ]	0.1315	–	–	–	–
BFM $\gamma_{0.8}$	0.1107	0.038	0.200	0.495	0.792
BFM $\gamma_{0.98}$	0.1327	0.048	0.213	0.451	0.601
AH-BFM	0.1020	0.049	0.195	0.422	0.590

TABLE III : Ablation of multi-horizon BFM tracking and behavior embedding performance on 100STYLE [ 36 ] .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Hidden width	Hidden layers
Body actor	2048	6
Forward representation	2048	6
Critics	2048	6
Backward encoder	1024	1
Motion discriminator	1024	3
Executable latent dimension	256

Appendix

TABLE IV : BFM architecture used in our implementation.

Parameter	Value
Forward-map learning rate	$3\times 10^{-4}$
Backward-map learning rate	$10^{-5}$
Actor / critic learning rate	$3\times 10^{-4}$
Discriminator learning rate	$10^{-5}$
Minibatch size	1024
Training updates	100,000

Appendix

TABLE V : BFM optimization parameters. Target networks use Polyak updates

\bar{\theta}\leftarrow(1-\rho)\bar{\theta}+\rho\theta

Parameter	Motion expert	SF adapter
Rollout steps / environment	32	32
Learning rate	$3\!\times\!10^{-4}$	$10^{-5}$
Optimization epochs	3	5
Minibatches / rollout	8	4
PPO clipping parameter	0.2	0.2
Discount	0.99	0.99

Appendix

TABLE VI : PPO parameters for the motion expert and the SF-guided composition adapter.

Parameter	Value
Optimizer	AdamW
Learning rate	$3\times 10^{-4}$
Weight decay	$10^{-4}$
Gradient-norm limit	1
Minibatches / rollout	4
Optimization passes / rollout	5

Appendix

TABLE VII : Latent-projector optimization parameters.

Component or parameter	Configuration	Component or parameter	Configuration
Executable body latent	$d_{z}=256$	Transformer width / latent blocks	1024 / 16
Prediction horizon	$1$ s	Attention heads / per-head width	8 / 128
Body / optional hand state	58 / 14	Feed-forward expansion / activation	4 / GELU
Optional hand target	14 per latent step	Attention / feed-forward dropout	0 / 0.2
Depth image / patch shape	$144\times 256$ / $12\times 16$	Conditioning registers	8
Depth range / spatial tokens	$0.12$ – $3.0$ m / 192	Flow-time embedding scale	1000

Appendix

TABLE VIII : Architecture of the unified latent flow planner.

Parameter	Value
Optimizer	AdamW
Learning rate	$2\times 10^{-4}$
Weight decay	$10^{-2}$
Warm-up updates	100
Post-warm-up schedule	Constant
Batch size	256

Appendix

TABLE IX : Offline planner optimization parameters.

Task	Variation	Evaluation range
Box pickup	Box geometry, placement, and mass	Base size $20\times 30\times 40$ cm, independently scaled by $s_{x},s_{y}\in[0.5,1.5]$ and $s_{z}\in[0.5,1.0]$ , yielding sizes from $10\times 15\times 20$ cm to $30\times 45\times 40$ cm. Boxes are placed either on the ground or on supports $0.25$ – $0.50$ m high, with mass sampled from $0.5$ – $1.5$ kg.
Door opening	Door geometry and handle placement	Door width $0.8$ – $1.1$ m and height $1.9$ – $2.2$ m. Handle height is $0.85$ – $0.95$ m above the ground, with lateral inset $0.08$ – $0.15$ m.
Object grasping	Object identity and shape	$128$ distinct object instances are evaluated with four takes per object ( $512$ episodes total), selected through teacher-feasibility screening from $685$ object instances and $2{,}991$ usable demonstrations.

Appendix

TABLE X : Simulation evaluation variations.

Humanoid Loco-Manipulation

Humanoid loco-manipulation studies how robots coordinate walking, balance, and whole-body interaction to move and handle objects.

See all 86 papers

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
Aug 7, 2026Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1Humanoid Loco-Manipulation Latent Action Learning
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy
DiPLO: Self-distilled Diffusion Planning for Loco-manipulation
Oct 17, 2026Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand +4Humanoid Loco-Manipulation Diffusion-Based Planning
Technical University of Munich (TUM), Munich, Germany. · Institute for Advanced Study (IAS), Technical University of Munich, Munich, Germany. · McGill University, Montreal, Canada. +1
I-BFM: Reward-Conditioned Robust Humanoid Interaction via Unsupervised Reinforcement Learning
Oct 5, 2026Ziqi Han, Yitang Li, Junhan Sun +8Representation Learning Reinforcement Learning
Tongji University · Tsinghua University · Zhejiang University +3

Aug 7, 2026J/K move · Enter open · S save

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

Cheng Guo, Mingzhe Ni, Angelo Cangelosi, Arash Ajoudani

Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy

Humanoid Loco-Manipulation Latent Action Learning Robot Skill Learning Model-Based RL

Open paper AskPDF arXiv

Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.

Abstract

Humanoid Loco-Manipulation

Explore similar work

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation

I-BFM: Reward-Conditioned Robust Humanoid Interaction via Unsupervised Reinforcement Learning