VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

Aditya Shirwatkar; Satyam Gupta; Shishir Kolathaya

cs.ROMar 15, 2026

VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

Authors: Aditya Shirwatkar, Satyam Gupta, Shishir Kolathaya

Organizations: Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.

Abstract

Perceptive locomotion for legged robots requires anticipating and adapting to complex, dynamic environments. Model Predictive Control (MPC) serves as a strong baseline, providing interpretable motion planning with constraint enforcement, but struggles with high-dimensional perceptual inputs and rapidly changing terrain. In contrast, model-free Reinforcement Learning (RL) adapts well across visually challenging scenarios but lacks planning. To bridge this gap, we propose VIP-Loco, a framework that integrates vision-based scene understanding with RL and planning. During training, an internal model maps proprioceptive states and depth images into compact kinodynamic features used by the RL policy. At deployment, the learned models are used within an infinite-horizon MPC formulation, combining adaptability with structured planning. We validate VIP-Loco in simulation on challenging locomotion tasks, including slopes, stairs, crawling, tilting, gap jumping, and climbing, across three robot morphologies: a quadruped (Unitree Go1), a biped (Cassie), and a wheeled-biped (TronA1-W). Through ablations and comparisons with state-of-the-art methods, we show that VIP-Loco unifies planning and perception, enabling robust, interpretable locomotion in diverse environments.

Figures & tables

Fig. 1 : Conceptual outline of VIP-Loco – The framework learns a compact internal model from vision and proprioception during training, which is then utilized by an infinite-horizon MPC planner at deployment to enable anticipatory, constraint-aware locomotion.

Fig. 2 : Overview of VIP-Loco Framework – The proposed framework consists of two major components: (1) Learning Stage (left) – The internal model includes a GRU cell $g_{\varphi}$ (operating at 10 Hz, updating recurrent memory $h$ at each step) coupled with a CNN-MLP encoder for depth processing, an encoder/dynamics pair for latent state estimation, and reward/value heads. The Expert Actor (50 Hz) receives the imagined rollout $\mathcal{X}$ and hidden state $h$ via stop-gradient. (2) Deployment Stage (right) – a data-driven MPC that uses the learned internal model to iteratively sample and refine trajectories, leveraging vision-based scene understanding to select actions that maximize long-term reward while satisfying kinodynamic constraints.

	Go1 (Quadruped)	Cassie (Biped)	TronA1-W (Wheeled Biped)
Core Locomotion Rewards
Linear Velocity Tracking	$1.5$	$2.5$	$5.0$
Angular Velocity Tracking	$0.5$	$0.5$	$2.5$
Torques Penalty	$1\text{e}^{-7}$	$1\text{e}^{-7}$	$1\text{e}^{-7}$
DoF Acceleration Penalty	$2.5\text{e}^{-7}$	$2.5\text{e}^{-7}$	$2.5\text{e}^{-7}$
Action Rate	$-0.03$	$-0.03$	$-0.03$

TABLE I : Reward scales used for training, grouped into core locomotion and environment-constraint rewards.

a^{(j)}_{0:H-1}\sim\mathcal{N}(\mu_{i-1},\sigma_{i-1}),\hskip 8.50012pt\forall j\in[1,M]

Algorithm 2 Data-Driven

\infty

Horizon MPC (Deployment)

Fig. 3 : Training comparison for Go1 (quadruped) over $5$ seeds – Top: episodic return progression over training iterations. Bottom: average terrain level successfully mastered. VIP-Loco (Variational) achieves the highest and most stable returns, steadily mastering harder terrains (levels $\geq$ 6). WMP performs competitively but asymptotically regresses to easier terrains. VIP-Loco (Consistency) converges earlier and stagnates at lower levels.

Fig. 4 : Comparison of locomotion performance for Go1 (quadruped) across terrains of increasing difficulty and $5$ seeds – The plots show the success rate (top row) and average return (bottom row) for five locomotion methods: HIM-Loco, PIP-Loco, WMP, VIP-Loco (with Consistency Loss), and VIP-Loco (with Variational Loss). Each column corresponds to a different terrain type and the x-axis indicates terrain difficulty level (0 = easiest, 8 = hardest).

Robot	Method	Slopes	Stairs	Climb	Gaps	Crawl	Tilt
Go1 (Quadruped)	VIP-Loco w/o Planning	$\mathbf{28.37\pm 2.59}$	$26.66\pm 2.54$	$22.80\pm 3.29$	$24.80\pm 3.29$	$20.80\pm 2.82$	$16.18\pm 2.54$
Go1 (Quadruped)	VIP-Loco	${27.22\pm 1.34}$	$\mathbf{27.85\pm 3.28}$	$\mathbf{23.26\pm 2.45}$	$\mathbf{26.33\pm 2.38}$	$\mathbf{22.63\pm 3.74}$	$\mathbf{18.78\pm 2.13}$
Cassie (Biped)	VIP-Loco w/o Planning	${22.54\pm 1.83}$	$23.66\pm 3.79$	$21.46\pm 2.46$	$\mathbf{21.34\pm 2.91}$	$\mathbf{18.52\pm 1.91}$	$12.33\pm 1.98$
Cassie (Biped)	VIP-Loco	$\mathbf{23.19\pm 2.94}$	$\mathbf{23.85\pm 1.64}$	$\mathbf{27.42\pm 2.72}$	$20.50\pm 3.06$	$17.69\pm 2.22$	$\mathbf{14.40\pm 3.21}$
TronA1-W (Wheeled Biped)	VIP-Loco w/o Planning	$33.47\pm 1.38$	$31.32\pm 1.22$	$29.39\pm 3.92$	$\mathbf{31.19\pm 2.26}$	$27.55\pm 2.73$	$21.82\pm 2.01$
TronA1-W (Wheeled Biped)	VIP-Loco	$\mathbf{38.29\pm 2.39}$	$\mathbf{34.05\pm 2.31}$	$\mathbf{32.87\pm 1.50}$	$30.22\pm 2.99$	$\mathbf{29.27\pm 3.09}$	$\mathbf{24.03\pm 1.17}$

TABLE II : Average episodic return comparison across six terrains and three robots after

5

k iterations (averaged over

5

seeds).

Panel of Figure 8 — Fig. 5 : Qualitative evaluation for VIP-Loco with planning across three robot morphologies: (a) Go1 (quadruped), (b) Cassie (biped), and (c) TronA1-W (wheeled biped). Each subfigure includes (left) predicted vs. actual CoM height trajectories for Climb and Crawl tasks, and (right) corresponding execution frames. The MPC predictions and actual measurements demonstrate interpretable dynamics modeling, while the snapshots show task-specific stable behaviors across diverse morphologies.

Terrain-Aware Robot Locomotion

Terrain-aware robot locomotion studies how robots sense, predict, and adapt movement to uneven, deformable, or otherwise challenging ground.

See all 148 papers

Beyond Fixed Visual History: Learning Motion-Adaptive Temporal Receptive Fields for Complex-Terrain Quadruped Locomotion
Oct 10, 2026Hao Zhang, Yuping Yan, Yitian Hong +5Quadruped Locomotion Vision-Based Robot Control
Westlake University, Hangzhou, China. · East China University of Science and Technology, Shanghai, China.
ReCo: Response-Consistent Locomotion with Policy-Aware MPC for Legged Manipulation
Oct 1, 2026Kuankuan Sima, Yichao Gao, Chenxi Gu +2Model Predictive Control RL for Legged Locomotion
Department of Electrical and Computer Engineering, National University of Singapore, Singapore 119077, Singapore · Department of Mechanical Engineering, National University of Singapore, Singapore 119077, Singapore
RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion
Mar 11, 2026Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi +3Robot Locomotion Legged Robotics
All authors are with the Humanoids and Human-Centred Mechatronics (HHCM) lab, Istituto Italiano di Tecnologia (IIT), Via San Quirico 19d, 16163 Genova · Andrea Patrizi is also with the Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova, Via All’Opera Pia 13, 16145 Genova.

Oct 10, 2026J/K move · Enter open · S save

Beyond Fixed Visual History: Learning Motion-Adaptive Temporal Receptive Fields for Complex-Terrain Quadruped Locomotion

Hao Zhang, Yuping Yan, Yitian Hong, Luying Feng+4

Westlake University, Hangzhou, China. · East China University of Science and Technology, Shanghai, China.

Quadruped Locomotion Vision-Based Robot Control Visuomotor Policy Learning Terrain-Aware Robot Locomotion

Open paper AskPDF arXiv

Versatile quadruped locomotion depends on a tight coupling between what the robot perceives and what its body can execute. Yet existing visual controllers typically organize depth history using fixed temporal patterns, overlooking that the same temporal span corresponds to different spatial ranges as the robot's speed, posture, and gait phase change. Consequently, the same visual history may be insufficient in one motion state but outdated or misaligned in another. To address this limitation, we propose TempoLoco, a body-aware dynamic visual perception framework that organizes visual memory according to the robot's current locomotion state. TempoLoco learns multiple complementary temporal views from a long depth buffer, encodes them within a shared feature space, and dynamically aggregates them using proprioceptive history and multi-scale terrain evidence. This enables a single policy to align visual anticipation with the robot's current motion capabilities across complex terrains, including stairs, platforms, gaps, discrete obstacles, rough ground, and slopes. From the experiment analysis, TempoLoco achieves a 78.4% success rate, compared with 58.2% for the strongest fixed-history baseline and 62.1% for uniform multi-scale fusion. In separate capability evaluations, it completes 41/45 simulated trials across nine terrain types and 23/25 real-world trials on a Unitree B2 with a fixed arm, demonstrating robust and deployable locomotion through body-aware dynamic visual perception.

Abstract

Terrain-Aware Robot Locomotion

Explore similar work

Beyond Fixed Visual History: Learning Motion-Adaptive Temporal Receptive Fields for Complex-Terrain Quadruped Locomotion

ReCo: Response-Consistent Locomotion with Policy-Aware MPC for Legged Manipulation

RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion