VIGOR-MPC: Vision-Guided Objective Regulation MPC

Jelena Trisovic; Rahel Rickenbach; Marek Landert; Panagiotis Minos; Melanie N. Zeilinger

cs.ROOct 14, 2026

VIGOR-MPC: Vision-Guided Objective Regulation MPC

Authors: Jelena Trisovic, Rahel Rickenbach, Marek Landert, Panagiotis Minos, Melanie N. Zeilinger

Organizations: Institute for Dynamic Systems and Control, ETH Zurich, Switzerland · ETH AI Center

Abstract

Controlling an agent in an environment solely perceived through onboard vision is challenging: perception yields noisy, variable-length, and partially occluded context, while onboard compute is shared between perception and control, limiting the planning horizon executable in real time. We propose VIGOR-MPC, a modular student-teacher framework, agnostic to the perception pipeline, which addresses reference tracking across navigation tasks by distilling a privileged, field-of-view-aware long-horizon MPC into a computationally lightweight short-horizon MPC whose cost parameters are determined online from vision. The teacher operates in simulation with full state knowledge and a multi-trajectory formulation that guarantees the agent can stay within its field of view. The student observes only the noisy, occluded context produced by a perception module and learns, via a multi-level recurrent mapping, to infer cost parameters that aim to recover the teacher's behavior despite missing information. We evaluate VIGOR-MPC on vision-based autonomous racing in simulation and on hardware. Using only onboard perception of the track, VIGOR-MPC generalizes to tracks unseen during training and approaches the lap times of the privileged long-horizon teacher while reducing per-step computation by over 90%.

Figures & tables

Figure 1: Overview of the VIGOR-MPC framework. The student employs a predefined, parameterized MPC structure whose cost parameters are generated by a memory-capable neural network from noisy and partial environmental observations. The mapping is trained via imitation learning using a privileged teacher with full environmental knowledge. The vision-based challenges addressed by the framework, together with the resulting control requirements, are illustrated on the right.

Figure 2: Illustration of the considered example setup.

Figure 3: Schematic overview of the proposed context-processing pipeline. To obtain a fixed-dimensional representation, missing entries are padded with non-informative values and accompanied by binary masks indicating valid observations. The resulting context sequence is processed by an RNN, which extracts relevant temporal information while maintaining an internal memory state, enabling retention of previously observed but currently occluded context.

† Computed over $198$ initial states; $2$ outliers in which the MPC failed to converge were removed.
Controller	Imitation error $\downarrow$	Lap time [s] $\downarrow$		Execution time [ms] $\downarrow$
$\mathit{MPC}_{MT^{*},N_{L}}$	$\mathbf{0.0103\pm 0.0216}$	$\mathbf{7.39\pm 0.02}$	$+0.4\%$	$545.0\pm 250.8$ †	${0.0\pm 46.0\%}$
$\mathit{MPC}_{N_{S}}$	$0.0219\pm 0.0315$	$7.91\pm 0.05$	$+7.5\%$	$35.3\pm 4.1$	$93.5\pm 1.1\%$
VIGOR-MPC	${0.0126\pm 0.0335}$	${7.44\pm 0.03}$	${+0.6\%}$	$\mathbf{31.6\pm 4.7}$	$\mathbf{94.2\pm 1.0\%}$

Table 1: Imitation error computed as a root mean square error (RMSE) on states and control inputs over the first

5

prediction steps with respect to

\mathit{MPC}_{MT,N_{L}}

, lap time, and per-step execution time, both reported as absolute values and relative changes with respect to the teacher

\mathit{MPC}_{MT,N_{L}}

. Mean and standard deviation over

50

initial states for the lap time and

200

sample states for the imitation error and execution time calculation. Evaluations are done on the track of Figure 4 without obstacles.

Figure 4: Visualization of environments with randomly generated obstacle configurations and closed loop application of VIGOR-MPC.

Figure 5: Closed-loop employment of VIGOR-MPC on three different track layouts (with randomly placed obstacle configurations), where only the track layout on the left was seen during training.

	Train	Test Track 1		Test Track 2
Controller	Track	top	bottom	left	right
$\mathit{MPC}_{MT,N_{L}}$	${17.75}$ s	$7.36$ s	$7.46$ s	$11.29$ s	$11.29$ s
VIGOR-MPC	$18.12$ s	$7.39$ s	$7.46$ s	$11.32$ s	$11.32$ s

Table 2: Lap time comparison of VIGOR-MPC and perfect teacher on two unseen tracks with two object configurations each.

FOV-Aware MT-MPC Teacher	Masking	Context Recurr.	Time Recurr.	Success
✗	✗	✗	✗	$0\%$
✓	✗	✗	✗	$4\%$
✓	✓	✗	✗	$3\%$
✓	✗	✓	✗	$6\%$
✓	✓	✓	✗	$46\%$
✓	✓	✓	✓	$94\%$

Table 3: Ablation over the FOV-aware components, starting from an obstacle-aware ZipMPC baseline. Each row cumulatively enables the corresponding component, reporting lap completion rate over 100 random obstacle configurations.

Panel of Figure 6 — Figure 6: (Left and center) Impact of multi-frame recurrency for 100 initial conditions: critical frame drops right before an obstacle handled successfully when multi-frame memory is available. (Right) Lateral cost adjustment shifts references.

Figure 7: Closed-loop deployment of VIGOR-MPC on hardware.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	$l_{r}$	$l_{f}$	$a$	$\tau$
Value	0.13429	0.12571	19.995	0.422

Appendix

Table 4: Powertrain and slip angle parameter values.

Figure 8: Illustration of the car’s position in the Frenet frame, including its progress coordinate

\sigma

, its lateral deviation

d

, and its respective orientation

\phi

. Its velocity

v

and the steering angle

\delta

are also shown, as well as the car’s rear and front length,

l_{r}

and

l_{f}

Parameter	Value
$\sigma_{\min}$	0
$\sigma_{\max}$	-
$\phi_{\min}$	$-2$
$\phi_{\max}$	2
$v_{\min}$	0
$v_{\max}$	1.1–2.2

Appendix

Table 5: State and input constraint parameter values.

Figure 9: Example of an egocentric view from the car and the obstacle detection.

Figure 10: Closed-loop employment of VIGOR-MPC on different obstacle layouts (with randomly placed obstacle configurations).

Method	Median	IQR	p90
VIGOR-MPC	$0.00320$	$[0.00055,\,0.01341]$	$0.02913$
$MPC_{MT^{*},N_{L}}$	$0.00152$	$[0.00023,\,0.00947]$	$0.02871$
$MPC_{N_{S}}$	$0.00657$	$[0.00107,\,0.03552]$	$0.05999$

Appendix

Table 6: Heavy-tail statistics of the imitation error: median, interquartile range (IQR), and 90th percentile (p90).

Figure 11: Closed-loop employment of VIGOR-MPC on different obstacle layouts (with randomly placed obstacle configurations) of a track not seen during training.

Figure 12: Closed-loop employment of VIGOR-MPC on different obstacle layouts (with intentionally placed obstacle configurations).

Figure 13: Closed-loop employment of the end-to-end baseline on different obstacle layouts (with randomly placed obstacle configurations corresponding to those shown in Figure 10 ).

$N_{L}$	$15$	$25^{\dagger}$	$30$	$40$
Lap time [s]	$7.46$	$7.36$	$7.33$	$7.33$
Solve time [ms]	$537.3$	$978.2$	$1284.5$	$1982.3$

Appendix

Table 7: Lap-time and per-step solve time of long horizon MT-MPC with increasing horizon length, indicating the used horizon length with a cross.

Figure 14: Visualization of additional successfully handled obstacle configurations of VIGOR-MPC in closed-loop deployment on our real-world setup.

Figure 15: Comparison of our short horizon MPC baseline $MPC_{N_{S}}$ and VIGOR-MPC in closed-loop deployment on our real-world setup. The $MPC_{N_{S}}$ laps are visualized in the image on the left and the one in the middle, VIGOR-MPC is deployed in the image on the right

Figure 16: Deployment of VIGOR-MPC on our hardware track, with a model that has only been trained on the leftmost track of Figure 5 and dynamical obstacle replacement.

Figure 17: Open-loop VIGOR-MPC prediction as training progresses against long horizon MT-MPC open-loop prediction (solid, in blue) and short-horizon MPC open-loop prediction (dashed, in green).

Figure 18: Evolution of the VIGOR-MPC trajectories across epochs.

Figure 19: Visualization of the quadrotor experiments. Left image: quadrotor racing application in 3D, with an illustration of VIGOR-MPC’s racing line and the lap times of $MPC_{MT^{*},N_{L}}$ , $MPC_{N_{S}}$ , and VIGOR-MPC shown at the bottom, where $N_{L}=25$ and $N_{S}=6$ . Middle image: explanatory figure illustrating how a reference-tracking task in a constrained environment can be reformulated as a path-following task. Right image: application of VIGOR-MPC with quadrotor dynamics to solve a reference-tracking task in a pillared environment.

Vision-Based Robot Control

Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.

See all 221 papers

VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion
Mar 15, 2026Aditya Shirwatkar, Satyam Gupta, Shishir KolathayaTerrain-Aware Robot Locomotion Vision-Based Robot Control
Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.
ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC
May 6, 2026Yurui Du, Pinhao Song, Yutong Hu +1Visual SSMs Model Predictive Control
KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images, B-3000 Leuven, Belgium · Flanders Make at KU Leuven, B-3000 Leuven, Belgium · KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics, B-3000 Leuven, Belgium
Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields
Mar 7, 2026Fangguo Zhao, Hanbing Zhang, Zhouheng Li +2Autonomous Drone Racing Aerial Robotics
College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China

Mar 15, 2026J/K move · Enter open · S save

VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

Aditya Shirwatkar, Satyam Gupta, Shishir Kolathaya

Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.

Terrain-Aware Robot Locomotion Vision-Based Robot Control RL for Legged Locomotion Model Predictive Control

Open paper AskPDF arXiv

Perceptive locomotion for legged robots requires anticipating and adapting to complex, dynamic environments. Model Predictive Control (MPC) serves as a strong baseline, providing interpretable motion planning with constraint enforcement, but struggles with high-dimensional perceptual inputs and rapidly changing terrain. In contrast, model-free Reinforcement Learning (RL) adapts well across visually challenging scenarios but lacks planning. To bridge this gap, we propose VIP-Loco, a framework that integrates vision-based scene understanding with RL and planning. During training, an internal model maps proprioceptive states and depth images into compact kinodynamic features used by the RL policy. At deployment, the learned models are used within an infinite-horizon MPC formulation, combining adaptability with structured planning. We validate VIP-Loco in simulation on challenging locomotion tasks, including slopes, stairs, crawling, tilting, gap jumping, and climbing, across three robot morphologies: a quadruped (Unitree Go1), a biped (Cassie), and a wheeled-biped (TronA1-W). Through ablations and comparisons with state-of-the-art methods, we show that VIP-Loco unifies planning and perception, enabling robust, interpretable locomotion in diverse environments.

Abstract

Vision-Based Robot Control

Explore similar work

VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC

Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields