WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing

Guanzhong Sun; Junyi Ma; Yixuan Zhou; Yuxuan Wu; Yanzi Miao; Hesheng Wang

cs.ROSep 17, 2026

WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing

Authors: Guanzhong Sun, Junyi Ma, Yixuan Zhou, Yuxuan Wu, Yanzi Miao, Hesheng Wang

Organizations: School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China. · SJTU Global College, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. ·

Abstract

Closed-loop visual servoing requires predictions that indicate whether an action reduces task error, not only whether the action is plausible. We call this gap the prediction-control mismatch and introduce WM-VS, a target-centric progress-aligned world-model framework for closed-loop visual servoing. Offline target-region DINOv2 correspondences define a signed four-dimensional servo coordinate for translation, scale, and in-plane rotation. Stage 1 aligns action-conditioned latent transitions with this coordinate; Stage 2 freezes the world model and trains a reactive joint-velocity policy with action imitation, consequence supervision, and short imagined rollouts that favor error contraction. Deployment is RGB-only and reactive, without online trajectory optimization. On a real 7-DoF eye-to-hand system, WM-VS reaches a corner RMSE no larger than 10 percent of its initial value in 30/30 trials and retains this criterion at the final valid frame in 25/30 (83.33 percent). Removing future-error alignment reduces retention to 26.67 percent. The learned progress signal agrees with an external AprilTag corner error not used for training or control (mean Spearman rho = 0.8778). Without retraining, two unseen 3D targets achieve translation-error reductions of 86.48 percent and 90.27 percent and rotation-error reductions of 70.01 percent and 65.70 percent. These results link progress-aligned action consequences to repeated closed-loop correction and transfer. Code and data will be released as open source.

Figures & tables

Fig. 1: Overview of WM-VS. Frozen DINOv2 spatial features are focused on the task target and fused with robot history. Stage 1 aligns action-conditioned latent transitions with a target-centric directional servo coordinate whose normalized reduction defines visual task progress. Stage 2 freezes the aligned model and uses imagined action consequences to train a reactive goal-conditioned joint-velocity policy toward error contraction.

Fig. 2: Directional servo coordinate for action-consequence learning. The goal-conditioned policy predicts a joint action from the current and goal latents together with robot history. The aligned predictive representation maps the imagined consequence to the four-dimensional error coordinate. Action imitation constrains the policy near the data distribution, while the servo terms regularize local consequences and encourage short-horizon error contraction.

Method	Task-specific training	RGB	Depth	Online explicit features
WM-VS (Ours)	✓	✓	–	–
WM-VS w/o Future-Error Alignment	✓	✓	–	–
WM-VS w/o Error Guidance	✓	✓	–	–
WM-VS (BC-only)	✓	✓	–	–
VSNet-E2H	✓	✓	–	–
Moment-VS	–	✓	–	$\checkmark^{\dagger}$

TABLE I: Compared methods and online information used by the controller.

Panel of Figure 4 — Fig. 3: Best-observed real-robot precision. (a) WM-VS, (b) VSNet-E2H, (c) Moment-VS, and (d) ViT-VS-E2H under identical initial/desired configurations. Blue/red quadrilaterals mark initial/desired AprilTag corners.

Method	Conv. †	Best RMSE (px) §	Best trans. (mm) §	Best rot. ( ∘ ) §	Final ret. ‡	Post-min. 50-step non-inc./dec. (%)
WM-VS (Ours)	100.00%	$10.20\pm 2.30$	$13.21\pm 2.32$	$5.78\pm 2.55$	83.33%	$63.3/34.0$
WM-VS w/o Future-Error Alignment	86.67%	$12.16\pm 6.51$	$20.78\pm 9.51$	$6.49\pm 3.82$	26.67%	$65.6/26.3$
WM-VS w/o Error Guidance	36.67%	$13.51\pm 5.16$	$23.06\pm 14.05$	$9.09\pm 4.53$	23.33%	$40.3/16.7$
WM-VS (BC-only)	90.00%	$11.44\pm 8.42$	$12.86\pm 3.55$	$6.61\pm 3.25$	50.00%	$68.8/20.8$
VSNet-E2H	100.00%	$13.18\pm 2.17$	$14.55\pm 4.12$	$27.11\pm 4.31$	0.00%	$2.5/2.0$
Moment-VS	66.67%	$20.47\pm 11.55$	$25.52\pm 15.93$	$18.11\pm 7.29$	56.67%	$49.3/39.8$

TABLE II: Transient accuracy, final retention, and post-minimum local behavior over 30 real-robot trials.

Fig. 4: Representative closed-loop corner-RMSE trajectories. Trajectories continue beyond their minima to the final valid step, exposing post-minimum rebound that best error alone does not capture.

Fig. 5: Consistency between learned progress and an external normalized AprilTag corner error. One example WM-VS trajectory is shown. The red curve is the model-predicted visual error and the blue curve is $E_{\mathrm{tag}}$ , the external normalized AprilTag corner error in ( 13 ). The reported correlations are mean per-trajectory statistics over 30 real-robot trials.

Panel of Figure 8 — Fig. 6: Background interference evaluation. Representative best observed frames are shown for (a) a static black background and (b) a moving bag. The blue and red quadrilaterals mark the AprilTag corner detections at the initial and desired configurations.

Panel of Figure 9 — Fig. 7: Zero shot transfer to new 3D targets. The two targets differ in shape and surface material.

Target	$E_{\mathrm{trans}}$	$E_{\mathrm{rot}}$	Trans. red.	Rot. red.
	(mm)	(deg)	(%)	(%)
Silicone finger	$45.66\pm 19.46$	$10.34\pm 10.27$	$86.48\pm 5.85$	$70.01\pm 23.83$
Plastic hemisphere	$31.88\pm 13.72$	$9.76\pm 5.89$	$90.27\pm 4.34$	$65.70\pm 20.03$

TABLE III: Zero shot transfer results.

Vision-Based Robot Control

Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.

See all 222 papers

World Action Modeling with Progressive Visual Planning
Oct 1, 2026Fei Zhang, Zhaochong An, Duncan Frost +5World Model-Based Planning Robot Task Planning
Shanghai Jiao Tong University · SII · Meta +1
Achieve What You Imagined: Learning to Align Actions with Visual Plans
Sep 27, 2026Yuheng Qiao, Ziran Wei, Xiaohan Wang +5World Models for Robotics World Model-Based Planning
KTH · Beihang University · The Hong Kong University of Science and Technology (Guangzhou) +2
VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing
Sep 23, 2026Yimin Pan, Sen Wang, You Zhou +4Relative Pose Estimation Hand-Eye Calibration
All authors are with Agile Robots SE, Munich, Germany. · Technical University of Munich

Oct 1, 2026J/K move · Enter open · S save

World Action Modeling with Progressive Visual Planning

Fei Zhang, Zhaochong An, Duncan Frost, Yikai Wang+4

Shanghai Jiao Tong University · SII · Meta+1

World Model-Based Planning Robot Task Planning Vision-Language-Action Models Efficient Inference for World Action Models Robustness of VLA Models

Open paper AskPDF arXiv

World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WAMs address this by predicting a single future frame without generating the full video, but this approach neglects how to progress toward the goal. We present ProWAM, a progressive world action model that jointly predicts actions and an ordered sequence of sparse visual sub-goals, providing explicit visual guidance to anchor action generation throughout task execution. This design scales naturally, as sub-goal prediction can be learned from large-scale action-free videos, allowing the video backbone to offload complex visual planning from the action policy. For efficient action generation, ProWAM executes a single video-backbone forward pass to cache sparse sub-goal features, eliminating iterative full-video generation and requiring only lightweight action denoising during replanning. Across extensive evaluations, ProWAM achieves superior out-of-distribution robustness. On simulation benchmarks, it sets new state-of-the-art results on LIBERO-Plus (85.8%) and randomized RoboTwin (75.7%), outperforming the strongest baseline with relative gains of up to +35.9%. On RoboCasa365, ProWAM achieves a 48.1% success rate and 18.2% on the challenging Composite-Unseen split, ranking 4th overall. Crucially, in zero-shot real-world experiments, ProWAM achieves 70.0% success, outperforming the strongest baseline by +15.0 (from 55.0% to 70.0%, a +27.3% relative gain) in novel scenes. These results demonstrate the value of progress-indexed visual foresight for closed-loop control. Our program is in https://sii-ferenas.github.io/ProWAM-page.

Abstract

Vision-Based Robot Control

Explore similar work

World Action Modeling with Progressive Visual Planning

Achieve What You Imagined: Learning to Align Actions with Visual Plans

VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing