Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

Mingyu Sheng; Jianan Fan; Dongnan Liu; Guoyan Zheng; Ron Kikinis; Weidong Cai

cs.ROOct 22, 2026

Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

Authors: Mingyu Sheng, Jianan Fan, Dongnan Liu, Guoyan Zheng, Ron Kikinis, Weidong Cai

Organizations: School of Computer Science, the University of Sydney, Sydney, NSW, Australia · Institude of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Brigham and Women’s Hospital, and Harvard Medical School, Boston, MA, United States

Abstract

End-to-end control policies trained via Reinforcement Learning (RL) have emerged as an alternative to conventional surgical robotic control approaches, with recent studies exploring state-based policies trained on privileged state observations. However, such approaches rely on precise physical states that are difficult to obtain reliably in surgical environments. Vision-based RL provides an alternative by learning manipulation policies directly from RGB observations, avoiding explicit state estimation and manually engineered perception pipelines while enabling policies to acquire task-relevant visual representations. Nevertheless, obtaining vision-based policies via pure RL methods for surgical manipulation remains challenging. Thus, imitation learning (IL) becomes another solution, yet it requires a large number of expert demonstrations, which are difficult to obtain in minimally invasive surgery. In this work, we propose a framework for state-to-vision policy distillation in laparoscopic robotic manipulation. Specifically, a state-based policy with privileged observations is employed as the teacher, and a vision-based policy learns from RGB observations and robot proprioceptive information, which requires less expert demonstrations than IL and introduces superior performance than RL methods. We conduct extensive experiments on various surgical manipulation tasks in the simulated environment. Experimental results show that, in the context of laparoscopic robotic manipulation, training vision-based policies with general-purpose RL methods is challenging, whereas our proposed state-to-vision policy distillation leads to more consistent and stable performance across tasks.

Figures & tables

Fig. 1: Semantic decomposition of this work. (a) The first step is to train a state-based policy. (b) Then, the state-based policy is employed as the teacher to transfer policy knowledge to the student network by aligning their action distributions. A new replay buffer $\mathfrak{B}_{2}$ is used in (b) because $\mathfrak{B}_{1}$ in (a) contains transitions collected by earlier versions of the teacher policy, which may interfere with the student policy’s learning of the final teacher policy.

Derivative	FKL	RKL	JD	WD	MSE
$\displaystyle\frac{\partial D}{\partial\mu_{s}}$	$\displaystyle\frac{\Delta\mu}{\sigma_{s}^{2}}$	$\displaystyle\frac{\Delta\mu}{\sigma_{t}^{2}}$	$\displaystyle\Delta\mu\left(\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}\right)$	$\displaystyle 2\Delta\mu$	$\displaystyle 2\Delta\mu$
$\displaystyle\frac{\partial D}{\partial\sigma_{s}}$	$\displaystyle\frac{1}{\sigma_{s}}-\frac{\sigma_{t}^{2}+\Delta\mu^{2}}{\sigma_{s}^{3}}$	$\displaystyle-\frac{1}{\sigma_{s}}+\frac{\sigma_{s}}{\sigma_{t}^{2}}$	$\displaystyle\frac{\sigma_{s}}{\sigma_{t}^{2}}-\frac{\sigma_{t}^{2}+\Delta\mu^{2}}{\sigma_{s}^{3}}$	$\displaystyle 2(\sigma_{s}-\sigma_{t})$	$0$
$\displaystyle\frac{\partial^{2}D}{\partial\mu_{s}^{2}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}$	$\displaystyle\frac{1}{\sigma_{t}^{2}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}$	$2$	$2$
$\displaystyle\frac{\partial^{2}D}{\partial\sigma_{s}^{2}}$	$\displaystyle-\frac{1}{\sigma_{s}^{2}}+\frac{3(\sigma_{t}^{2}+\Delta\mu^{2})}{\sigma_{s}^{4}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}$	$\displaystyle\frac{1}{\sigma_{t}^{2}}+\frac{3(\sigma_{t}^{2}+\Delta\mu^{2})}{\sigma_{s}^{4}}$	$2$	$0$
$\displaystyle\frac{\partial^{2}D}{\partial\mu_{s}\partial\sigma_{s}}$	$\displaystyle-\frac{2\Delta\mu}{\sigma_{s}^{3}}$	$0$	$\displaystyle-\frac{2\Delta\mu}{\sigma_{s}^{3}}$	$0$	$0$

TABLE I: First- and second-order derivatives of different distribution matching objectives with respect to the visual policy parameters

\mu_{s}

and

\sigma_{s}

. We define

\Delta\mu=\mu_{s}-\mu_{t}

. The row for

\frac{\partial D}{\partial\sigma_{s}\mu_{s}}

is omitted because

\frac{\partial^{2}D}{\partial\sigma_{s}\mu_{s}}=\frac{\partial^{2}D}{\partial\mu_{s}\sigma_{s}}

Fig. 2: Toy illustration of the PD regions of the FKL and JD Hessians under different teacher policy parameters. The PD region of FKL is always a proper subset of that of JD.

Method	Learning	NeedleReach	GauzeRetrieve	NeedlePick	PegTransfer	NeedleRegrasp
CURL [ 11 ]	RL	98.8	1.8	0.0	0.0	0.0
DrQv2 [ 13 ]	RL	99.8	2.8	1.2	0.0	0.0
Diffusion [ 16 ]	RL	80.2	13.4	22.4	2.2	1.4
ACT [ 27 ]	IL	N/A	N/A	$76.0^{*}$	N/A	N/A
ALOHA [ 15 ]	IL	99.8	76.8	93.6	16.4	10.0
Ours (FKL)	PDL	98.6	66.0	70.8	75.4	71.6

TABLE II: Success Rate (%) of Various Methods Across Different Tasks. The ”Learning” column indicates Reinforcement Learning (RL), Imitation Learning (IL), or Policy Distillation (PDL). Bold marks the best result for each task across different methods. Underline denotes the second-best one. Results marked with ”*” are directly copied from original publications.

Fig. 3: Feature maps extracted from the last convolutional layer. (a), (b), (c), and (d) represent GauzeRetrieve, NeedlePick, PegTransfer, and NeedleRegrasp, respectively. The small red orb marks the position to which the target objects (e.g., needle and gauze) should be moved. For a human-friendly view, the raw feature maps ( $64\times 64\times 32$ ) are averaged across the channel dimension and refined via Gaussian blurring and interpolation.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation
May 28, 2026Victor Kowalski, Chengxi Li, Dongheui LeeCross-Modal Knowledge Distillation Robot Skill Learning
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany
ReDOA: Real-World Distillation via Observation Alignment for Image-Based Manipulation Policy Learning
Oct 19, 2026Kotaro Kajitsuka, Masaki Murooka, Fumio KanehiroVisuomotor Policy Learning Robotic Manipulation
University of Tsukuba, Tsukuba, Japan. · CNRS-AIST JRL (Joint Robotics Laboratory), IRL3218, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan.
Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control
Sep 27, 2026Denis Shcherba, Adrian Abel, Eckart Cobo-Briesewitz +3Visuomotor Policy Learning Robotic Manipulation
Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany · Technische Universität Berlin, Germany

May 28, 2026J/K move · Enter open · S save

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

Victor Kowalski, Chengxi Li, Dongheui Lee

Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

Cross-Modal Knowledge Distillation Robot Skill Learning Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation Robotic RL Robot Policy Generalization+2

Open paper AskPDF arXiv

When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.

Abstract

Robotic Manipulation

Explore similar work

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

ReDOA: Real-World Distillation via Observation Alignment for Image-Based Manipulation Policy Learning

Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control