Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

Mingyu Sheng; Jianan Fan; Dongnan Liu; Guoyan Zheng; Ron Kikinis; Weidong Cai

cs.ROOct 22, 2026

Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

Authors: Mingyu Sheng, Jianan Fan, Dongnan Liu, Guoyan Zheng, Ron Kikinis, Weidong Cai

Organizations: School of Computer Science, the University of Sydney, Sydney, NSW, Australia · Institude of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Brigham and Women’s Hospital, and Harvard Medical School, Boston, MA, United States

Abstract

End-to-end control policies trained via Reinforcement Learning (RL) have emerged as an alternative to conventional surgical robotic control approaches, with recent studies exploring state-based policies trained on privileged state observations. However, such approaches rely on precise physical states that are difficult to obtain reliably in surgical environments. Vision-based RL provides an alternative by learning manipulation policies directly from RGB observations, avoiding explicit state estimation and manually engineered perception pipelines while enabling policies to acquire task-relevant visual representations. Nevertheless, obtaining vision-based policies via pure RL methods for surgical manipulation remains challenging. Thus, imitation learning (IL) becomes another solution, yet it requires a large number of expert demonstrations, which are difficult to obtain in minimally invasive surgery. In this work, we propose a framework for state-to-vision policy distillation in laparoscopic robotic manipulation. Specifically, a state-based policy with privileged observations is employed as the teacher, and a vision-based policy learns from RGB observations and robot proprioceptive information, which requires less expert demonstrations than IL and introduces superior performance than RL methods. We conduct extensive experiments on various surgical manipulation tasks in the simulated environment. Experimental results show that, in the context of laparoscopic robotic manipulation, training vision-based policies with general-purpose RL methods is challenging, whereas our proposed state-to-vision policy distillation leads to more consistent and stable performance across tasks.

Figures & tables

Fig. 1: Semantic decomposition of this work. (a) The first step is to train a state-based policy. (b) Then, the state-based policy is employed as the teacher to transfer policy knowledge to the student network by aligning their action distributions. A new replay buffer $\mathfrak{B}_{2}$ is used in (b) because $\mathfrak{B}_{1}$ in (a) contains transitions collected by earlier versions of the teacher policy, which may interfere with the student policy’s learning of the final teacher policy.

Derivative	FKL	RKL	JD	WD	MSE
$\displaystyle\frac{\partial D}{\partial\mu_{s}}$	$\displaystyle\frac{\Delta\mu}{\sigma_{s}^{2}}$	$\displaystyle\frac{\Delta\mu}{\sigma_{t}^{2}}$	$\displaystyle\Delta\mu\left(\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}\right)$	$\displaystyle 2\Delta\mu$	$\displaystyle 2\Delta\mu$
$\displaystyle\frac{\partial D}{\partial\sigma_{s}}$	$\displaystyle\frac{1}{\sigma_{s}}-\frac{\sigma_{t}^{2}+\Delta\mu^{2}}{\sigma_{s}^{3}}$	$\displaystyle-\frac{1}{\sigma_{s}}+\frac{\sigma_{s}}{\sigma_{t}^{2}}$	$\displaystyle\frac{\sigma_{s}}{\sigma_{t}^{2}}-\frac{\sigma_{t}^{2}+\Delta\mu^{2}}{\sigma_{s}^{3}}$	$\displaystyle 2(\sigma_{s}-\sigma_{t})$	$0$
$\displaystyle\frac{\partial^{2}D}{\partial\mu_{s}^{2}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}$	$\displaystyle\frac{1}{\sigma_{t}^{2}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}$	$2$	$2$
$\displaystyle\frac{\partial^{2}D}{\partial\sigma_{s}^{2}}$	$\displaystyle-\frac{1}{\sigma_{s}^{2}}+\frac{3(\sigma_{t}^{2}+\Delta\mu^{2})}{\sigma_{s}^{4}}$	$\displaystyle\frac{1}{\sigma_{s}^{2}}+\frac{1}{\sigma_{t}^{2}}$	$\displaystyle\frac{1}{\sigma_{t}^{2}}+\frac{3(\sigma_{t}^{2}+\Delta\mu^{2})}{\sigma_{s}^{4}}$	$2$	$0$
$\displaystyle\frac{\partial^{2}D}{\partial\mu_{s}\partial\sigma_{s}}$	$\displaystyle-\frac{2\Delta\mu}{\sigma_{s}^{3}}$	$0$	$\displaystyle-\frac{2\Delta\mu}{\sigma_{s}^{3}}$	$0$	$0$

TABLE I: First- and second-order derivatives of different distribution matching objectives with respect to the visual policy parameters

\mu_{s}

and

\sigma_{s}

. We define

\Delta\mu=\mu_{s}-\mu_{t}

. The row for

\frac{\partial D}{\partial\sigma_{s}\mu_{s}}

is omitted because

\frac{\partial^{2}D}{\partial\sigma_{s}\mu_{s}}=\frac{\partial^{2}D}{\partial\mu_{s}\sigma_{s}}

Fig. 2: Toy illustration of the PD regions of the FKL and JD Hessians under different teacher policy parameters. The PD region of FKL is always a proper subset of that of JD.

Method	Learning	NeedleReach	GauzeRetrieve	NeedlePick	PegTransfer	NeedleRegrasp
CURL [ 11 ]	RL	98.8	1.8	0.0	0.0	0.0
DrQv2 [ 13 ]	RL	99.8	2.8	1.2	0.0	0.0
Diffusion [ 16 ]	RL	80.2	13.4	22.4	2.2	1.4
ACT [ 27 ]	IL	N/A	N/A	$76.0^{*}$	N/A	N/A
ALOHA [ 15 ]	IL	99.8	76.8	93.6	16.4	10.0
Ours (FKL)	PDL	98.6	66.0	70.8	75.4	71.6

TABLE II: Success Rate (%) of Various Methods Across Different Tasks. The ”Learning” column indicates Reinforcement Learning (RL), Imitation Learning (IL), or Policy Distillation (PDL). Bold marks the best result for each task across different methods. Underline denotes the second-best one. Results marked with ”*” are directly copied from original publications.

Fig. 3: Feature maps extracted from the last convolutional layer. (a), (b), (c), and (d) represent GauzeRetrieve, NeedlePick, PegTransfer, and NeedleRegrasp, respectively. The small red orb marks the position to which the target objects (e.g., needle and gauze) should be moved. For a human-friendly view, the raw feature maps ( $64\times 64\times 32$ ) are averaged across the channel dimension and refined via Gaussian blurring and interpolation.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

May 28, 2026

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.

Victor Kowalski, Chengxi Li, Dongheui Lee

Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

Cross-Modal Knowledge Distillation Robot Skill Learning Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation Robotic RL Robot Policy Generalization+2

Ask PDF

Oct 19, 2026

ReDOA: Real-World Distillation via Observation Alignment for Image-Based Manipulation Policy Learning

Data-driven learning for general-purpose robotic manipulation presents a fundamental trade-off between data-collection cost and adaptability to real-world environments. Imitation learning (IL) requires costly human demonstrations, whereas simulation-based learning is vulnerable to sim-to-real gaps in dynamics and sensor observations. We propose ReDOA (Real-World Distillation via Observation Alignment), a framework that decouples control learning from perception learning to learn image-based policies in the real world. A state-based teacher policy trained in simulation is executed on a real robot, and its control knowledge is distilled into an image-based student policy using the collected trajectories. Observation Alignment resolves observation mismatches between data collection and deployment. This framework collects successful, efficiently executed trajectories paired with real-world visual observations, without requiring demonstrations from human experts. In real-robot experiments on three non-prehensile manipulation tasks, including bimanual manipulation, our method achieves better overall performance when jointly considering task success and execution efficiency than both IL based on human demonstrations and policy distillation in simulation. https://isri-aist.github.io/redoa-project-page/

Kotaro Kajitsuka, Masaki Murooka, Fumio Kanehiro

University of Tsukuba, Tsukuba, Japan. · CNRS-AIST JRL (Joint Robotics Laboratory), IRL3218, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan.

Visuomotor Policy Learning Robotic Manipulation Policy Distillation Sim-to-Real Transfer

Ask PDF

Sep 27, 2026

Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control

Simulation-trained manipulation policies can exploit privileged state information to learn effective contact-rich behaviours, but deployment requires acting from partial observations such as noisy camera images. A common solution is teacher-student distillation, in which a visuomotor policy is trained to reproduce the actions of the privileged expert. This requires the student to jointly infer the task-relevant state and relearn the expert's action mapping that is already available. An alternative is to reuse the state-based expert and learn only a perceptual interface that reconstructs its missing state inputs. However, minimising the state estimate error alone does not necessarily minimise the downstream control error induced by these estimates. To bridge this gap, we train a visual state estimator using both direct state supervision and an action-consistency loss backpropagated through the frozen, differentiable expert. A scheduled objective first establishes a physically meaningful state estimate and progressively emphasises errors that affect the expert's actions. Across five goal-conditioned manipulation tasks, retaining the expert consistently outperforms direct pixel-to-action imitation from the same expert demonstration corpus. We further demonstrate sim-to-real transfer on a physical Panda robot, achieving 76% success without retraining the underlying expert.

Denis Shcherba, Adrian Abel, Eckart Cobo-Briesewitz +3

Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany · Technische Universität Berlin, Germany

Visuomotor Policy Learning Robotic Manipulation Robot State Estimation Policy Learning

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

ReDOA: Real-World Distillation via Observation Alignment for Image-Based Manipulation Policy Learning

Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control