Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

Kai Li; Shiyu Zhao

cs.ROMar 21, 2026

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

Authors: Kai Li, Shiyu Zhao

Organizations: College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

Abstract

Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.

Figures & tables

Fig. 1: The teacher policy (left) leverages appearance-invariant omnidirectional depth images generated by concatenating multi-view inputs, while the student (right) relies on a single-view RGB image. Both the teacher and the student imitate the action $\mathbf{a}$ from the expert data, with the student additionally distilling the feature embedding $\mathbf{z}$ from the teacher. Compared to the teacher, the student policy is computationally lightweight and more suitable for deployment on lightweight low-cost mobile robots.

Fig. 2: Overview of the proposed method. (a) shows the knowledge transfer flow from the state-based expert, to the omnidirectional-depth-based teacher, and the RGB-based student. (b) shows the detailed pipeline of our method. In contrast with the vanilla form of IL, our method imitates both the action output and intermediate visual embeddings, which is highlighted in the dash-line boxes. LP in the teacher denotes linear projection, which is used for embedding dimension matching.

Fig. 3: Omnidirectional RGB and depth images from DATv2. The top row shows the simulation and the bottom row shows the real world. Omni-view is formed by concatenating multi-camera images. Camera 1 is used for the single-view student policy.

Fig. 4: Embedding similarity comparison of different image encoders across scenes. Higher similarity values indicate more consistent embeddings and stronger scene transferability. $\mu$ is the mean value of similarity.

Fig. 5: t-SNE [ 30 ] visualization of visual embeddings of different scenes from DINOv2 [ 31 ] , Zhang et al. [ 12 ] and ours. Different colors represent different scenes.

Fig. 6: The rows show (a) RGB images, (b) depth images, (c) Full Gradient [ 32 ] (FG) attention maps from a policy trained directly on RGB, and (d) FG attention maps from a policy trained with our knowledge distillation method. The attention maps in row (c) are distracted by irrelevant features. In contrast, the maps in row (d) align more with human intuition, consistently focusing on obstacles when they are nearby. The columns show different simulation and real-world scenes. The encoder backbones of (c) and (d) are both Resnet50.

Method	Modality	View	Finetuned	AE Train $\downarrow$	AE Test $\downarrow$
Resnet18 [ 27 ]	RGB	Omni	✗	0.0038	0.0226
Resnet18 [ 27 ]	RGB	Single	✗	0.0051	0.0734
Resnet50 [ 27 ]	RGB	Omni	✗	0.0027	0.0093
Resnet50 [ 27 ]	RGB	Single	✗	0.0042	0.0112
DINOv2 [ 31 ]	RGB	Omni	✗	0.0030	0.0097
DINOv2 [ 31 ]	RGB	Single	✗	0.0049	0.0129

TABLE I: White rows show action error results for methods without embedding distillation, using different input modalities and camera views. Gray rows show results with the proposed knowledge distillation method. Finetuned indicates that the encoder is further trained on our custom robot dataset, while non-finetuned encoders use frozen weights pretrained on a large-scale public dataset.

Method	Modality	View	Finetuned	AE $\downarrow$	SR (%) $\uparrow$	MD (m) $\downarrow$	#Parameters	Encoder Time (ms) $\downarrow$	DATv2 Time (ms)
Resnet18 [ 27 ]	RGB	Single	✗	0.0739	31.0	4.08 ± 0.34	11.7M	16.51 (26.64)	-
Resnet18 [ 27 ]	RGB	Omni	✗	0.0479	40.0	5.19 ± 0.42	11.7M	21.36 (29.54)	-
Resnet18 [ 27 ]	Depth	Single	✓	0.0050	69.0	8.28 ± 0.33	11.7M	10.15 (17.62)	25.69 (34.88)
Resnet18 [ 27 ]	Depth	Omni	✓	0.0037	74.0	8.94 ± 0.31	11.7M	12.20 (19.90)	54.97 (82.20)
Resnet50 [ 27 ]	RGB	Single	✗	0.0229	38.0	5.35 ± 0.21	23.5M	24.10 (33.81)	-
Resnet50 [ 27 ]	RGB	Omni	✗	0.0115	49.0	7.09 ± 0.30	23.5M	45.34 (58.78)	-

TABLE II: Results for the robot navigation task. All onboard models are accelerated with NVIDIA TensorRT using FP16 precision. For a fair comparison, all policies incorporate temporal information using an LSTM with the same sequence length of 5. The table reports Action Error (AE), Success Rate (SR), and Moving Distance (MD). The onboard time is presented as the mean, with the 99% percentile shown in parentheses. The latency of the raw image is around 30 ms. For methods using depth as input, the inference time of DATv2 must be included in the total latency.

Method	AE (Train) $\downarrow$	AE (Test) $\downarrow$	SR % $\uparrow$	MD (m) $\uparrow$
RoboSaGA [ 5 ]	0.0088	0.0301	47.0	5.59 ± 0.51
VISARL [ 14 ]	0.0095	0.0295	40.0	5.02 ± 0.47
Zhang et al. [ 12 ]	0.0054	0.0082	62.0	8.30 ± 0.22
Ours-Rnet50	0.0021	0.0067	72.0	8.49 ± 0.25

TABLE III: Comparison with other methods in the literature that focus on enhancing the performance of visuomotor policies.

Fig. 7: Typical experiment trajectories of robots under policies trained with different strategies and input modalities. The target is moving in a circle (Yellow trajectory). SV denotes single-view image input, and OV denotes omni-view image input.

Fig. 8: Robot trajectories in real-world experiment. The white is the robot and the red is the dynamic target.

Explore similar work

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
May 11, 2026Evans Han, Yunfan Jiang, Yingke Wang +6Robot Policy Learning Visuomotor Policy Learning
1Stanford University · 2Northwestern University · 3Lambda, Inc
Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation
Mar 2, 2026Jan Finke, Wayne Paul Martis, Adrian Schmelter +3Robot Navigation Vision-Based Robot Control
VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation
May 28, 2026Victor Kowalski, Chengxi Li, Dongheui LeeCross-Modal Knowledge Distillation Robot Skill Learning
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

May 11, 2026J/K move · Enter open · S save

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

Evans Han, Yunfan Jiang, Yingke Wang, Haoyue Xiao+5

1Stanford University · 2Northwestern University · 3Lambda, Inc

Robot Policy Learning Visuomotor Policy Learning Robotic Manipulation Stereo Vision

Open paper AskPDF arXiv

Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.

Abstract

Explore similar work

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation