cs.RONov 3, 2026

All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy

Authors: Byungkun Lee, Minho Park, Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Kyungmin Lee, Yunho Kim, 

Organizations: KAIST · Holiday Robotics · KRAFTON · Neuromeka

Abstract

Visual Foundation Models (VFMs) provide semantic cues such as depth and segmentation useful for robotic manipulation, but directly deploying them in closed-loop control is impractical due to high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations. We propose All-For-One (AFO), a two-stage distillation framework that leverages VFMs strictly as privileged information during training, using downstream robotic data. Following the privileged learning paradigm, AFO decouples semantic representation learning from inference-time policy execution. In the All stage, a teacher policy fuses RGB observations with outputs from multiple VFMs to learn a semantically enriched representation. In the For-One stage, this knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway for transferring foundation-model knowledge to robotic control.

Figures & tables

Explore similar work

CardsList
  1. Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models

    Jul 12, 2026Xiatao Sun, Yuan Zhuang, Mateo Sanchez Lopez Negrete +9Robot Foundation ModelsVisual Grounding

  2. Unifying Embodied Foundation Models via a Model-to-Policy Formulation

    Oct 16, 2026Xinyi Hong, Mingyang RenRobot Foundation ModelsVision-Language-Action Models

  3. STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation

    Jan 28, 2026Alexandre Chapin, Emmanuel Dellandréa, Liming ChenVision Foundation Model AdaptationGeneralization in Robotic Manipulation