Visual Foundation Models (VFMs) provide semantic cues such as depth and segmentation useful for robotic manipulation, but directly deploying them in closed-loop control is impractical due to high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations. We propose All-For-One (AFO), a two-stage distillation framework that leverages VFMs strictly as privileged information during training, using downstream robotic data. Following the privileged learning paradigm, AFO decouples semantic representation learning from inference-time policy execution. In the All stage, a teacher policy fuses RGB observations with outputs from multiple VFMs to learn a semantically enriched representation. In the For-One stage, this knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway for transferring foundation-model knowledge to robotic control.
Figures & tables
Figure 1: Overcoming the Compute–Performance Tradeoff. Learning from scratch (LfS) [ 17 ] is fast but inaccurate, whereas VFM backbones [ 43 , 50 , 39 , 35 , 37 ] are accurate but slow. By distilling large-scale pretrained VFMs [ 10 , 46 , 58 ] into a compact student, AFO attains high success rates while remaining computationally efficient at inference.
Figure 2: Overview of the All-For-One (AFO) Framework. (a) All. A teacher network employs multiple VFMs ( e.g . , Molmo [ 10 ] , SAM 2 [ 46 ] , and DAM 2 [ 58 ] ) with learnable lightweight encoders ( fRGBT , fobjectT , fdepthT ). The resulting representations are aggregated and used to train a policy with a BC loss. (b) For One. A student lightweight RGB encoder ( fRGBS ) solely learns from the teacher network via a feature distillation loss with BC loss, which benefits from the rich knowledge of large VFMs.
Figure 3: Visual illustration of the simulation and real-world benchmarks. We adopt the Adroit [ 44 ] and MetaWorld [ 59 ] tasks as our simulated environments, and clutter sweep, clean toolbox, and pick-and-place as our real-world tasks. We further evaluate under visual distribution shifts, including desk mat and lighting changes and increased clutter.
Method
Adroit [ 44 ]
MetaWorld [ 59 ]
Average (7 tasks)
Efficiency
Pen
Relocate
Assembly
Bin
Button
Drawer
Hammer
Latency (ms)
# Params
Use Pretrained VFMs at inference time.
DINOv2 [ 39 ]
72.80
59.20
100.00
89.60
89.60
100.00
98.40
87.09 ± 1.60
12.48
88M
R3M [ 37 ]
72.00
70.40
100.00
88.80
96.80
100.00
96.00
89.14 ± 0.57
6.41
26M
MVP [ 43 ]
68.00
56.80
100.00
95.20
75.20
100.00
98.40
84.80 ± 0.96
7.11
87M
VC-1 [ 35 ]
68.80
52.80
100.00
93.60
75.20
100.00
100.00
84.34 ± 0.87
6.84
87M
Table 1: AFO outperforms all baselines at minimal computational cost. AFO’s student matches or surpasses VFM-based baselines on Adroit and MetaWorld while running at the lowest latency and parameter count, on par with the smallest from-scratch baseline (LfS).
Problem [0.2ex]Setting
Method
Real-world
Average (3 tasks)
Efficiency
Clutter Sweep
Clean Toolbox
Pick & Place
Latency (ms)
# Params
ID
ACT [ 62 ]
68.89
47.50
60.00
58.80 ± 8.45
4.32
34.19M
Theia [ 50 ]
66.70
50.00
58.90
58.53 ± 4.88
7.94
125.24M
AFO
74.40
49.17
70.00
64.52 ± 2.16
4.32
34.19M
OOD
ACT [ 62 ]
48.88
43.33
37.78
43.33 ± 5.14
4.32
34.19M
Theia [ 50 ]
44.44
41.66
43.33
43.14 ± 3.36
7.94
125.24M
Table 2: Comparison on Real-world Imitation Learning. Success rates under in-distribution (ID) and visual out-of-distribution (OOD) conditions on three real-world tasks. While all methods perform comparably in distribution, AFO degrades far less under visual shifts, surpassing both baselines by a large margin in OOD at the same cost as the lightweight ACT.
Figure 4: Failure Analysis of VFM-Based Teacher Policies. Directly using VFMs in the teacher introduces brittle failure modes. (a) Segmentation errors: SAM 2 misses the target or masks an irrelevant object. (b) Depth errors: DAM 2 mis-estimates geometry, e.g . , predicting a shadow as a closer surface. (c) Temporal inconsistencies: depth estimates fluctuate across adjacent frames. (d) These errors substantially degrade performance: on Adroit Relocate, the teacher succeeds 87% without VFM errors, but only 45% when an error occurs.
Teacher Modality
Teacher Cost
Average (7 tasks)
RGB
Depth
Segment.
Latency (ms)
# Params.
Teacher
Student
✓
–
–
2.11
2.72M
84.57 ± 2.02
–
✓
✓
–
42.78
100.3M
88.34 ± 1.65
89.26 ± 3.49
✓
–
✓
70.95
8.10B
85.37 ± 4.51
91.54 ± 1.48
✓
✓
✓
110.07
8.20B
87.09 ± 1.54
92.57 ± 1.51
Table 3: Effect of teacher modalities on distillation. Depth and segmentation each improve the teacher, but teacher accuracy does not predict distillation quality: only the teacher with both modalities yields the best RGB-only student (2.11 ms, 2.72M).
Distillation Configuration
Adroit (2 tasks)
MetaWorld (5 tasks)
Average (7 tasks)
Dataset
Distill Target
Capacity
ImageNet
VFM Features
ViT
68.80 ± 5.40
94.88 ± 2.37
87.43 ± 2.74
Downstream
VFM Features
ViT
80.80 ± 5.02
94.88 ± 2.09
90.86 ± 1.40
Downstream
VFM Outputs
ViT
88.80 ± 3.03
99.68 ± 0.44
96.57 ± 0.81
Downstream
VFM Outputs
CNN
81.60 ± 1.67
96.96 ± 1.54
92.57 ± 1.51
Table 4: Comprehensive Comparison with Theia Distillation Objective. The first row is the original Theia and the last row is AFO (Student); the two middle rows vary one factor at a time.
Teacher
Distillation Configuration
Average (7 tasks)
Segment.
Depth
Loss
Alignment
λ
BC-only from scratch (LfS), no distillation
84.57 ± 2.02
–
DAM
cos
per-modality
dynamic
89.26 ± 3.49
SAM
–
cos
per-modality
dynamic
91.54 ± 1.48
SAM
DAM
ℓ2
per-modality
dynamic
92.00 ± 2.49
SAM
DAM
cos
fused
dynamic
92.00 ± 1.46
Table 5: Comprehensive Ablation Studies. Removing a teacher modality costs more than changing the distillation configuration: dropping segmentation degrades performance noticeably, whereas the loss, alignment, and λ variants all remain close to our default.
Hyperparameter
Value
Optimizer
Adam
Learning rate
1×10−3
Momentum
β1,β2=0.9,0.999
Batch size
256
LR schedule
cosine ( 10−3→10−6 )
Warmup epochs
5
Table A1 : Training hyperparameters.
Method
Success Rate
p -value
Adroit (2 tasks)
MetaWorld (5 tasks)
Average (7 tasks)
Adroit (2 tasks)
MetaWorld (5 tasks)
Average (7 tasks)
DINOv2 [ 39 ]
66.00 ± 3.16
95.52 ± 1.45
87.09 ± 1.60
< 0.0001
0.1668
0.0005
R3M [ 37 ]
71.20 ± 3.63
96.32 ± 1.21
89.14 ± 0.57
0.0014
0.4871
0.0048
MVP [ 43 ]
62.40 ± 5.55
93.76 ± 1.73
84.80 ± 0.96
0.0009
0.0152
< 0.0001
VC-1 [ 35 ]
60.80 ± 4.15
93.76 ± 1.64
84.34 ± 0.87
0.0001
0.0130
< 0.0001
Theia [ 50 ]
68.80 ± 5.40
94.88 ± 2.37
87.43 ± 2.74
0.0045
0.1450
0.0097
Table A2 : Significance of AFO (Student) over each method, two-sided Welch’s t -test over five seeds.
Model
Efficiency
Adroit
MetaWorld
Average
Inference
#Params
Pen
Relocate
Assembly
Bin
Button
Drawer
Hammer
Adroit
MetaWorld
7 tasks
DINOv2 [ 39 ]
12.48ms
88.26M
72.80 ± 3.35
59.20 ± 9.12
100.00 ± 0.00
89.60 ± 6.69
89.60 ± 4.56
100.00 ± 0.00
98.40 ± 2.19
66.00 ± 3.16
95.52 ± 1.45
87.09 ± 1.60
R3M [ 37 ]
6.41ms
25.60M
72.00 ± 7.48
70.40 ± 3.58
100.00 ± 0.00
88.80 ± 5.22
96.80 ± 3.35
100.00 ± 0.00
96.00 ± 0.00
71.20 ± 3.63
96.32 ± 1.21
89.14 ± 0.57
MVP [ 43 ]
7.11ms
87.40M
68.00 ± 4.90
56.80 ± 6.57
100.00 ± 0.00
95.20 ± 4.38
75.20 ± 10.73
100.00 ± 0.00
98.40 ± 2.19
62.40 ± 5.55
93.76 ± 1.73
84.80 ± 0.96
VC-1 [ 35 ]
6.84ms
87.40M
68.80 ± 3.35
52.80 ± 6.57
100.00 ± 0.00
93.60 ± 3.58
75.20 ± 7.69
100.00 ± 0.00
100.00 ± 0.00
60.80 ± 4.15
93.76 ± 1.64
84.34 ± 0.87
Theia [ 50 ]
7.94ms
88.40M
72.00 ± 4.90
65.60 ± 7.80
100.00 ± 0.00
91.20 ± 7.69
84.00 ± 11.31
100.00 ± 0.00
99.20 ± 1.79
68.80 ± 5.40
94.88 ± 2.37
87.43 ± 2.74
Table B3 : Full performance table with standard deviations. We report the mean success rate and standard deviation ( ± ) across 5 seeds. For all baselines, random-shift data augmentation is applied. For ViT baselines, spatial tokens (not [CLS] ) are used as policy input.
Method
Distillation Configuration
Adroit
MetaWorld
Average
Dataset
Distill Target
Capacity
Pen
Relocate
Assembly
Bin
Button
Drawer
Hammer
Adroit
MetaWorld
7 tasks
Theia (Orig.) [ 50 ]
ImageNet
VFM Features
ViT
72.00 ± 4.90
65.60 ± 7.80
100.00 ± 0.00
91.20 ± 7.69
84.00 ± 11.31
100.00 ± 0.00
99.20 ± 1.79
68.80 ± 5.40
94.88 ± 2.37
87.43 ± 2.74
Downstream
VFM Features
ViT
82.40 ± 3.58
79.20 ± 9.55
100.00 ± 0.00
92.00 ± 6.32
89.60 ± 5.37
100.00 ± 0.00
92.80 ± 5.93
80.80 ± 5.02
94.88 ± 2.09
90.86 ± 1.40
Downstream
VFM Outputs
ViT
80.80 ± 3.35
96.80 ± 3.35
100.00 ± 0.00
100.00 ± 0.00
99.20 ± 1.79
100.00 ± 0.00
99.20 ± 1.79
88.80 ± 3.03
99.68 ± 0.44
96.57 ± 0.81
AFO (Student)
Downstream
VFM Outputs
CNN
79.20 ± 3.35
84.00 ± 2.83
92.00 ± 4.90
96.80 ± 1.79
96.00 ± 4.90
100.00 ± 0.00
100.00 ± 0.00
81.60 ± 1.67
96.96 ± 1.54
92.57 ± 1.51
Table B4 : Per-task results for the comparison with the Theia distillation objective. Mean success rate and standard deviation ( ± ) across 5 seeds.
Distillation Configuration
Teacher
Adroit
MetaWorld
Average
Loss
Alignment
λ
Segment.
Depth
Pen
Relocate
Assembly
Bin
Button
Drawer
Hammer
Adroit
MetaWorld
7 tasks
BC-only from scratch (LfS), no distillation
74.40 ± 7.27
48.80 ± 4.38
76.00 ± 15.75
96.80 ± 3.35
96.00 ± 4.90
100.00 ± 0.00
100.00 ± 0.00
61.60 ± 3.58
93.76 ± 2.85
84.57 ± 2.02
ℓ2
per-modality
dynamic
SAM
DAM
79.20 ± 7.69
85.60 ± 2.19
82.40 ± 9.21
100.00 ± 0.00
96.80 ± 5.22
100.00 ± 0.00
100.00 ± 0.00
82.40 ± 2.97
95.84 ± 2.49
92.00 ± 2.49
cos
fused
dynamic
SAM
DAM
74.40 ± 5.37
83.20 ± 3.35
91.20 ± 7.16
99.20 ± 1.79
96.00 ± 2.83
100.00 ± 0.00
100.00 ± 0.00
78.80 ± 3.63
97.28 ± 1.56
92.00 ± 1.46
cos
per-modality
fixed 1.0
SAM
DAM
76.00 ± 2.83
82.40 ± 8.29
87.20 ± 9.55
98.40 ± 2.19
96.00 ± 4.90
100.00 ± 0.00
100.00 ± 0.00
79.20 ± 5.40
96.32 ± 2.16
91.43 ± 2.10
cos
per-modality
dynamic
–
DAM
74.40 ± 6.07
68.00 ± 9.38
88.00 ± 9.38
98.40 ± 2.19
96.00 ± 4.00
100.00 ± 0.00
100.00 ± 0.00
71.20 ± 6.72
96.48 ± 2.37
89.26 ± 3.49
Table B5 : Per-task results for the ablation study. Mean success rate and standard deviation ( ± ) across 5 seeds.
Figure C1 : Qualitative outputs of Depth Anything v2 and Segment Anything v2 across tasks. Each row shows (from left) the RGB input, depth map (normalized to [0,1] ), and segmentation mask overlay. Both successful predictions and failure cases are illustrated.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Robotic foundation models still need task-specific fine-tuning before deployment, and the fine-tuned policies often break under modest changes in scene layout, lighting, or nearby distractors. We trace this brittleness to shortcut learning: fine-tuning supervises actions but not the visual evidence the policy uses, so the policy can settle on scene-level correlations that predict the demonstrations without causing success. We propose Artificial Foveated Perception (AFP), a lightweight, policy-agnostic module that takes the same vision and language inputs as existing Vision-Language-Action and World Action Model pipelines and predicts task-conditioned masks over the relevant objects, the robot, and other action-critical regions. During fine-tuning the masks serve as an auxiliary grounding signal that aligns the policy's visual attention with task-relevant regions; the policy architecture is unchanged, and at inference the policy runs on the original observation stream with no AFP call in the control loop. In simulation with four robotic foundation models and on a real robot with π0.5, AFP improves generalization under environmental perturbations, reduces overfitting, and shortens fine-tuning. Ablations over mask quality and grounding-loss design show that these gains come from directing policy learning toward task-relevant visual evidence. Code, data, and videos are available at https://apollo-lab-yale.github.io/26-CoRL-AFP-website/.
Vision-language-action (VLA) models and world-action models (WAMs) have emerged as promising approaches to robot learning and control. These models integrate multimodal perception and action generation, but differ in their architectures, output representations, and learning objectives. In this paper, we present a unified, architecture-agnostic probabilistic formulation for embodied foundation models based on stochastic kernels. Our formulation has two key components: (i) we characterize embodied foundation models through conditional output distributions and action-realization kernels, which together induce task-conditioned robot policies; and (ii) we connect pretraining and post-training to policy deployment, accommodating model-specific pretraining objectives and offline or online post-training strategies. We instantiate the formulation for OpenVLA and DreamZero, covering autoregressive action-token prediction and joint video-action generation.
Xinyi Hong, Mingyang Ren
Department of Statistics, School of Mathematical Sciences, Shanghai Jiao Tong University
Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and controllability in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of task-aware slots for robotic manipulation. Rather than fully tuning large backbones on the task, STORM employs an efficient two-stage training strategy: few layers of object-centric representation are first trained on top of the frozen backbone through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy for task alignement. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery benchmarks and robotic manipulation tasks show that STORM improves control performance and generalization to visual shifts (distractors, textures, lighting) compared to directly using frozen or fine-tuned foundation model features, or existing object-centric representations. STORM serves not only as an efficient mechanism for refining generic foundation model features, but also as a novel way of injecting beneficial structural and semantic bias into policy learning.