FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

Filip Grigorov; Kourosh Darvish; Nandita Vijaykumar

cs.ROOct 8, 2026

FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

Authors: Filip Grigorov, Kourosh Darvish, Nandita Vijaykumar

Organizations: University of Toronto Canada

Abstract

Vision-based reinforcement learning for robotic manipulation is sample-inefficient because RGB-D observations are high-dimensional and noisy. Privileged state information available in simulation can accelerate training, but its absence at test time creates a train-test modality gap. We propose FOCUS, a single-stage PPO framework that trains the critic on privileged state while automatically regulating whether the actor collects rollouts from RGB-D or privileged-state latents. Regulation is driven by the KL divergence between the action distributions induced by the two modalities, while representation alignment encourages consistent action selection across them. Together, these mechanisms limit RGB-D rollouts when the actor's action distributions from RGB-D and privileged-state latents disagree. As they align, RGB-D exposure increases, shifting on-policy training toward the RGB-D inputs used at test time. Across five manipulation tasks, FOCUS raises average test success from 0.71 to 0.93 relative to the strongest RGB-D-at-test baseline on each task. When accounting for each method's complete training pipeline, budget-normalized training-success AUC increases from 0.47 to 0.65. On Pick-and-Place, test success rises from 0.47 to 0.86, while AUC increases from 0.12 to 0.61, a 5.0x improvement in learning efficiency over the fixed interaction budget.

Figures & tables

Figure 1: Overview of FOCUS. During training, cross-modal policy disagreement is aggregated into running statistics that determine the RGB-D exposure probability

\lambda_{r}

for subsequent actor rollouts. A KL-guided rollout switch samples either the privileged-state or RGB-D pathway and keeps the selected pathway fixed throughout the rollout, while the critic remains anchored to privileged state. At inference, privileged state and the switching mechanism are removed, and the actor operates only from RGB-D, proprioception, and the previous action.

Panel of Figure 2 — Figure 2: Representative successful test-time rollouts of FOCUS. Sequential frames show FOCUS executing Lift-Cube (top), Push-Cube (middle), and Pick-and-Place (bottom) using the deployment observation pathway. At test time, privileged state and modality switching are removed, and the policy acts from RGB-D observations, proprioception, and the previous action. Frames are labeled by normalized rollout time $t/T$ .

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place †
states-only	1.000 $\pm$ 0.000	0.555 $\pm$ 0.628	0.972 $\pm$ 0.040	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000
RGB-D-only	1.000 $\pm$ 0.000	0.233 $\pm$ 0.024	0.753 $\pm$ 0.098	0.983 $\pm$ 0.024	0.453 $\pm$ 0.122
RGB-D-asymmetric	1.000 $\pm$ 0.000	0.350 $\pm$ 0.338	0.192 $\pm$ 0.059	0.811 $\pm$ 0.157	0.467 $\pm$ 0.660
teacher-student (3-stage, final student)	1.000 $\pm$ 0.000	0.181 $\pm$ 0.122	0.595 $\pm$ 0.032	0.978 $\pm$ 0.031	0.289 $\pm$ 0.110
FOCUS (ours)	1.000 $\pm$ 0.000	0.975 $\pm$ 0.012	0.850 $\pm$ 0.071	0.978 $\pm$ 0.018	0.858 $\pm$ 0.069

Table 1: Test episode success rate over a 9k-step evaluation rollout. An episode is counted as successful if the task succeeds at least once before termination. Higher is better.

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place
states-only	0.966 $\pm$ 0.002	0.211 $\pm$ 0.268	0.447 $\pm$ 0.007	0.912 $\pm$ 0.016	0.656 $\pm$ 0.009
RGB-D-only	0.889 $\pm$ 0.003	0.023 $\pm$ 0.002	0.288 $\pm$ 0.036	0.859 $\pm$ 0.021	0.094 $\pm$ 0.056
RGB-D-asymmetric	0.891 $\pm$ 0.020	0.158 $\pm$ 0.190	0.094 $\pm$ 0.003	0.886 $\pm$ 0.015	0.122 $\pm$ 0.171
teacher-student (final student)	0.958 $\pm$ 0.010	0.258 $\pm$ 0.321	0.233 $\pm$ 0.012	0.930 $\pm$ 0.013	0.155 $\pm$ 0.003
teacher-student (3-stage, e2e)	0.411 $\pm$ 0.004	0.111 $\pm$ 0.137	0.100 $\pm$ 0.005	0.399 $\pm$ 0.006	0.071 $\pm$ 0.001
FOCUS (ours)	0.969 $\pm$ 0.001	0.451 $\pm$ 0.040	0.337 $\pm$ 0.167	0.864 $\pm$ 0.025	0.612 $\pm$ 0.037

Table 2: Budget-normalized Training Success AUC summarizes sample efficiency over the full training trajectory. AUC is divided by each method’s task-specific simulator-interaction horizon. For teacher-student, we report both the final student phase alone as a diagnostic and the comparable end-to-end 3-stage cost used in our conclusions. Higher is better.

Panel of Figure 3 — Figure 3: Training success EMA on the more discriminative tasks. Teacher–student curves show the final student PPO phase, while Table 2 reports end-to-end normalized AUC. Full curves and Step@50 are provided in Appendices F and E , respectively.

Variant	Test success			Norm. Training Success AUC
	Push	Lift	Pick - and - Place †	Push	Lift	Pick - and - Place
FOCUS	$0.850$ [-2pt] $\pm\,0.071$	$0.975$ [-2pt] $\pm\,0.012$	$0.858$ [-2pt] $\pm\,0.069$	$0.337$ [-2pt] $\pm\,0.167$	$0.451$ [-2pt] $\pm\,0.040$	$0.612$ [-2pt] $\pm\,0.037$
Training-objective ablations
w/o decoder	$0.669$ [-2pt] $\pm\,0.090$	$0.986$ [-2pt] $\pm\,0.020$	$0.997$ [-2pt] $\pm\,0.003$	$0.406$ [-2pt] $\pm\,0.104$	$0.472$ [-2pt] $\pm\,0.003$	$0.666$ [-2pt] $\pm\,0.080$
w/o aligner ( $f_{\varphi}$ bypass)	$0.854$ [-2pt] $\pm\,0.038$	$0.964$ [-2pt] $\pm\,0.011$	$0.458$ [-2pt] $\pm\,0.458$	$0.445$ [-2pt] $\pm\,0.013$	$0.464$ [-2pt] $\pm\,0.030$	$0.283$ [-2pt] $\pm\,0.283$
w/o RL agreement	$0.856$ [-2pt] $\pm\,0.032$	$0.950$ [-2pt] $\pm\,0.063$	$0.822$ [-2pt] $\pm\,0.050$	$0.345$ [-2pt] $\pm\,0.063$	$0.415$ [-2pt] $\pm\,0.102$	$0.631$ [-2pt] $\pm\,0.064$

Table 3: Ablations on the three more discriminative tasks. All variants use the same training budget for Push/Lift/PnP (1500/1500/2500 training iterations), are trained with two seeds, and are evaluated with five test seeds. AUC is budget-normalized by each task horizon. Higher is better for test success and normalized AUC.

Figure 4: Representative Push-Cube switch diagnostic. The KL-guided switch reduces exposure-weighted upper-tail mismatch,

\lambda_{r}\cdot\mathrm{KL}_{p90}

, relative to a fixed ramp and a variant without the upper-quantile budget, while improving training success. Here,

\mathrm{KL}_{p90}

denotes the 90th percentile of cross-modal policy KL, and

\lambda_{r}

is the Bernoulli probability that the actor uses RGB-D latents for an entire rollout.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place
states-only	18.5 $\pm$ 6.4 (2/2)	693 (1/2)	556.5 $\pm$ 27.6 (2/2)	31.5 $\pm$ 3.5 (2/2)	522.0 $\pm$ 46.7 (2/2)
RGB-D-only	15.5 $\pm$ 9.2 (2/2)	Not reached (0/2)	1447.5 $\pm$ 20.5 (2/2)	51.5 $\pm$ 4.9 (2/2)	Not reached (0/2)
RGB-D-asymmetric	12.5 $\pm$ 5.0 (2/2)	1098 (1/2)	Not reached (0/2)	50.0 $\pm$ 4.2 (2/2)	1611 (1/2)
teacher-student (final student)	1.0 $\pm$ 1.4 (2/2)	262 (1/2)	Not reached (0/2)	5.0 $\pm$ 5.7 (2/2)	Not reached (0/2)
teacher-student (3-stage, e2e)	2001.0 $\pm$ 1.4 (2/2)	2262 (1/2)	Not reached (0/2)	2005.0 $\pm$ 5.7 (2/2)	Not reached (0/2)
FOCUS (ours)	11.5 $\pm$ 3.5 (2/2)	461.5 $\pm$ 112.4 (2/2)	461.5 $\pm$ 19.1 (2/2)	91.5 $\pm$ 20.5 (2/2)	470.0 $\pm$ 107.5 (2/2)

Appendix

Table 4: Train Step@50% success occupancy reports the first training point at which success occupancy reaches 50%. Lower is better. Parentheses indicate the number of training seeds that reached the threshold, e.g., 693 (1/2). “Not reached” means no seed reached the threshold. For teacher-student, “final student” reports final fine-tuning, while “3-stage, e2e” counts the full pipeline.

Panel of Figure 5 — Figure 5: Full training success EMA curves for all tasks. Curves show training-time success. For teacher-student, curves show the final student PPO phase; Table 2 reports end-to-end normalized AUC in the main paper.

Panel of Figure 6 — Figure 6: Per-episode training success rate for all tasks. Earlier rises and higher plateaus indicate better sample efficiency.

Panel of Figure 7 — Figure 7: Encoder activation visualizations (channel-averaged intermediate feature magnitudes) before and after image exposure for Lift, Push, and Open-Drawer.

Panel of Figure 8 — Figure 8: RGB-D encoder activation probe (channel-averaged intermediate feature magnitudes) before and after image exposure for Lift, Push, Pick-and-Place, and Open-Drawer. These plots are qualitative diagnostics of the learned visual representation.

Panel of Figure 9 — Figure 9: t-SNE projections of state and image latent representations before and after alignment for Lift, Push, Pick-and-Place, and Open-Drawer. These plots are qualitative and are included only as supporting diagnostics.

Training and PPO defaults
Latent dimension $d$	$256$
PPO epochs	$1$
PPO mini-batches	$1$
Clip parameter $\epsilon$	$0.2$
Discount $\gamma$	$0.998$
GAE parameter $\lambda$	$0.95$

Appendix

Table 5: Main default hyperparameters for FOCUS unless otherwise noted.

Additional PPO / optimization details
Value loss weight $\alpha_{\text{val}}$	$1.0$
Entropy weight $\beta_{\text{ent}}$	$0.0$
Clipped value loss	enabled
PPO KL target $\kappa_{\mathrm{PPO}}$	$0.01$ when adaptive LR scheduling is enabled
RL optimizer details	Adam, betas $(0.9,0.999)$ , weight decay $0$
Representation optimizer details	Adam, betas $(0.9,0.999)$ , weight decay $10^{-5}$

Appendix

Table 6: Additional implementation defaults complementing Table 5 .

Architecture summary (latent dim $d{=}256$ )
State encoder $E_{s}$	Linear $(\text{states}\!\rightarrow\!d)$ $\rightarrow$ Swish $\rightarrow$ LayerNorm.
RGB-D encoder $E_{o}$	Shared-trunk ResNet-18 RGB-D encoder. Separate RGB/depth stems (RGB Conv1 from pretrained ResNet-18 [ 4 ] ; depth Conv1 initialized from averaged RGB weights, with optional validity-mask channel) $\rightarrow$ summed stems $\rightarrow$ shared ResNet-18 trunk (BN/ReLU/MaxPool, Layer1–Layer4) $\rightarrow$ ECA channel attention [ 29 ] (+ optional CBAM-style SpatialGate [ 31 ] ) $\rightarrow$ AdaptiveAvgPool2d $(1,1)$ $\rightarrow$ Linear $(512\!\rightarrow\!d)$ $\rightarrow$ LayerNorm (projection: Identity).
Fusion $g_{\psi}$	Residual fusion of a chosen latent $z\in\mathbb{R}^{d}$ with proprioception and previous action: concat $([z;\mathbf{s}^{prop}_{t};\mathbf{a}_{t-1}])$ $\rightarrow$ LayerNorm $\rightarrow$ Linear $(m\!\rightarrow\!d)$ (zero-init) $\rightarrow$ residual add $z{+}\Delta$ . ( Same fusion block used for actor and critic inputs; critic always uses $z_{t,s}$ .)
Aligner $f_{\varphi}$	Symmetric cross-attention aligner operating on raw latents $(\mathbf{z}_{t,s},\mathbf{z}_{t,o})$ (2 heads). Per stream: CrossAttn $(q{=}z,\,kv{=}z_{\text{other}})$ $\rightarrow$ concat $([z;\mathrm{att}])$ $\rightarrow$ LayerNorm $\rightarrow$ Linear $(2d\!\rightarrow\!d)$ $\rightarrow$ gated residual update.
State decoder $D_{s}$	Residual MLP decoder from raw latent to privileged-state target: $[d\!\rightarrow\!512\!\rightarrow\!1024\!\rightarrow\!1024\!\rightarrow\!512\!\rightarrow\!d_{\text{state}}]$ with Swish activations, Dropout $(0.1)$ after the first layer, final LayerNorm, and residual Linear $(d\!\rightarrow\!d_{\text{state}})$ .
Scratch CNN $E_{o}$ ablation	For the “RGB-D encoder from scratch” ablation, the ResNet-based $E_{o}$ is replaced by a lightweight two-branch RGB-D CNN. The RGB and depth streams each use three stride-2 Conv–Swish blocks with channels $32,64,128$ , followed by AdaptiveAvgPool2d $(4,4)$ , flattening, Linear $(\cdot\!\rightarrow\!d)$ , and LayerNorm. The two $d$ -dimensional branch features are concatenated and passed through a two-layer MLP to produce the final $d$ -dimensional image latent. The same RGB-D preprocessing contract is used as in the ResNet-based encoder.

Appendix

Table 7: Torch-level architecture components used in the main experiments (latent dim

d{=}256

Signal (TB tag)	Phase	What it captures
Performance & sample efficiency
Train/success_rate_ema	train	smoothed training success vs. env steps
Train/steps_to_50pct	train	Steps@50 (time-to-threshold)
Test/Success_rate	test	evaluation success rate over episodes/seeds
Test/rewards_mean	test	mean episode return (task-specific scale)
PPO update stability

Appendix

Table 8: Train/test metrics and additional diagnostics logged by our protocol.

Explore similar work

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation
May 28, 2026Victor Kowalski, Chengxi Li, Dongheui LeeCross-Modal Knowledge Distillation Robot Skill Learning
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Jun 17, 2026Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik +2RL for Robotics Reinforcement Learning
Department of Computing Science, University of Alberta · National Research Council Canada · School of Electrical Engineering and Computer Science, University of Ottawa +3
GeoProp: Grounding Robot State in Vision for Generalist Manipulation
Jul 8, 2026Guoyang Zhao, Quanhao Qian, Gongjie Zhang +5Generalization in Robotic Manipulation Visuomotor Policy Learning
Tongji University · DAMO Academy, Alibaba Group · HuPan Lab +2

May 28, 2026, cs.ROJ/K move · Enter open · S save

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

Victor Kowalski, Chengxi Li, Dongheui Lee

Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

Cross-Modal Knowledge Distillation Robot Skill Learning Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation Robotic RL Robot Policy Generalization+2

Open paper AskPDF arXiv

When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.

Abstract

Explore similar work

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

GeoProp: Grounding Robot State in Vision for Generalist Manipulation