FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

Filip Grigorov; Kourosh Darvish; Nandita Vijaykumar

cs.ROOct 8, 2026

FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

Authors: Filip Grigorov, Kourosh Darvish, Nandita Vijaykumar

Organizations: University of Toronto Canada

Abstract

Vision-based reinforcement learning for robotic manipulation is sample-inefficient because RGB-D observations are high-dimensional and noisy. Privileged state information available in simulation can accelerate training, but its absence at test time creates a train-test modality gap. We propose FOCUS, a single-stage PPO framework that trains the critic on privileged state while automatically regulating whether the actor collects rollouts from RGB-D or privileged-state latents. Regulation is driven by the KL divergence between the action distributions induced by the two modalities, while representation alignment encourages consistent action selection across them. Together, these mechanisms limit RGB-D rollouts when the actor's action distributions from RGB-D and privileged-state latents disagree. As they align, RGB-D exposure increases, shifting on-policy training toward the RGB-D inputs used at test time. Across five manipulation tasks, FOCUS raises average test success from 0.71 to 0.93 relative to the strongest RGB-D-at-test baseline on each task. When accounting for each method's complete training pipeline, budget-normalized training-success AUC increases from 0.47 to 0.65. On Pick-and-Place, test success rises from 0.47 to 0.86, while AUC increases from 0.12 to 0.61, a 5.0x improvement in learning efficiency over the fixed interaction budget.

Figures & tables

Figure 1: Overview of FOCUS. During training, cross-modal policy disagreement is aggregated into running statistics that determine the RGB-D exposure probability

\lambda_{r}

for subsequent actor rollouts. A KL-guided rollout switch samples either the privileged-state or RGB-D pathway and keeps the selected pathway fixed throughout the rollout, while the critic remains anchored to privileged state. At inference, privileged state and the switching mechanism are removed, and the actor operates only from RGB-D, proprioception, and the previous action.

Panel of Figure 2 — Figure 2: Representative successful test-time rollouts of FOCUS. Sequential frames show FOCUS executing Lift-Cube (top), Push-Cube (middle), and Pick-and-Place (bottom) using the deployment observation pathway. At test time, privileged state and modality switching are removed, and the policy acts from RGB-D observations, proprioception, and the previous action. Frames are labeled by normalized rollout time $t/T$ .

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place †
states-only	1.000 $\pm$ 0.000	0.555 $\pm$ 0.628	0.972 $\pm$ 0.040	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000
RGB-D-only	1.000 $\pm$ 0.000	0.233 $\pm$ 0.024	0.753 $\pm$ 0.098	0.983 $\pm$ 0.024	0.453 $\pm$ 0.122
RGB-D-asymmetric	1.000 $\pm$ 0.000	0.350 $\pm$ 0.338	0.192 $\pm$ 0.059	0.811 $\pm$ 0.157	0.467 $\pm$ 0.660
teacher-student (3-stage, final student)	1.000 $\pm$ 0.000	0.181 $\pm$ 0.122	0.595 $\pm$ 0.032	0.978 $\pm$ 0.031	0.289 $\pm$ 0.110
FOCUS (ours)	1.000 $\pm$ 0.000	0.975 $\pm$ 0.012	0.850 $\pm$ 0.071	0.978 $\pm$ 0.018	0.858 $\pm$ 0.069

Table 1: Test episode success rate over a 9k-step evaluation rollout. An episode is counted as successful if the task succeeds at least once before termination. Higher is better.

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place
states-only	0.966 $\pm$ 0.002	0.211 $\pm$ 0.268	0.447 $\pm$ 0.007	0.912 $\pm$ 0.016	0.656 $\pm$ 0.009
RGB-D-only	0.889 $\pm$ 0.003	0.023 $\pm$ 0.002	0.288 $\pm$ 0.036	0.859 $\pm$ 0.021	0.094 $\pm$ 0.056
RGB-D-asymmetric	0.891 $\pm$ 0.020	0.158 $\pm$ 0.190	0.094 $\pm$ 0.003	0.886 $\pm$ 0.015	0.122 $\pm$ 0.171
teacher-student (final student)	0.958 $\pm$ 0.010	0.258 $\pm$ 0.321	0.233 $\pm$ 0.012	0.930 $\pm$ 0.013	0.155 $\pm$ 0.003
teacher-student (3-stage, e2e)	0.411 $\pm$ 0.004	0.111 $\pm$ 0.137	0.100 $\pm$ 0.005	0.399 $\pm$ 0.006	0.071 $\pm$ 0.001
FOCUS (ours)	0.969 $\pm$ 0.001	0.451 $\pm$ 0.040	0.337 $\pm$ 0.167	0.864 $\pm$ 0.025	0.612 $\pm$ 0.037

Table 2: Budget-normalized Training Success AUC summarizes sample efficiency over the full training trajectory. AUC is divided by each method’s task-specific simulator-interaction horizon. For teacher-student, we report both the final student phase alone as a diagnostic and the comparable end-to-end 3-stage cost used in our conclusions. Higher is better.

Panel of Figure 3 — Figure 3: Training success EMA on the more discriminative tasks. Teacher–student curves show the final student PPO phase, while Table 2 reports end-to-end normalized AUC. Full curves and Step@50 are provided in Appendices F and E , respectively.

Variant	Test success			Norm. Training Success AUC
	Push	Lift	Pick - and - Place †	Push	Lift	Pick - and - Place
FOCUS	$0.850$ [-2pt] $\pm\,0.071$	$0.975$ [-2pt] $\pm\,0.012$	$0.858$ [-2pt] $\pm\,0.069$	$0.337$ [-2pt] $\pm\,0.167$	$0.451$ [-2pt] $\pm\,0.040$	$0.612$ [-2pt] $\pm\,0.037$
Training-objective ablations
w/o decoder	$0.669$ [-2pt] $\pm\,0.090$	$0.986$ [-2pt] $\pm\,0.020$	$0.997$ [-2pt] $\pm\,0.003$	$0.406$ [-2pt] $\pm\,0.104$	$0.472$ [-2pt] $\pm\,0.003$	$0.666$ [-2pt] $\pm\,0.080$
w/o aligner ( $f_{\varphi}$ bypass)	$0.854$ [-2pt] $\pm\,0.038$	$0.964$ [-2pt] $\pm\,0.011$	$0.458$ [-2pt] $\pm\,0.458$	$0.445$ [-2pt] $\pm\,0.013$	$0.464$ [-2pt] $\pm\,0.030$	$0.283$ [-2pt] $\pm\,0.283$
w/o RL agreement	$0.856$ [-2pt] $\pm\,0.032$	$0.950$ [-2pt] $\pm\,0.063$	$0.822$ [-2pt] $\pm\,0.050$	$0.345$ [-2pt] $\pm\,0.063$	$0.415$ [-2pt] $\pm\,0.102$	$0.631$ [-2pt] $\pm\,0.064$

Table 3: Ablations on the three more discriminative tasks. All variants use the same training budget for Push/Lift/PnP (1500/1500/2500 training iterations), are trained with two seeds, and are evaluated with five test seeds. AUC is budget-normalized by each task horizon. Higher is better for test success and normalized AUC.

Figure 4: Representative Push-Cube switch diagnostic. The KL-guided switch reduces exposure-weighted upper-tail mismatch,

\lambda_{r}\cdot\mathrm{KL}_{p90}

, relative to a fixed ramp and a variant without the upper-quantile budget, while improving training success. Here,

\mathrm{KL}_{p90}

denotes the 90th percentile of cross-modal policy KL, and

\lambda_{r}

is the Bernoulli probability that the actor uses RGB-D latents for an entire rollout.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Reach	Lift	Push	Open - Drawer	Pick - and - Place
states-only	18.5 $\pm$ 6.4 (2/2)	693 (1/2)	556.5 $\pm$ 27.6 (2/2)	31.5 $\pm$ 3.5 (2/2)	522.0 $\pm$ 46.7 (2/2)
RGB-D-only	15.5 $\pm$ 9.2 (2/2)	Not reached (0/2)	1447.5 $\pm$ 20.5 (2/2)	51.5 $\pm$ 4.9 (2/2)	Not reached (0/2)
RGB-D-asymmetric	12.5 $\pm$ 5.0 (2/2)	1098 (1/2)	Not reached (0/2)	50.0 $\pm$ 4.2 (2/2)	1611 (1/2)
teacher-student (final student)	1.0 $\pm$ 1.4 (2/2)	262 (1/2)	Not reached (0/2)	5.0 $\pm$ 5.7 (2/2)	Not reached (0/2)
teacher-student (3-stage, e2e)	2001.0 $\pm$ 1.4 (2/2)	2262 (1/2)	Not reached (0/2)	2005.0 $\pm$ 5.7 (2/2)	Not reached (0/2)
FOCUS (ours)	11.5 $\pm$ 3.5 (2/2)	461.5 $\pm$ 112.4 (2/2)	461.5 $\pm$ 19.1 (2/2)	91.5 $\pm$ 20.5 (2/2)	470.0 $\pm$ 107.5 (2/2)

Appendix

Table 4: Train Step@50% success occupancy reports the first training point at which success occupancy reaches 50%. Lower is better. Parentheses indicate the number of training seeds that reached the threshold, e.g., 693 (1/2). “Not reached” means no seed reached the threshold. For teacher-student, “final student” reports final fine-tuning, while “3-stage, e2e” counts the full pipeline.

Panel of Figure 5 — Figure 5: Full training success EMA curves for all tasks. Curves show training-time success. For teacher-student, curves show the final student PPO phase; Table 2 reports end-to-end normalized AUC in the main paper.

Panel of Figure 6 — Figure 6: Per-episode training success rate for all tasks. Earlier rises and higher plateaus indicate better sample efficiency.

Panel of Figure 7 — Figure 7: Encoder activation visualizations (channel-averaged intermediate feature magnitudes) before and after image exposure for Lift, Push, and Open-Drawer.

Panel of Figure 8 — Figure 8: RGB-D encoder activation probe (channel-averaged intermediate feature magnitudes) before and after image exposure for Lift, Push, Pick-and-Place, and Open-Drawer. These plots are qualitative diagnostics of the learned visual representation.

Panel of Figure 9 — Figure 9: t-SNE projections of state and image latent representations before and after alignment for Lift, Push, Pick-and-Place, and Open-Drawer. These plots are qualitative and are included only as supporting diagnostics.

Training and PPO defaults
Latent dimension $d$	$256$
PPO epochs	$1$
PPO mini-batches	$1$
Clip parameter $\epsilon$	$0.2$
Discount $\gamma$	$0.998$
GAE parameter $\lambda$	$0.95$

Appendix

Table 5: Main default hyperparameters for FOCUS unless otherwise noted.

Additional PPO / optimization details
Value loss weight $\alpha_{\text{val}}$	$1.0$
Entropy weight $\beta_{\text{ent}}$	$0.0$
Clipped value loss	enabled
PPO KL target $\kappa_{\mathrm{PPO}}$	$0.01$ when adaptive LR scheduling is enabled
RL optimizer details	Adam, betas $(0.9,0.999)$ , weight decay $0$
Representation optimizer details	Adam, betas $(0.9,0.999)$ , weight decay $10^{-5}$

Appendix

Table 6: Additional implementation defaults complementing Table 5 .

Architecture summary (latent dim $d{=}256$ )
State encoder $E_{s}$	Linear $(\text{states}\!\rightarrow\!d)$ $\rightarrow$ Swish $\rightarrow$ LayerNorm.
RGB-D encoder $E_{o}$	Shared-trunk ResNet-18 RGB-D encoder. Separate RGB/depth stems (RGB Conv1 from pretrained ResNet-18 [ 4 ] ; depth Conv1 initialized from averaged RGB weights, with optional validity-mask channel) $\rightarrow$ summed stems $\rightarrow$ shared ResNet-18 trunk (BN/ReLU/MaxPool, Layer1–Layer4) $\rightarrow$ ECA channel attention [ 29 ] (+ optional CBAM-style SpatialGate [ 31 ] ) $\rightarrow$ AdaptiveAvgPool2d $(1,1)$ $\rightarrow$ Linear $(512\!\rightarrow\!d)$ $\rightarrow$ LayerNorm (projection: Identity).
Fusion $g_{\psi}$	Residual fusion of a chosen latent $z\in\mathbb{R}^{d}$ with proprioception and previous action: concat $([z;\mathbf{s}^{prop}_{t};\mathbf{a}_{t-1}])$ $\rightarrow$ LayerNorm $\rightarrow$ Linear $(m\!\rightarrow\!d)$ (zero-init) $\rightarrow$ residual add $z{+}\Delta$ . ( Same fusion block used for actor and critic inputs; critic always uses $z_{t,s}$ .)
Aligner $f_{\varphi}$	Symmetric cross-attention aligner operating on raw latents $(\mathbf{z}_{t,s},\mathbf{z}_{t,o})$ (2 heads). Per stream: CrossAttn $(q{=}z,\,kv{=}z_{\text{other}})$ $\rightarrow$ concat $([z;\mathrm{att}])$ $\rightarrow$ LayerNorm $\rightarrow$ Linear $(2d\!\rightarrow\!d)$ $\rightarrow$ gated residual update.
State decoder $D_{s}$	Residual MLP decoder from raw latent to privileged-state target: $[d\!\rightarrow\!512\!\rightarrow\!1024\!\rightarrow\!1024\!\rightarrow\!512\!\rightarrow\!d_{\text{state}}]$ with Swish activations, Dropout $(0.1)$ after the first layer, final LayerNorm, and residual Linear $(d\!\rightarrow\!d_{\text{state}})$ .
Scratch CNN $E_{o}$ ablation	For the “RGB-D encoder from scratch” ablation, the ResNet-based $E_{o}$ is replaced by a lightweight two-branch RGB-D CNN. The RGB and depth streams each use three stride-2 Conv–Swish blocks with channels $32,64,128$ , followed by AdaptiveAvgPool2d $(4,4)$ , flattening, Linear $(\cdot\!\rightarrow\!d)$ , and LayerNorm. The two $d$ -dimensional branch features are concatenated and passed through a two-layer MLP to produce the final $d$ -dimensional image latent. The same RGB-D preprocessing contract is used as in the ResNet-based encoder.

Appendix

Table 7: Torch-level architecture components used in the main experiments (latent dim

d{=}256

Signal (TB tag)	Phase	What it captures
Performance & sample efficiency
Train/success_rate_ema	train	smoothed training success vs. env steps
Train/steps_to_50pct	train	Steps@50 (time-to-threshold)
Test/Success_rate	test	evaluation success rate over episodes/seeds
Test/rewards_mean	test	mean episode return (task-specific scale)
PPO update stability

Appendix

Table 8: Train/test metrics and additional diagnostics logged by our protocol.

Explore similar work

May 28, 2026cs.RO

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.

Victor Kowalski, Chengxi Li, Dongheui Lee

Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

Cross-Modal Knowledge Distillation Robot Skill Learning Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation Robotic RL Robot Policy Generalization+2

Ask PDF

Jun 17, 2026cs.RO

Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.

Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik +2

Department of Computing Science, University of Alberta · National Research Council Canada · School of Electrical Engineering and Computer Science, University of Ottawa +3

RL for Robotics Reinforcement Learning Robotic Manipulation Vision-Based Robot Control

Ask PDF

Jul 8, 2026cs.RO

GeoProp: Grounding Robot State in Vision for Generalist Manipulation

Proprioception is fundamental to robotic manipulation, yet standard fusion methods often treat it as an isolated vector lacking explicit alignment with visual tokens. Without a direct correspondence between 3D kinematics and 2D feature maps, manipulation policies struggle to ground the robot's state within the scene, frequently underperforming even vision-only baselines. To address this, we introduce GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp projects the robot state onto the image plane to sample localized visual features, constructing a grounded state token. It then injects state-derived spatial priors into the corresponding visual features via FiLM modulation. To capture motion intent, GeoProp further samples features at a short-horizon predicted coordinate derived from recent kinematics, providing look-ahead visual context. Across 67 tasks, GeoProp improves Diffusion Policy by 8.7% on 63 simulation tasks and pi_0 by 4.0% on the RoboTwin subset, and yields a 10.6% average gain across both policy families in the real world, while adding only 2-3% to the parameter count. These results demonstrate that GeoProp is a simple yet high-impact inductive bias for generalist embodied policies. Project page: https://alibaba-damo-academy.github.io/GeoProp/.

Guoyang Zhao, Quanhao Qian, Gongjie Zhang +5

Tongji University · DAMO Academy, Alibaba Group · HuPan Lab +2

Generalization in Robotic Manipulation Visuomotor Policy Learning Multi-Task Robotic Manipulation Robotic Manipulation

Ask PDF