RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation

Han Qi; Heng Yang

cs.ROSep 22, 2026

RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation

Authors: Han Qi, Heng Yang

Organizations: Harvard School of Engineering and Applied Sciences Harvard University

Abstract

Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action. General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context. We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters. It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction. We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR. Across seven simulated manipulation settings, RoboMP-DINOv2 achieves 60.7% success under spatial shifts and 59.7% under scene clutter, compared with 50.7% and 41.0% for a DINOv2-based Diffusion Policy. Under unseen object colors, RoboMP-DINOv2-MCR achieves 72.5% success versus 35.1% for the strongest color-randomized baseline. Additional experiments and representation analyses show improved robustness while preserving behaviorally relevant scene information. Code is available at https://github.com/han20192019/RoboMP_DINOv2.

Figures & tables

Figure 1: Seven simulated manipulation settings under the three evaluation regimes. Columns denote task settings, and rows show representative observations from Clean Spatial OOD, Clutter OOD, and Appearance OOD.

Method	Place cube	Put can (clear)	Put can (obstacle)	Pull tool	Peg insertion	Long horizon	Two- arm	Avg.
Clean Spatial OOD
2D object-centric masking	.350	.190	.000	.000	.020	.007	.000	.081
3D object-centric masking	.850	.840	.070	.000	.085	.036	.120	.286
DP–DINOv2	.410	.820	.750	.150	.170	.812	.440	.507
RoboGround–DINOv2	.690	.820	.770	.000	.215	.789	.630	.559
RoboMP-DINOv2	.540	.860	.830	.170	.230	.848	.770	.607

Table 1: Success rates under held-out spatial configurations and scene clutter. All methods are trained on demonstrations collected in visually clean scenes. RoboMP-DINOv2 denotes the proposed mask-prompted full-scene encoder without masked-region color randomization. “Avg.” is the macro-average over the seven settings. The highest observed performance in each column and regime is shown in bold.

Figure 2: Matched architecture comparison under cluttered-scene evaluation. Bars show mean success over three training seeds evaluated on the same 50 initial conditions; error bars show the standard deviation across seeds. All variants produce a 256-dimensional visual representation and use the same downstream policy. All-one masks retain prompting but remove spatial localization. No mask prompting removes the additive prompt term while retaining the remaining RoboMP-DINOv2 architecture. DINOv2 CLS+mean uses the global DINOv2 class token and mean-pooled patch features without the contextualization transformer.

			Cosine $\downarrow$		Relative $\ell_{2}$ $\downarrow$
Task	Shift	$N$	DP–DINOv2	RoboMP-DINOv2 / -MCR	DP–DINOv2	RoboMP-DINOv2 / -MCR
Out-of-bin	Clutter	1269	$31.770\!\pm\!29.030$	$\mathbf{2.880\!\pm\!2.990}$	$22.938\!\pm\!10.619$	$\mathbf{6.707\!\pm\!3.555}$
	Appearance	1269	$21.752\!\pm\!30.249$	$\mathbf{.0028\!\pm\!.0064}$	$16.784\!\pm\!12.520$	$\mathbf{.181\!\pm\!.150}$
Pull tool	Clutter	2409	$.002\!\pm\!.005$	$\mathbf{.001\!\pm\!.002}$	$.153\!\pm\!.109$	$\mathbf{.126\!\pm\!.077}$
	Appearance	2409	$.240\!\pm\!.485$	$\mathbf{.007\!\pm\!.037}$	$1.856\!\pm\!1.509$	$\mathbf{.230\!\pm\!.281}$
Peg insertion	Clutter	1514	$.625\!\pm\!1.347$	$\mathbf{.249\!\pm\!.546}$	$2.642\!\pm\!2.373$	$\mathbf{1.636\!\pm\!1.516}$

Table 2: Feature distances between perturbed observations and their paired clean counterparts (mean

\pm

standard deviation over

N

observation pairs). Cosine distance is reported in units of

10^{-3}

and relative

\ell_{2}

distance in units of

10^{-2}

. DP–DINOv2 uses a 768-dimensional feature, whereas RoboMP-DINOv2 uses a 256-dimensional feature. Under Clutter, RoboMP-DINOv2 denotes the encoder trained without color randomization. Under Appearance, DP–DINOv2 is trained without color randomization, whereas RoboMP-DINOv2-MCR denotes the complete encoder-and-augmentation pipeline. Lower values indicate smaller representation changes under the evaluated perturbation; the lower mean is shown in bold.

Method	Place cube	Put can (clear)	Put can (obstacle)	Pull tool	Peg insertion	Long horizon	Two- arm	Avg.
2D object-centric masking (masked-area rand.)	.790	.040	.010	.000	.000	.003	.000	.120
3D object-centric masking (masked-area rand.)	.940	.030	.000	.000	.010	.015	.000	.142
DP–DINOv2 (full-image rand.)	.330	.050	.100	.100	.250	.498	.010	.191
RoboGround–DINOv2 (masked-area rand.)	.990	.650	.660	.000	.000	.160	.000	.351
RoboMP-DINOv2-MCR	.990	.710	.670	.820	.700	.906	.280	.725

Table 3: Success rates under Appearance OOD. All methods are trained with color randomization. DP–DINOv2 uses full-image randomization, whereas the mask-based methods apply randomization only within selected mask regions. “Avg.” is the macro-average over the seven settings. The highest observed performance in each column is shown in bold.

Method	Two arm	Place cube	Pull tool
DP + full-image rand.	0.00	0.02	0.00
DP + masked rand.	0.14	0.08	0.34
Only mask + mask rand.	0.00	0.04	0.00
RoboMP-DINOv2-MCR	0.26	0.24	0.38

Table 4: Appearance-randomization comparison under combined appearance-and-clutter shifts. Values report mean success rates across the three representative tasks.

Figure 3: Preserving behaviorally relevant color cues outside the randomized regions. The stove-light color determines the correct action. RoboMP-DINOv2-MCR retains access to this contextual signal and substantially outperforms DP–DINOv2 trained with full-image color randomization under both light conditions.

Panel of Figure 4 — Figure 4: Qualitative examples of imperfect evaluation-time mask predictions from Grounding DINO+SAM. Top row: RGB observations. Bottom row: predicted masks used by all mask-based methods. From left to right, the columns show Put Can (obstacle), Long Horizon, and Two-Arm stacking.

Explore similar work

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning
Jul 10, 2026Yi Li, Alexandre Chapin, Liming Chen +2Object-Centric Representations Robotic Manipulation Policies
TU Darmstadt · 1TU Darmstadt · LIRIS +3
Foundation and Small Models Coordination for Visuomotor Policy Learning
Mar 9, 2026Haoran Ding, Liang Ma, Yaxun Yang +7Visuomotor Policy Robotic Perception
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
Action-Effect Memory Pretraining for Robot Manipulation
Jun 10, 2026Yijing Zhou, Qiwei Liang, Sitong Zhuang +5Robotic Manipulation Dual-Branch Temporal Modeling
Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University

Jul 10, 2026, cs.ROJ/K move · Enter open · S save

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning

Yi Li, Alexandre Chapin, Liming Chen, Jan Peters+1

TU Darmstadt · 1TU Darmstadt · LIRIS+3

Object-Centric Representations Robotic Manipulation Policies Motion Imitation Imitation Learning Dense Baseline

Open paper AskPDF arXiv

Robotic manipulation policies rely on pre-trained vision models that give either a global scene embedding or a dense patch grid. Both mix task-relevant and task-irrelevant features. Object-centric slot representations are a structured alternative: they group features into a few per-object slots. We test what this structure buys on ManiSkill3 PickCube-v1, with a frozen encoder and a held-out-seed evaluation. Holding the policy, goal token, rendering, and calibration fixed and changing only the encoder, a frozen object-centric SPOT representation (DINO ViT-B/16 + Slot Attention) reaches 55.0

\pm

2.9% success, 22.4% above a dense DINO global-feature baseline (32.6

\pm

1.5%), with the same trainable policy and no encoder fine-tuning. More tokens alone do not help: a dense patch grid with 16x the tokens performs no better than the global feature. Adding an explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7

\pm

4.2%, just below a privileged 3D-oracle upper bound (71.7

\pm

4.1%). An automated kinematic failure taxonomy then separates spatial-precision (Near-Miss) failures from object-tracking (No-Grasp) failures: spatial grounding reduces Near-Miss while leaving No- Grasp unchanged. The same taxonomy transfers to the harder StackCube-v1 and points to occlusion as the main bottleneck.

Abstract

Explore similar work

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning

Foundation and Small Models Coordination for Visuomotor Policy Learning

Action-Effect Memory Pretraining for Robot Manipulation