Visual reinforcement learning (RL) policies that land a drone on a moving platform can fail when the landing pad looks different from the one used in training. Using binary target masks as the visual input removes most pixel-level appearance information while preserving the spatial representation of the target, and lets a text prompt specify the target. In simulation, we test whether a frozen (fixed-weight) mask-based DrQ-v2 policy can handle a new pad when only the prompt of its fine-tuned SAM 3 detector changes. A banana-themed pad lowered landing success from 75.6% to 12.0%, and SAM 3 rarely detected it under the original prompt. A revised prompt, chosen after testing several candidates, restored success to 68.4% with no weight updates. In a separate experiment with a policy trained on ground-truth masks, filling in masks that were missing while the pad was visible helped landing far more than the tested mask-centroid and area corrections. Tracking the pad over time supplied masks more often and also kept producing them after the pad was confirmed absent from the rendered image, but it did not establish a reliable improvement in landing success. Whether a usable mask reaches the policy at all thus appears to be an important limiting factor for this approach.
Figures & tables
Parameter
Value
Physics / PID rate
240Hz / 24Hz
Actor rate
12Hz
Perception update rate
6Hz
Camera model
pinhole, no distortion
Focal length fx=fy
63.455 px
Principal point cx=cy
41.5 px
TABLE I: Landing-task parameters.
Fig. 1: Overview of the shared perception, policy, and control pipeline. Direct SAM 3 masks take precedence and temporal predictions provide fallback observations. The evaluated frontends differ in their temporal anchoring (Section VI ). CLIP similarity is part of the perception metadata and does not reject masks.
Speed
A
B
C
D
( ms−1 )
0.00
35/6/9 (70%)
4/20/26 (8%)
34/8/8 (68%)
31/6/13 (62%)
0.25
38/2/10 (76%)
6/13/31 (12%)
34/8/8 (68%)
42/3/5 (84%)
0.50
42/3/5 (84%)
8/6/36 (16%)
40/7/3 (80%)
44/2/4 (88%)
0.75
42/1/7 (84%)
6/4/40 (12%)
37/6/7 (74%)
41/3/6 (82%)
1.00
32/1/17 (64%)
6/2/42 (12%)
26/4/20 (52%)
36/0/14 (72%)
TABLE II: Appearance and prompt evaluation with the frozen actor and the Temporal Recovery frontend, 50 paired episodes per speed and condition. Entries are success/crash/timeout (success rate). A: original pad, original prompt; B: banana-themed pad, original prompt; C: banana-themed pad, re-specified prompt; D: sampled within-family greyscale pad texture, original prompt.
Observation condition
Screen
Confirmation
(50 ep.)
(200 ep.)
Ground-truth mask
41/50
165 (82.5%)
Raw SAM-derived mask
0/50
3 (1.5%)
Centroid, area, or joint correction
0/50
–
Visible-dropout oracle repair
42/50
174 (87.0%)
TABLE III: Frozen-policy substitution diagnostic with a separately trained ground-truth-mask actor, the same fine-tuned SAM 3 checkpoint as the main study, and the earlier protocol of Appendix A . Each repair modifies one class of SAM error; the actor is unchanged. These results are not comparable with those of the main actor.
Fig. 2: Example of temporal target recovery. A direct SAM 3 detection provides the earlier target observation (left). On a later frame without a direct detection (centre), the Temporal Recovery frontend propagates a target mask from the earlier visual evidence (right). The example is illustrative; closed-loop effects are reported in Section VI-B .
Metric
Base
Temporal Recovery
Terminal usable-target coverage
83.2%
94.5%
Post-absence output
17.9%
30.1%
TABLE IV: Target continuity and post-absence output of the two frontends, pooled over the 300 episodes per frontend of the first evaluation (Table V ). Each frontend generates its own trajectories, so the values are descriptive. Terminal-coverage counts: 5,567/6,694 and 6,597/6,978 renderer-visible fresh frames; post-absence output counts: 430/2401 and 586/1945 fresh frames.
Condition
Base
Temporal Recovery
Δ SR (pp)
p
P
44/1/5 (88%)
48/1/1 (96%)
+8
0.289
S0
30/9/11 (60%)
40/6/4 (80%)
+20
0.076
S1
37/3/10 (74%)
36/5/9 (72%)
−2
1.000
S2
42/4/4 (84%)
45/1/4 (90%)
+6
0.508
S3
47/1/2 (94%)
44/2/4 (88%)
−6
0.453
S4
37/3/10 (74%)
40/2/8 (80%)
+6
0.549
TABLE V: Landing outcomes of the frozen actor with the two frontends in the first evaluation, 50 matched episodes per condition. P and S1–S4 use separate seed blocks. Entries are success/crash/timeout (success rate). p : exact two-sided McNemar test, without multiplicity correction.
Cond.
Base
Temporal Rec.
Δ SR (pp)
95% CI (pp)
p
P
83/5/12
88/2/10
+5.0
[−4.73,+14.74]
0.405
S0
63/9/28
67/10/23
+4.0
[−9.02,+16.83]
0.652
TABLE VI: Independent evaluation on 100 new matched episodes per condition. Entries are success/crash/timeout. The difference, its matched-pair 95% confidence interval (Newcombe square-and-add method with Wilson limits, pointwise and approximate), and the exact two-sided McNemar p come from a post hoc analysis that is conditional on the six unresolved pairs of Appendix B ; the original protocol withheld paired inference.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Diagnostic
Main
Horizon / actor rate
24s / 6Hz
30s / 12Hz
Initial altitude h
0.8– 6m
10m
Horizontal offset
radial, ≤0.1h
U[−3,3] m/axis
Platform heading
±180∘
±15∘
Platform speed ( ms−1 )
0 (30%), else [0,0.5)
U[0,1]
Pad size
0.675m
1m
Appendix
TABLE VII: Protocol of the substitution diagnostic, compared with the main protocol of Section III .
Condition / seed
First step
First contact
Max. pre-contact
>1\SIUnitSymbolMicro
step
difference ( \SIUnitSymbolMicro )
P / 22300015
5000
5015
3.80
P / 22300036
5037
5039
1.49
P / 22300047
4915
4917
1.55
P / 22300048
4981
7196
1.43
P / 22300098
4968
4970
1.31
Appendix
TABLE VIII: Independent-evaluation pairs with unresolved pre-contact divergence. One physics step is 1/240 s.
Analysis
Cond.
Pairs
Disc.
Δ SR (pp)
95% CI (pp)
p
All pairs
P
100
9/14
+5.00
[−4.73,+14.74]
0.405
S0
100
20/24
+4.00
[−9.02,+16.83]
0.652
Six excluded
P
95
9/12
+3.16
[−6.70,+13.03]
0.664
S0
99
19/24
+5.05
[−7.97,+17.83]
0.542
Appendix
TABLE IX: Post hoc conditional reanalysis of the independent evaluation: Temporal Recovery minus base frontend. Disc.: discordant pairs, successes with the base frontend only / with Temporal Recovery only.
Condition
Memory-restricted
P
41/2/7 (82%)
S0
44/1/5 (88%)
S1
43/2/5 (86%)
S2
38/5/7 (76%)
S3
44/1/5 (88%)
S4
41/2/7 (82%)
Appendix
TABLE X: Landing outcomes of the memory-restricted variant in the first evaluation, 50 matched episodes per condition, matched with Table V . Entries are success/crash/timeout (success rate).
Setting
Value
Training (DrQ-v2)
Total actor decisions
1.25×106
Mask stack
3×84×84
Low-dimensional history
3×15
Optimiser / learning rate
Adam / 1×10−4
Batch size / replay capacity
256 / 160,000
Appendix
TABLE XI: Training and reward settings of the main actor.
Component
Configuration
SHA-256
Actor
1.25 M decisions, seed 1
926c61819980
SAM 3
fine-tuned, 1008 px
120dc5228a36
GT-mask actor
diagnostic only
43bf505f11a5
DAM4SAM
sam21pp_hiera_t.yaml
7402e0d864fa
CLIP
ViT-B/32 ( timm , OpenAI)
bd41409c7f2b
Temporal Recovery
runtime wrapper
ccea0ddabc1c
Appendix
TABLE XII: Frozen components (first 12 hexadecimal characters of each SHA-256 digest).
Visual reinforcement learning trains agents to perceive images and learn actions through interaction, enabling control when states are not directly observed.
Query-based Vision-Language-Action (VLA) models offer low-latency inference that is attractive for bimanual robotic manipulation, but we observe that they can still exhibit discontinuous actions and execution failures in complex dual-arm tasks. We hypothesize that unstable multi-view and language fusion is one contributing factor in these failures, often coinciding with attention spreading to distracting regions. To improve robustness, we introduce the Modality Masking Mechanism (M3), an embarrassingly simple, training-only strategy that requires no architectural changes or large-scale robot pretraining. M3 stochastically masks subsets of modality channels during training, exposing the policy to controlled partial observations and encouraging it to rely less on distracting cues and more on evidence that remains reliable. We evaluate M3 on ten bimanual tasks from RoboTwin 2.0 and on three long-horizon real-world tasks. Compared with the Adapter baseline, M3 improves average success by 21.7% in the Clean setting and 11.4% in Clean2Rand, where policies are trained on clean demonstrations and evaluated on randomized scenes, while also improving averaged real-world full-task success by over 30%. These results suggest that structured training-time masking is a practical way to improve the robustness of query-based VLA policies for bimanual manipulation.
Dongzhou Cheng, Ziang Li, Yixiao Zhou +6
Southeast University · Shanghai Innovation Institute · Wuhan University +4
Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action. General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context. We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters. It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction. We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR. Across seven simulated manipulation settings, RoboMP-DINOv2 achieves 60.7% success under spatial shifts and 59.7% under scene clutter, compared with 50.7% and 41.0% for a DINOv2-based Diffusion Policy. Under unseen object colors, RoboMP-DINOv2-MCR achieves 72.5% success versus 35.1% for the strongest color-randomized baseline. Additional experiments and representation analyses show improved robustness while preserving behaviorally relevant scene information. Code is available at https://github.com/han20192019/RoboMP_DINOv2.
Han Qi, Heng Yang
Harvard School of Engineering and Applied Sciences Harvard University
World Action Models (WAMs) present a promising paradigm for robotic control via video prediction. However, current WAMs suffer from fundamental spatial bottlenecks: standard text inputs introduce referential ambiguity in cluttered scenes, while unstructured RGB predictions lack semantic grounding and remain biased by task-irrelevant backgrounds. To overcome these limitations, we introduce MaskWAM, an object-centric world-action model. By jointly integrating masks as both explicit inputs and predictions via a unified Mixture of Transformers (MoT), MaskWAM unlocks robust policy generalization. This design provides two key benefits: (1) predicting future masks yields object-centric semantic supervision that suppresses visual noise, significantly enhancing even standard text-conditioned WAMs; and (2) coupling this predictive supervision with first-frame visual prompts, such as target object masks, establishes a precise spatial anchor that substantially reduces language ambiguity. Crucially, as WAMs are inherently vision-driven architectures, direct mask conditioning yields substantially stronger guidance than text alone, establishing a precise and robust paradigm for manipulating unseen objects. Evaluations on LIBERO, RoboTwin, and real-world tasks demonstrate that MaskWAM significantly outperforms baselines in both language-clear and language-ambiguous tasks.
Hanyang Yu, Haitao Lin, Jingbo Zhang +4
1The Hong Kong University of Science and Technology · ‡Work done during an internship at Tencent Robotics X · 2Tencent Robotics X +1