Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption

Zhiyuan Yang; Jiaju He; Ruiteng Zhao; Anqi Cheng; Haiyue Zhu; Kezhi Mao

cs.ROOct 17, 2026

Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption

Authors: Zhiyuan Yang, Jiaju He, Ruiteng Zhao, Anqi Cheng, Haiyue Zhu, Kezhi Mao

Organizations: Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore · National University of Singapore (NUS), Singapore

Abstract

3D manipulation policies derive actions from explicit scene geometry rather than color images alone, enabling them to reason over metric distances, object relations, and robot-scene reachability. This geometric interface is also a vulnerability: commodity RGB-D sensors return missing measurements, boundary artifacts, and plausible but wrong surfaces around transparent, reflective, thin, or textureless objects. We study this failure under realistic depth corruption and observe a sharp depth-noise cliff across modern 3D policies, even when they are trained with matched noisy depth. This residual gap motivates a spatial-encoding approach: corrupted depth becomes entangled with the token coordinates and geometry features consumed by the action expert. We propose Noise-Invariant Spatial Encoding (NISE), a 3D policy architecture that learns trustworthy geometry features even facing imperfect depth. NISE uses Geometric Cross-View Self-Supervision to shape shared policy features with calibrated multi-view depth, and Gaussian Marginalized 3D Attention to carry position uncertainty into deployed attention rather than treating every lifted token as exact. Across simulated benchmarks with sensor-realistic corruption and real-robot tasks involving transparent and reflective materials, NISE improves depth-noise robustness while preserving clean-depth performance, supporting 3D policies designed for commodity RGB-D deployment rather than clean-depth benchmarks alone.

Figures & tables

Figure 1: Realistic depth corruption breaks what 3D policies can trust. (a) In a transparent test-tube manipulation scene, raw depth contains missing and geometrically wrong measurements; a standard 3D policy consumes this corrupted geometry as exact and fails, while NISE succeeds. (b) Depth-noise robustness on RLBench PerAct-18 [ 1 , 2 ] and MetaWorld [ 3 ] . NISE maintains higher success rates as depth corruption increases, with a smaller clean-to-saturated decline than the compared baselines.

Figure 2: NISE framework. Per-camera RGB-D is encoded by a frozen visual backbone ViT and a trainable DPT-style head [ 21 ] into a shared spatial feature map. NISE acts on this shared map: during training, Geometric Cross-View Self-Supervision attaches a cross-view geometry head to encourage view-consistent geometry in the same features consumed by the policy; at deployment, this head is removed. Gaussian Marginalized 3D Attention remains in the deployed policy and carries per-token coordinate uncertainty into 3D attention, so lifted tokens influence action features according to their spatial reliability.

Method	Close Jar	Drag Stick	Insert Peg	Meat off Grill	Open Drawer	Place Cups	Place Wine	Push Buttons	Put Cupboard	Avg.
PerAct [ 2 ]	$55.2{\pm}4.7$	$89.6{\pm}4.1$	$5.6{\pm}4.1$	$70.4{\pm}2.0$	$88.0{\pm}5.7$	$2.4{\pm}3.2$	$44.8{\pm}7.8$	$92.8{\pm}3.0$	$28.0{\pm}4.4$	$49.4$
RVT [ 36 ]	$52.0{\pm}2.5$	$\underline{99.2{\pm}1.6}$	$11.2{\pm}3.0$	$88.0{\pm}2.5$	$71.2{\pm}6.9$	$4.0{\pm}2.5$	$91.0{\pm}5.2$	$\mathbf{100.0{\pm}0.0}$	$49.6{\pm}3.2$	$62.9$
3D Diffuser Actor [ 8 ]	$96.0{\pm}2.5$	$\mathbf{100.0{\pm}0.0}$	$65.6{\pm}4.1$	$\underline{96.8{\pm}1.6}$	$89.6{\pm}4.1$	$24.0{\pm}7.6$	$93.6{\pm}4.8$	$\underline{98.4{\pm}2.0}$	$\mathbf{85.6{\pm}4.1}$	$81.3$
RVT-2 [ 6 ]	$\mathbf{100.0{\pm}0.0}$	$99.0{\pm}1.7$	$40.0{\pm}0.0$	$\mathbf{99.0{\pm}1.7}$	$74.0{\pm}11.8$	$38.0{\pm}4.5$	$\underline{95.0{\pm}3.3}$	$\mathbf{100.0{\pm}0.0}$	$66.0{\pm}4.5$	$81.4$
ARP [ 37 ]	$\underline{97.6}$	$88.0$	$53.2$	$96.0$	$\underline{90.4}$	$48.0$	$92.0$	$\mathbf{100.0}$	$68.0$	$81.6$
TVVE [ 38 ]	$\mathbf{100.0{\pm}0.0}$	$\mathbf{100.0{\pm}0.0}$	$\mathbf{98.0{\pm}2.8}$	$94.0{\pm}2.8$	$90.0{\pm}2.8$	$\underline{54.0{\pm}2.8}$	$92.0{\pm}5.7$	$\mathbf{100.0{\pm}0.0}$	$74.0{\pm}8.5$	$\underline{86.6}$

Table 1: Quantitative results on RLBench PerAct-18. Per-task success rates (%, mean

\pm

standard deviation where available) on the standard 18-task PerAct suite under clean-depth evaluation. NISE reports mean

\pm

standard deviation over repeated evaluations of a fixed checkpoint. Published baselines use clean-depth training, while NISE uses noisy-depth training. The best and second-best distinct means per column are bold and underlined, respectively, including ties.

Figure 3: Real-robot setup: single-arm UR5e (left) and bimanual UR3e (right). Camera labels identify Azure Kinect and RealSense D455/D405.

RLBench PerAct-18
Method	Clean	Mild	Saturated
3DDA [ 8 ]	$64.8{\pm}9.1$	$60.2{\pm}9.7$	$47.8{\pm}12.0$
RVT-2 [ 6 ]	$65.2{\pm}9.5$	$62.4{\pm}9.4$	$48.5{\pm}13.8$
SAM2Act [ 7 ]	$68.1{\pm}9.8$	$63.7{\pm}8.9$	$50.9{\pm}11.9$
NISE	$\mathbf{89.0{\pm}4.3}$	$\mathbf{85.7{\pm}4.5}$	$\mathbf{84.1{\pm}4.6}$

Table 2: Robustness to depth corruption after noise-aware training. Task-averaged success rates (%) on RLBench PerAct-18 and MetaWorld under clean, mild-noise, and saturated-noise evaluation. Values are means

\pm

standard deviations across three independent training runs.

Trained on Clean Depth Setting GCVSS GM3A SR (%)

\Delta

FLOPs Baseline

\times

\times

52.1{\pm}6.1

+0.0

+ GM3A

\times

\checkmark

53.3{\pm}5.8

+0.8

+ GCVSS

\checkmark

\times

62.0{\pm}5.2

+0.0

+ Both

\checkmark

\checkmark

68.6{\pm}4.8

+0.8

Trained on Noisy Depth Setting GCVSS GM3A SR (%)

\Delta

FLOPs Baseline

\times

\times

75.5{\pm}4.5

+0.0

+ GM3A

\times

\checkmark

78.2{\pm}4.0

+0.8

+ GCVSS

\checkmark

\times

80.5{\pm}3.5

+0.0

NISE

\checkmark

\checkmark

\mathbf{85.7{\pm}4.5}

+0.8

Table 3: Baseline control and component ablation. Mean success rate on RLBench PerAct-18 under mild-noise depth evaluation . Both clean-depth and noisy-depth blocks use the same frozen-ViT+DPT encoder and action expert, isolating the effects of GCVSS and GM3A.

\Delta

FLOPs reports the additional deployed inference cost in GFLOPs.

Figure 4: Real-robot Success Rate. Each task group compares DP3, 3DDA, RVT-2, and NISE , with

N=30

trials per task and method.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Depth-corruption families used as a controlled diagnostic. The panels show the simulator depth wrapper applied to a clean rendered frame, isolating random holes, boundary-localized artifacts, and displaced lookup near discontinuities. The figure is a compact visual explanation of the procedural knobs in Equations 4 , 6 and 7 ; sensor-specific calibration is captured by the preset scales rather than by exact rendering of a single camera model.

Preset	$\sigma_{\ell}$ (px)	$\sigma_{s}$ (px)	$p_{\mathrm{edge}}$	$p_{\mathrm{hole}}$
Clean	0.00	0.000	0.000	0.0000
Mild	0.30	0.150	0.300	0.0020
Saturated	0.50	0.250	0.500	0.0050

Appendix

Table 4: Depth-noise presets. The main paper uses clean, mild, and saturated; saturated corresponds to the high-stress endpoint used in the teaser and main summary table. Saturated is named moderate in the implementation, with the same parameter values.

Policy stack	Ablation row	GCVSS	GM3A	Clean	Mild	Saturated
Native 3DDA insertion
3DDA	Plain noise-aware	$\times$	$\times$	41.0	54.0	56.7
3DDA	+ GCVSS	$\checkmark$	$\times$	45.0	62.0	63.5
3DDA	+ GM3A	$\times$	$\checkmark$	43.0	59.0	61.0
3DDA	+ GCVSS + GM3A	$\checkmark$	$\checkmark$	47.0	68.0	70.3
Native RVT-2 insertion

Appendix

Table 5: Native-stack ablations on RLBench. Success rates (%) are means over three RLBench tasks: push_buttons, insert_peg, and stack_cups. The 3DDA block keeps the native 3DDA feature pyramid and diffusion action expert; the RVT-2 block keeps the virtual-view/keypose policy family. Within each policy block, all four variants are fine-tuned on saturated-noise demonstrations using the same protocol, varying only the inclusion of GCVSS and GM3A.

Input / policy	Clean	Mild	Saturated
NISE + raw depth	$89.0{\pm}4.3$	$85.7{\pm}4.5$	$84.1{\pm}4.6$
NISE + D3RoMa repair	$89.0{\pm}4.2$	$86.1{\pm}4.3$	$84.8{\pm}4.5$
NISE + Marigold-DC repair	$\mathbf{89.1{\pm}4.1}$	$\mathbf{86.4{\pm}4.2}$	$\mathbf{85.2{\pm}4.4}$

Appendix

Table 6: Effect of depth repair on NISE . Mean success rate

\pm

standard deviation (%) on RLBench PerAct-18 under clean, mild-noise, and saturated-noise evaluation. We compare NISE with raw depth and with depth preprocessed by D3RoMa or Marigold-DC before the policy consumes RGB-D. Best means per column are bold.

Figure 6: CDM

\times

policy control on real-robot lid removal. Evaluation uses Remove the Petri Lid under real sensor noise, with

30

trials per configuration. Each cell reports the success rate and successful trials out of

30

. The base model uses the same policy stack as NISE with GCVSS and GM3A disabled.

Panel of Figure 7 — Figure 7: Cross-view depth becomes a geometry target only after calibrated warping. The source camera is forward-warped into the target camera frame and filtered by a visibility/validity mask before it supervises the shared feature map. This is the practical difference between a raw cross-camera noisy target and GCVSS: the auxiliary signal is still noisy, but it is expressed in the same camera frame as the target feature map.

Target construction	Success (%)
Clean-depth references
Per-camera clean target	64.0
Masked clean consensus	62.4
Noisy-depth targets
Raw cross-camera target	38.2
Warped target without z-buffer visibility	55.2

Appendix

Table 7: GCVSS target-construction and component ablation. Success rates (%) on RLBench Stack Cups under saturated depth corruption. All variants use the same policy stack and training budget. The component controls remove z-buffer visibility filtering or validity masking from the full calibrated cross-view target construction. Clean-depth targets are simulation references; the best noisy-depth variant is bold.

GM3A variant	Clean	Mild	Saturated
Frequency-only	79.3	70.7	66.7
Adaptive residual	85.3	77.3	73.3

Appendix

Table 8: Frequency-only versus adaptive GM3A. Success rates (%) across three RLBench tasks (Push Buttons, Insert Peg, and Stack Cups) under clean, mild-noise, and saturated-noise evaluation. Both variants use the same training data and seeds.

Task type	Task	GCVSS only	GCVSS + GM3A	$\Delta$
Coarse actuation	push_buttons	$100.0{\pm}0.0$	$100.0{\pm}0.0$	$+0.0$
Coarse actuation	open_drawer	$96.0{\pm}2.8$	$96.0{\pm}2.8$	$+0.0$
Coarse actuation	turn_tap	$98.0{\pm}2.8$	$98.0{\pm}2.8$	$+0.0$
Precision geometry	insert_peg	$58.0{\pm}5.7$	$68.0{\pm}5.0$	$+10.0$
Precision geometry	stack_cups	$62.0{\pm}5.0$	$72.0{\pm}4.6$	$+10.0$

Appendix

Table 9: Precision-task GM3A analysis. Success rates (%) are measured on five RLBench tasks under mild-noise evaluation with the same-stack noisy-depth policy. The comparison fixes GCVSS and toggles GM3A, isolating the deployed attention change. GM3A yields larger gains on the tested precision-sensitive tasks, while coarse-task means are unchanged.

Setting	Mild drift $\downarrow$	Mild CKA $\uparrow$	Saturated drift $\downarrow$	Saturated CKA $\uparrow$	Mild SR (%)
Clean-trained same-stack baseline	0.44	0.55	0.63	0.39	52.1
Noisy-trained same-stack baseline	0.31	0.68	0.48	0.54	75.5
+ GM3A	0.28	0.71	0.42	0.59	78.2
+ GCVSS	0.18	0.82	0.26	0.74	80.5
NISE	0.12	0.89	0.17	0.84	85.7

Appendix

Table 10: Direct feature-stability diagnostic on RLBench. Same-scene paired evaluation compares the shared feature map under clean depth against the same RGB/proprioception with mild or saturated corrupted depth. Cosine drift is lower-is-better; CKA is higher-is-better. The mild-noise SR column reports the corresponding same-stack success rate when available.

Task type	Task	Reliable surface	Depth boundary	Transparent/specular proxy	Contact region
Coarse	push_buttons	0.06	0.15	0.18	0.10
Coarse	open_drawer	0.07	0.16	0.19	0.12
Precision	insert_peg	0.08	0.27	0.31	0.29
Precision	stack_cups	0.09	0.25	0.34	0.28
Real transparent	remove_petri_lid	0.10	0.31	0.39	0.32
Real transparent	pour_test_tube	0.11	0.33	0.42	0.35

Appendix

Table 11: GM3A dampening by task and region. Values report mean high-frequency dampening

1-\alpha

on saturated-depth RLBench and real-robot diagnostic frames; larger values indicate stronger suppression of unreliable 3D positional phase. GM3A keeps reliable regions close to ordinary 3D-RoPE while increasing dampening around depth boundaries, transparent/specular proxies, and precision-task contact regions.

Method	Training depth	Clean	Mild	Saturated
3DDA	Clean only	$81.3{\pm}5.0$	$55.0{\pm}11.7$	$15.0{\pm}13.4$
NISE	Clean only	$89.7{\pm}4.1$	$68.6{\pm}4.8$	$63.0{\pm}8.5$
NISE	Mixed	$89.0{\pm}4.3$	$85.7{\pm}4.5$	$84.1{\pm}4.6$

Appendix

Table 12: Clean-only and mixed-noise training on RLBench PerAct-18. Task-averaged success rates (%, mean

\pm

standard deviation) on RLBench PerAct-18 under clean, mild, and saturated depth evaluation. Clean-only training uses clean simulator depth; mixed training samples clean/mild/saturated inputs with a

1{:}1{:}1

ratio. The mixed-training NISE row reproduces Table 2 , with

\pm

denoting the standard deviation across three independent training runs.

Setting	GCVSS	GM3A	$\Delta$ Params (K)	$\Delta$ FLOPs	Latency (ms)	Throughput (Hz)
Same-stack baseline	$\times$	$\times$	$0$	$+0.0$	$36.1{\pm}0.4$	$27.7$
+ GCVSS	$\checkmark$	$\times$	$0$	$+0.0$	$36.2{\pm}0.5$	$27.6$
+ GM3A	$\times$	$\checkmark$	$8.44$	$+0.8$	$37.4{\pm}0.5$	$26.7$
NISE	$\checkmark$	$\checkmark$	$8.44$	$+0.8$	$37.5{\pm}0.6$	$26.6$

Appendix

Table 13: Runtime and deployed parameter and FLOP overhead. Measurements use the RLBench PerAct-18 same-stack policy with four RGB-D views, batch size one, and policy-forward timing on the RTX 5080 after warmup. Timing excludes simulator rendering and robot I/O.

\Delta

Params (thousands) and

\Delta

FLOPs (GFLOPs) report deployed overhead relative to the same-stack baseline without GCVSS or GM3A; the training-only GCVSS head is excluded.

MetaWorld easy tasks
Method	Btn Press	Btn Topdown	Btn TD Wall	Btn Wall	Coffee Btn	Dial Turn	Door Close
3D Diffusion Policy	$62{\pm}15$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$72{\pm}25$	$73{\pm}34$	$53{\pm}15$	$\mathbf{100{\pm}0}$
3D Flow Matching	$0{\pm}0$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$67{\pm}31$	$\underline{97{\pm}5}$	$\mathbf{70{\pm}7}$	$\mathbf{100{\pm}0}$
3D ManiFlow	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$\underline{67{\pm}13}$	$\mathbf{100{\pm}0}$
NISE	$\underline{88{\pm}5}$	$\mathbf{100{\pm}0}$	$\mathbf{100{\pm}0}$	$\underline{90{\pm}6}$	$94{\pm}4$	$\mathbf{70{\pm}8}$	$\underline{88{\pm}5}$
MetaWorld easy tasks

Appendix

Table 14: MetaWorld per-task results, part I. The NISE row reports clean-depth success rates from our MetaWorld protocol. Values are success rates (%). For NISE ,

\pm

denotes the standard deviation across repeated evaluations. The best and second-best distinct means per column are bold and underlined, respectively, including ties.

MetaWorld medium tasks
Method	Box Close	Coffee Pull	Coffee Push	Hammer	Peg Insert Side	Push Wall	Soccer
3D Diffusion Policy	$18{\pm}8$	$52{\pm}23$	$55{\pm}0$	$\mathbf{77{\pm}6}$	$58{\pm}6$	$60{\pm}8$	$8{\pm}5$
3D Flow Matching	$18{\pm}8$	$67{\pm}2$	$\underline{58{\pm}15}$	$\underline{75{\pm}22}$	$\underline{68{\pm}5}$	$15{\pm}15$	$\underline{10{\pm}4}$
3D ManiFlow	$\underline{45{\pm}12}$	$\mathbf{97{\pm}2}$	$\mathbf{82{\pm}6}$	$42{\pm}24$	$\mathbf{88{\pm}8}$	$\mathbf{93{\pm}2}$	$7{\pm}6$
NISE	$\mathbf{60{\pm}7}$	$\underline{94{\pm}4}$	$\mathbf{82{\pm}5}$	$66{\pm}8$	$\mathbf{88{\pm}5}$	$\underline{90{\pm}5}$	$\mathbf{34{\pm}8}$
MetaWorld medium / hard tasks

Appendix

Table 15: MetaWorld per-task results, part II. This table continues the clean-depth per-task comparison in Table 14 . The best and second-best distinct means per column are bold and underlined, respectively, including ties.

Task	DP3	3DDA	RVT-2	NISE
Assemble the Gear	4/30, 13.3 [5.3, 29.7]	7/30, 23.3 [11.8, 40.9]	11/30, 36.7 [21.9, 54.5]	25/30, 83.3 [66.4, 92.7]
Remove the Petri Lid	6/30, 20.0 [9.5, 37.3]	10/30, 33.3 [19.2, 51.2]	5/30, 16.7 [7.3, 33.6]	20/30, 66.7 [48.8, 80.8]
Insert the Metal Pipe	3/30, 10.0 [3.5, 25.6]	8/30, 26.7 [14.2, 44.4]	12/30, 40.0 [24.6, 57.7]	22/30, 73.3 [55.6, 85.8]
Pour Test Tube into Mortar	2/30, 6.7 [1.8, 21.3]	6/30, 20.0 [9.5, 37.3]	4/30, 13.3 [5.3, 29.7]	13/30, 43.3 [27.4, 60.8]
Vision-Sorted Gear Collection	4/30, 13.3 [5.3, 29.7]	3/30, 10.0 [3.5, 25.6]	7/30, 23.3 [11.8, 40.9]	16/30, 53.3 [36.1, 69.8]

Appendix

Table 16: Real-robot per-task trial counts with Wilson 95% confidence intervals. Each task entry is

k/30

successes, success rate (%), and Wilson interval.

Sweep	Value	Val SR (%)	Selected	Readout
GCVSS weight $\lambda_{\mathrm{geo}}$
GCVSS	0.00	78.2	–	GM3A-only control; no cross-view geometry shaping
GCVSS	0.03	82.4	–	useful signal, but still underweighted
GCVSS	0.10	85.7	$\checkmark$	best balance between geometry shaping and policy supervision
GCVSS	0.30	84.8	–	similar but begins to overemphasize the auxiliary target
GCVSS	1.00	81.6	–	geometry loss competes too strongly with action learning

Appendix

Table 17: Hyperparameter sweep for the reported loss weights. Validation success rates (%) on RLBench PerAct-18 under mild-noise evaluation with the fixed NISE stack.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Sep 21, 2026

REDACT: Robust Perceptive Locomotion under Unseen Visual Corruption

Depth-conditioned locomotion policies have demonstrated impressive agile maneuvers, but can be steered to unpredictable actions when observations are outside their training distribution. Occlusion, invalid returns, sensor noise, and visual distractors can shift deployment observations away from nominal simulated depth. While synthetic sensor augmentation targets specified degradations, it does not by itself define behavior under corruption families omitted from training. To address gaps in training-time coverage, we present REDACT (Retaining Evidence Despite Artifacts for Continued Traversal), a teacher-student framework combining an improved visual encoder architecture, persistent feature masking, and a novel consensus-gating algorithm to retain useful depth information under unmodeled corruption. The gate uses approximate conformal calibration on clean observations alone, requiring no prior knowledge of the corruption type. Trained on clean simulated depth, REDACT retains useful visual information under unseen corruption, supporting higher traversal success than existing parkour baselines. Evaluation of depth augmentation across corruption families further shows that REDACT improves robustness where augmentation coverage is missing. Real-world trials demonstrate zero-shot transfer to structured and forested environments with unfamiliar scene content.

Natapat Kirdwichai, Tobias Driskell-Poole, Andrei Sontea +3

The authors are with the Faculty of Engineering and Physical Sciences and the Faculty of Environmental and Life Sciences, University of Southampton, United Kingdom.

Image Corruption Robustness Robot Locomotion Vision-Based Robot Control Robust RL

Ask PDF

Sep 3, 2026

Rethinking 3D Noise: Learning 3D-Aware Video Priors via Optimization-Free Morphological Perturbations

3D scene representations like NeRF and 3D Gaussian Splatting (3DGS) suffer severe artifacts in sparse-view settings. Recent generative 3D artifact fixers attempt to address this, but rely on paired corrupted and clean renders requiring costly, per-scene reconstructions across varying view configurations. While 2D image augmentations act as instant regularizers, no explicit equivalents exist for 3D representations to preserve spatial consistency across views, an essential property for 3D-aware training. We propose 3D Morphological Perturbations as an optimization-free regularizer that preserves spatial consistency. Leveraging explicit 3DGS, we treat each Gaussian as a fundamental building block - analogous to a 2D pixel - and apply perturbations across its morphological parameter space via scale, rotation, and pruning. Our method eliminates per-scene 3DGS optimization loops from dataset curation while enabling models to learn stronger geometric priors than sparse-view baselines in diagnostic ablations conducted on a lightweight video diffusion sandbox. Scaled to a 14B-parameter video model via ControlNet, our approach maintains visual fidelity while reducing mean depth error by 12.5% over state-of-the-art image-to-image 3D artifact refiners, ultimately boosting downstream robotics policy success rates by up to 8.0% across 3 of 4 manipulation tasks.

Onat Şahin, Mohammad Altillawi, George Eskandar +2

Technical University of Munich · Huawei Heisenberg Research Center · Tavus

3D Gaussian Splatting Data Augmentation Sparse-View 3D Reconstruction

Ask PDF

Apr 16, 2026

R3D: Revisiting 3D Policy Learning

3D policy learning promises superior generalization and cross-embodiment transfer, but progress has been hindered by training instabilities and severe overfitting, precluding the adoption of powerful 3D perception models. In this work, we systematically diagnose these failures, identifying the omission of 3D data augmentation and the adverse effects of Batch Normalization as primary causes. We propose a new architecture coupling a scalable transformer-based 3D encoder with a diffusion decoder, engineered specifically for stability at scale and designed to leverage large-scale pre-training. Our approach significantly outperforms state-of-the-art 3D baselines on challenging manipulation benchmarks, establishing a new and robust foundation for scalable 3D imitation learning. Project Page: https://r3d-policy.github.io/

Zhengdong Hong, Shenrui Wu, Haozhe Cui +8

Zhejiang University · ShanghaiTech University

3D ViTs Visuomotor Policy Learning Diffusion Policy Imitation Learning Robot Imitation Learning

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

REDACT: Robust Perceptive Locomotion under Unseen Visual Corruption

Rethinking 3D Noise: Learning 3D-Aware Video Priors via Optimization-Free Morphological Perturbations

R3D: Revisiting 3D Policy Learning