Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning

Changbo Yan; Zhongbo Zhang; Zaibin Zhang; Yifan Wang; Lijun Wang; Huchuan Lu

cs.ROSep 10, 2026

Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning

Authors: Changbo Yan, Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Lijun Wang, Huchuan Lu

Organizations: Dalian University of Technology

Abstract

3D point-cloud observations are inherently ambiguous in complex, cluttered manipulation scenes, where target objects may be partially occluded or tightly intermingled with visually similar distractors. As a result, standard 3D diffusion policies often struggle to localize and exploit task-relevant geometry as scene complexity grows. We propose Attention-DP3, a spatially object-aware 3D diffusion policy that injects object-level geometric cues via attention while keeping the DP3 diffusion backbone unchanged. Our pipeline performs open-vocabulary 2D segmentation on RGB images, then lifts predicted target masks into 3D using calibrated camera geometry to obtain object-centric geometric priors. We incorporate these cues through Tri-field Attentional Conditioning, which constructs three complementary fields: (i) a targetness field to anchor the target object, (ii) an intra-target saliency field to emphasize task-relevant geometry within the target, and (iii) a backgroundness field to suppress distractors and clutter. Experiments on Adroit, DexArt, MetaWorld, and the real-world SO101 platform show consistent improvements over DP3, achieving state-of-the-art performance across benchmarks. Notably, as distractor objects increase, DP3 drops sharply, whereas Attention-DP3 remains stable and outperforms DP3 by up to 31% under heavy clutter. The code is publicly available at https://github.com/zhangzhongbo2213/Attention-DP3.

Figures & tables

Figure 1: Standard 3D policies vs. Attention-DP3. (Left) In cluttered scenes, sparse point clouds can be perceptually ambiguous under occlusion and distractors, causing unreliable target association. (Right) Attention-DP3 lifts object masks from RGB into 3D and uses geometry-aligned, object-aware attentional conditioning to guide action generation, helping the policy focus on what to grasp and where it is in 3D.

Figure 2: Attention-DP3 overview. (a) Pipeline. The policy conditions on synchronized RGB, 3D point clouds, and proprioception. A 3D Perception Module encodes global geometry, a State Encoder embeds proprioception, and an Attn-Enhanced Module maps open-vocabulary 2D cues to an attention feature. Their embeddings condition a diffusion Decision Module to denoise an action trajectory. (b) Attn-Enhanced Module. A frozen grounding-and-segmentation pipeline predicts a text-conditioned 2D mask, which is lifted to three geometry-aligned fields: Targetness, Intra-target Saliency, and Backgroundness. These fields are individually encoded by Field Encoders and subsequently aggregated via an MLP. (c) Perception Encoder. A lightweight point-cloud backbone encodes raw point clouds into a compact 3D representation.

Method	Adroit			Avg	DexArt				Avg
Method	Door	Pen	Hammer	Avg	Laptop	Faucet	Toilet	Bucket	Avg
IBC [ 6 ]	0.00	0.00	0.09	0.03	0.03	0.07	0.00	0.14	0.06
BCRNN [ 24 ]	0.00	0.00	0.09	0.03	0.03	0.01	0.00	0.05	0.02
AdaFlow [ 11 ]	0.27	0.18	0.45	0.30	–	–	–	–	–
CP [ 27 ]	0.31	0.13	0.45	0.30	–	–	–	–	–
$\mathrm{H}^{3}\mathrm{DP}$ [ 23 ]	–	–	–	–	0.81	0.34	0.70	0.28	0.53

Table 1: Success Rates on Adroit and DexArt Benchmarks. Missing entries (–) indicate methods not evaluated on that benchmark.

Method	Easy				Medium				Hard				Very Hard				Overall
	handle-pull	handle-pull-side	$\cdots$	Avg	push-wall	hammer	$\cdots$	Avg	pick-place	push	$\cdots$	Avg	pick-place-wall	disassemble	$\cdots$	Avg
DP	0.27	0.23	–	0.836	0.20	0.15	–	0.311	0.00	0.30	–	0.090	0.05	0.43	–	0.266	0.574
DP3	0.34	0.53	–	0.857	0.49	0.76	–	0.525	0.12	0.51	–	0.248	0.35	0.69	–	0.442	0.669
VITA	0.20	0.52	–	0.839	0.49	0.90	–	0.546	0.15	0.70	–	0.315	0.48	0.61	–	0.548	0.683
Ours	0.46	0.62	–	0.870	0.92	0.95	–	0.599	0.54	0.82	–	0.378	0.60	0.86	–	0.620	0.726

Table 2: MetaWorld results by difficulty with representative tasks. For each difficulty, we show two representative tasks plus an ellipsis indicating additional tasks in the same split, and report the difficulty-average (Avg) over all tasks in that split; Overall is the mean over all MetaWorld tasks. Full per-task results are provided in the supplementary material.

Task	Stack Cube	Push Cube	Place Cube	Average
DP3	0.50	0.30	0.75	0.52
Attention-DP3	0.70	0.65	0.85	0.73

Table 3: Real-world results on SO101. Attention-DP3 consistently outperforms DP3 across all tasks.

Figure 3: Real-world experimental setting on SO101. We evaluate three tasks, including Place Cube, Push Cube, and Stack Cube.

Panel of Figure 4 — Figure 4: Robustness to visual clutter. Our attention mechanism improves robustness under unseen distractors across both tasks.

Figure 5: Qualitative comparison on MetaWorld Stick-Push with four distractor blocks. Top: DP3; bottom: Attention-DP3.

Method	Orig.	+Clip	+Stick	+Extra
DP3	0.52	0.43	0.38	0.13
Attention-DP3	0.73	0.72	0.65	0.45

Table 4: Real-world clutter robustness on SO101. Average success rate across Place Cube, Push Cube, and Stack Cube under progressively added distractors.

Variant	T	I	B	Door	Stick-Pull	Pick-Place	Push-Wall	Avg.
Full	$\checkmark$	$\checkmark$	$\checkmark$	0.83	0.59	0.54	0.92	0.72
T only	$\checkmark$			0.74	0.53	0.33	0.57	0.5425
I only		$\checkmark$		0.73	0.51	0.32	0.64	0.55
B only			$\checkmark$	0.68	0.50	0.33	0.63	0.535
T+I	$\checkmark$	$\checkmark$		0.80	0.30	0.39	0.58	0.5175
T+B	$\checkmark$		$\checkmark$	0.75	0.56	0.36	0.58	0.5625

Table 5: Ablation on semantic attention fields. We report success rates on Adroit Door and MetaWorld tasks. Avg. is the mean over the four tasks.

Fusion Strategy	Encoder	Door	Stick-Pull	Pick-Place	Push-Wall	Avg.
Late fusion	3 AttnEncoders + DP3Encoder	0.83	0.59	0.54	0.92	0.72
Late fusion	Single AttnEncoder + DP3Encoder	0.69	0.42	0.36	0.62	0.5225
Early fusion	Attn+PC concat $\rightarrow$ DP3 Encoder	0.66	0.02	0.07	0.27	0.255

Table 6: Ablation on fusion strategy. We report success rates on Adroit Door and MetaWorld tasks. Avg. is the mean over the four tasks.

Task	DP3 (Baseline)	Attn-DP3 (Ours)	Latency Gap
Adroit Pen	0.120 s	0.213 s	+0.093 s
Adroit Door	0.124 s	0.174 s	+0.050 s
Adroit Hammer	0.123 s	0.209 s	+0.086 s
Mean	0.122 s	0.198 s	+0.076 s

Table 7: Effective Control Step Latency. Per-step latency

L_{step}

for execution window

k=8

(

T_{phys}=0.8

s), averaging physical execution and inference latency

T_{inf}

. Measurements use one NVIDIA RTX 3090 GPU.

Task	Total Failures	Attn-DP3 (Fail via DINO)	DP3 (Overlap Fail)	DP3 (Total Fail)
Coffee-Push	10	8	7 / 8	9 / 10
Peg-Unplug-Side	10	7	5 / 7	8 / 10
Reach-Wall	10	0	N/A	10 / 10

Table 8: Diagnostic Analysis of Failure Mode Overlap. We isolate 10 Attention-DP3 failure episodes and rerun DP3 with identical seeds. "Overlap Fail" reports DP3 failures when Attention-DP3 suffers Grounding DINO errors.

Method	Hammer	Box-Close	Push-Wall	Pick-Place	Shelf-Place	Stick-Pull
DP3	0.76	0.42	0.49	0.12	0.17	0.27
Attn-DP3 (Swin-B)	0.95	0.52	0.92	0.54	0.22	0.59
Attn-DP3 (Swin-T)	1.00	0.43	0.84	0.47	0.17	0.53

Table 9: Robustness to Perceptual Foundation Models. Success rates of DP3, Attention-DP3, and Attention-DP3 (Swin-T) using a lightweight Grounding DINO Swin-T backbone.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,704 papers

PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
Jul 2, 2026Haofei Xu, Rundi Wu, Philipp Henzler +7Diffusion Transformer Monocular 3D Reconstruction
Google · ETH Zurich · University of Tübingen, Tübingen AI Center +3
PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation
Mar 18, 2026Wenbin Tan, Jiawen Lin, Fangyong Wang +43D Referring Segmentation 3D Visual Grounding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen, China · School of Big Data, Tongren University, Tongren, China · Hanjiang National Laboratory, Wuhan, China +2
ReMAP-DP: Reprojected Multi-view Aligned PointMaps for Diffusion Policy
Mar 16, 2026Xinzhang Yang, Renjun Wu, Jinyan Liu +1Vision-Based Robot Control Diffusion Policy
School of Computer Science, Beijing Institute of Technology, China.

Jul 2, 2026J/K move · Enter open · S save

PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

Haofei Xu, Rundi Wu, Philipp Henzler, Nikolai Kalischek+6

Google · ETH Zurich · University of Tübingen, Tübingen AI Center+3

Diffusion Transformer Monocular 3D Reconstruction Pixel-Space Diffusion Models Single-View 3D Reconstruction 3D Diffusion Models Diffusion Models

Open paper AskPDF arXiv

State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.

Abstract

Robotic Manipulation

Explore similar work

PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation

ReMAP-DP: Reprojected Multi-view Aligned PointMaps for Diffusion Policy