Learning-based 3D Scanning plays a crucial role in enabling efficient and accurate scanning of target objects. However, recent reinforcement learning-based methods often require large-scale training data and still struggle to generalize to unseen object categories.In this work, we propose a data-efficient 3D scanning framework that uses Diffusion Policy to imitate human-like scanning strategies. To enhance robustness and generalization, we adopt the Occupancy Grid Mapping instead of direct point cloud processing, offering improved noise resilience and handling of diverse object geometries. We also introduce a hybrid approach combining a sphere-based space representation with a path optimization procedure that ensures path safety and scanning efficiency. This approach addresses limitations in conventional imitation learning, such as redundant or unpredictable behavior. We evaluate our method on diverse unseen objects in both shape and scale. Ours achieves higher coverage and shorter paths than baselines, while remaining robust to sensor noise. We further confirm practical feasibility and stable operation in real-world execution.
Figures & tables
Fig. 1 : Generalizable 3D Scanning Policy. Our proposed framework ScanDP can generalize to unseen objects with a small amount of training data (Right). Prior works lack generalization capability and are not robust to changes in conditions (left).
Input
Camera
Encoder
Pose
Image
Point Cloud
Count
Mobile
Diffusion Policy [ 11 ]
✓
✓
✗
2
✗
ResNet
3D Diffusion Policy [ 15 ]
✓
✗
✓
1
✗
MLP
GenDP [ 17 ]
✓
✓
✓
4
✗
PointNet++ [ 18 ]
ScanDP (ours)
✓
✗
✓
1
✓
SparseConv [ 19 ]
TABLE I : Comparison of prior works.
Fig. 2 : Method Overview. ScanDP consists of two main components: Path Generation (Upper part) and Path Optimization (Lower part). In the Path Generation phase, actions are generated from a diffusion policy conditioned on an occupancy grid map (OGM) and the current camera pose. In the Path Optimization phase, the generated actions are refined through a bubble-based collision filter and viewpoint extraction to optimize the scanning trajectory.
TABLE II : Evaluation results of Scanning policies. All policies are trained only on trajectories scanning the Stanford Bunny . Coverage is shown in mean ± std. We find that ScanDP consistently achieves the highest coverage while maintaining low variance.
Fig. 3 : Path length comparison (Scale × 1.0)
Fig. 4 : Path length comparison (Scale × 1.5)
Fig. 5 : Visualization of movement paths from DP3 and ScanDP. ScanDP with path optimization attains the shortest path length and smoothest movement. We find that DP3 tends to get stuck in a particular location when scanning objects not seen during training.
TABLE III: Coverage [%] under noisy inputs.
Camera
L515 ( 70∘×43∘ )
D435 ( 87∘×58∘ )
D415 ( 65∘×40∘ )
DP3 [ 15 ]
58.36
60.89
61.97
ScanDP
89.44
83.13
97.40
TABLE IV: Coverage [%] under different FoV.
Variant
Coverage [%] ↑
Path length [m] ↓
ScanDP
97.84
19.56
Thresholded OGM
42.38
51.72
Grid size = 0.01 m
N/A
N/A
Grid size = 0.04 m
65.49
21.58
TABLE V: Ablation results. The default configuration (probabilistic OGM and 0.02 m grid size) gives the best overall trade-off.
Fig. 6 : Real-world experiment setup. We use a 6-DoF manipulator with an RGB-D sensor, combined with a turntable to provide an additional rotational degree of freedom.
Fig. 7 : Real-world Evaluation Results. ScanDP demonstrates feasible and stable scanning behavior in real-world conditions.
We present Seen2Scene, the first flow matching-based approach that trains directly on incomplete, real-world 3D scans for scene completion and generation. Unlike prior methods that rely on complete and hence synthetic 3D data, our approach introduces visibility-guided flow matching, which explicitly masks out unknown regions in real scans, enabling effective learning from real-world, partial observations. We represent 3D scenes using truncated signed distance field (TSDF) volumes encoded in sparse grids and employ a sparse transformer to efficiently model complex scene structures while masking unknown regions. We employ 3D layout boxes as an input conditioning signal, and our approach is flexibly adapted to various other inputs such as text or partial scans. By learning directly from real-world, incomplete 3D scans, Seen2Scene enables realistic 3D scene completion for complex, cluttered real environments. Experiments demonstrate that our model produces coherent, complete, and realistic 3D scenes, outperforming baselines in completion accuracy and generation quality.
Quan Meng, Yujin Chen, Lei Li +2
Technical University of Munich · University of Virginia
3D policy learning promises superior generalization and cross-embodiment transfer, but progress has been hindered by training instabilities and severe overfitting, precluding the adoption of powerful 3D perception models. In this work, we systematically diagnose these failures, identifying the omission of 3D data augmentation and the adverse effects of Batch Normalization as primary causes. We propose a new architecture coupling a scalable transformer-based 3D encoder with a diffusion decoder, engineered specifically for stability at scale and designed to leverage large-scale pre-training. Our approach significantly outperforms state-of-the-art 3D baselines on challenging manipulation benchmarks, establishing a new and robust foundation for scalable 3D imitation learning. Project Page: https://r3d-policy.github.io/
LiDAR perception is severely limited by the distance-dependent sparsity of distant objects. While diffusion models can recover dense geometry, they suffer from prohibitive latency and physical hallucinations manifesting as ghost points. We propose Scanline-Consistent Range-Aware Diffusion, a framework that treats densification as probabilistic refinement rather than generation. By leveraging Partial Diffusion (SDEdit) on a coarse prior, we achieve high-fidelity results in just 156ms. Our novel Ray-Consistency loss and Negative Ray Augmentation enforce sensor physics to suppress artifacts. Our method achieves state-of-the-art results on KITTI-360 and nuScenes, directly boosting off-the-shelf 3D detectors without retraining. Code will be made available.
Zeping Zhang, Robert Laganière
VIVA Research Lab, Department of Electrical and Computer Engineering, University of Ottawa, Ottawa, Ontario, Canada