Point cloud upsampling is essential for 3D applications but remains challenging due to sparse, under-constrained observations. Multi-view images can resolve geometric ambiguity, but existing multi-modal methods do not fully exploit visual cues. We propose ControlFlow3D, a framework that distills knowledge from pretrained foundation models into a lightweight generation network during training, enabling strong performance without multi-view images available at test time. Specifically, our method performs conditional flow matching in a structured latent space defined by a frozen 3D encoder. Multi-view image tokens and 3D latent tokens are jointly clustered into shared prototypes via unsupervised soft KMeans; the resulting prototypes guide the flow trajectory through velocity control and are aligned by a consistency loss that progressively transfers image knowledge into the base network. A Mamba-based geometry decoder then maps latent features to dense coordinates with linear complexity. We also construct ShapeNetPU, a 35K-object multi-modal benchmark that is 31 times larger than prior datasets. Extensive experiments across multiple benchmarks demonstrate the effectiveness of our method and its strong generalization capability.
Figures & tables
Figure 1: Motivation and empirical overview of ControlFlow3D.
Figure 2: Overview of our Latent Flow Matching. We first use a frozen pretrained 3D encoder (Utonia) to define a structured target latent space for dense geometry. Within this space, we design a conditional flow-matching generation network that transports sparse-conditioned features toward the dense prior target. To distill multi-view knowledge into this network, we introduce an unsupervised prototype-guided velocity controller that jointly clusters image Fimg and 3D tokens F3D into shared prototypes. During inference, the model takes only the sparse point cloud as input and produces the final dense point cloud.
Figure 3: Architecture of the Patch-wise Mamba-based Geometry Decoder. The decoder fuses local patch features with the terminal latent endpoint x1 , then applies cascaded GeoFFN and Mamba blocks to predict dense patches with supervision on intermediate and final outputs.
Table 4
Figure 4: Qualitative visualization on ShapeNetPU under the 4× upsampling setting.
Figure 5: Qualitative visualization on PU-GAN under the 4× upsampling setting.
Figure 6: More visualization results. (a) Visualization of ScanNet; (b) Visualization of Arbitrary-Rate Upsampling.
Variant
CD ↓
HD ↓
P2F ↓
Image Distillation
w/o distillation
0.329
2.842
1.139
w/ distillation
0.280
2.259
0.818
2D Backbone
DINOv2 [ 42 ]
0.292
2.374
0.884
VGGT [ 15 ]
0.280
2.259
0.818
Table 3: Ablation studies on ShapeNetPU under 4× upsampling. Lower is better.
Figure 7: Qualitative comparison with and without image distillation on ShapeNetPU under the 4× upsampling setting.
Figure 8: Shared-prototype visualization of a chair: colored 3D input (left), eight RGB views (top right), and assignment maps (bottom right). Colors denote shared prototype indices.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Dataset statistics and benchmark comparison for ShapeNetPU. (a) The category distribution and ShapeNet synset mapping cover 13 object classes. (b) Compared with PU-GAN and PU1K, ShapeNetPU retains the standard sparse and dense point-cloud patch sizes while additionally providing multi-view images.
Figure 10: Visualization of point-wise Utonia features on representative shapes. The feature responses vary smoothly over coherent geometric regions, indicating that the frozen 3D encoder provides a structured latent space for sparse-to-dense transport.
Variant
λFM
λproto
CD ↓
HD ↓
P2F ↓
No auxiliary latent loss
0
0
0.338
2.901
1.168
Unit auxiliary weights
1
1
0.347
3.018
1.241
Weak FM
0.0015 → 0.00075
0.01 → 0.005
0.292
2.388
0.891
Strong FM
0.006 → 0.003
0.01 → 0.005
0.290
2.371
0.878
Weak prototype alignment
0.003 → 0.0015
0.005 → 0.0025
0.287
2.342
0.860
Strong prototype alignment
0.003 → 0.0015
0.02 → 0.01
0.286
2.336
0.852
Appendix
Table 4: Loss-weight ablation on ShapeNetPU under 4× upsampling. Lower is better. The default setting is used in all main experiments.
Variant
Main changed factor
CD ↓
HD ↓
P2F ↓
No image-guided controller
Base velocity only
0.329
2.842
1.139
All-token image fusion
Raw VGGT cross-attn
0.303
2.501
0.958
Image-only prototypes
No target 3D tokens
0.296
2.421
0.900
Hard query assignment
Nearest prototype
0.291
2.366
0.872
Scale-only controller
Use g⊙vθ only
0.293
2.394
0.886
Bias-only controller
Use vθ+b only
0.289
2.352
0.862
Appendix
Table 5: Additional ablations of the prototype controller on ShapeNetPU under 4× upsampling. Lower is better.
Variant
Main changed factor
CD ↓
HD ↓
P2F ↓
Gaussian source
Gaussian x0
0.318
2.721
1.061
Learned source
Learned x0
0.301
2.494
0.941
Train Utonia
Jointly update encoder
0.305
2.548
0.956
Independent target
Unaligned target tokens
0.307
2.566
0.968
1-NN target alignment
Replace 3-NN with 1-NN
0.292
2.373
0.875
No source perturbation
Set σ=0
0.287
2.326
0.850
Appendix
Table 6: Additional ablations of latent-state construction and geometry decoding on ShapeNetPU under 4× upsampling. Lower is better.
Factor
Setting
CD ↓
HD ↓
P2F ↓
Number of training views
Views
1 view
0.306
2.536
0.994
Views
2 views
0.297
2.430
0.918
Views
4 views
0.287
2.328
0.850
Views
12 views
0.284
2.302
0.838
Views
8 views (default)
0.280
2.259
0.818
Appendix
Table 7: Additional budget ablations on ShapeNetPU under 4× upsampling. Each block changes one factor while keeping all other defaults fixed. Lower is better.
Figure 11: More visualization results on ShapeNetPU.
Figure 12: More visualization results on PU-GAN under the 4× upsampling setting.
Noise Levels
τ=0.01
τ=0.02
τ=0.05
τ=0.10
Methods
CD ↓
HD ↓
P2F ↓
CD ↓
HD ↓
P2F ↓
CD ↓
HD ↓
P2F ↓
CD ↓
HD ↓
P2F ↓
Grad-PU [ 4 ]
0.638
5.462
4.684
1.176
8.936
9.752
1.384
10.284
15.876
1.692
11.438
18.420
APU-LDI [ 41 ]
0.586
5.108
3.986
1.084
8.412
8.926
1.276
9.672
14.286
1.548
10.864
17.126
PUDM [ 29 ]
0.542
4.946
4.214
0.986
7.846
8.684
1.168
9.142
13.768
1.406
10.326
16.204
PUFM [ 31 ]
0.518
4.812
4.386
0.948
7.632
8.984
1.126
9.284
14.026
1.358
10.052
15.862
Ours
0.431
3.620
2.384
0.792
6.684
5.426
0.982
8.216
9.384
1.138
9.428
11.562
Appendix
Table 8: Noise robustness comparison on ShapeNetPU under 4× upsampling. Lower values are better.
Rates
PUFM [ 31 ]
APU-LDI [ 41 ]
Ours
CD ↓
HD ↓
P2F ↓
CD ↓
HD ↓
P2F ↓
CD ↓
HD ↓
P2F ↓
5×
0.325
3.260
1.707
0.342
3.105
1.505
0.268
2.300
0.893
7×
0.306
3.238
1.762
0.311
3.025
1.620
0.258
2.410
0.970
8×
0.299
3.252
1.790
0.299
3.045
1.600
0.251
2.335
1.002
15×
0.268
3.226
1.890
0.251
2.970
1.745
0.225
2.465
1.150
20×
0.255
3.178
1.945
0.238
2.925
1.860
0.217
2.575
1.210
Appendix
Table 9: Arbitrary-rate upsampling comparison with PUFM and APU-LDI on ShapeNetPU. Lower values are better.
Figure 13: Failure analysis of ControlFlow3D. Red boxes highlight challenging regions in the sparse input, our output, and the dense ground truth (GT).
Generating high-quality 3D point clouds requires capturing both global shape topology and local geometric details. Existing flow-based methods rely on continuous normalizing flows (CNFs) that demand expensive ODE solving and trace estimation during training, while diffusion models require hundreds of iterative denoising steps. Moreover, most approaches adopt single-level generation directly in point space, disregarding the hierarchical structure natural to 3D shapes. We propose Hierarchical Flow Matching (HFM) that extends flow matching to bilevel structure for unconditional 3D point cloud generation. HFM decomposes the task into two levels via optimal-transport flow matching: a Latent Flow Matching models the global shape manifold in a compact latent space, and a Conditional Point Flow Matching reconstructs detailed point clouds conditioned on the latent code. Both flows are trained with simple MSE regression losses. The resulting straight OT paths enable efficient sampling with as few as 15 Euler steps per flow, while the structured latent space supports downstream tasks including classification. Extensive experiments on ShapeNet and ModelNet benchmarks demonstrate that HFM achieves competitive or even best performance compared with prior state-of-the-art methods.
3D point clouds generation requires capturing both part-level layout and surface detail while preserving permutation equivariance. Existing latent generative models often encode an entire shape into a single vector, which limits their ability to retain part-level structure. We propose TokenFlow, a hierarchical flow matching framework that represents each shape as a set of latent tokens and couples a rectified flow over these tokens with a conditional flow over points. A bottleneck of M≪N learnable tokens aggregates global context through cross-attention at O(NM) cost. On ShapeNet, TokenFlow achieves the best MMD and coverage across airplane, car, and chair, while obtaining 1-NNA comparable to the strongest baselines. On ModelNet, its learned latent set reaches the best reported linear-evaluation accuracy on ModelNet10. These results show that representing a shape as a set of latent tokens, aggregated at O(NM) cost, provides an effective representation for 3D point cloud generation.
Flow-matching models have recently emerged as a powerful framework for generative modeling, including 3D point cloud synthesis. However, their deployment is limited by the need for multiple sequential sampling steps at inference time. MeanFlow enables single-step generation by learning an average velocity field, but its quality depends heavily on the geometry of the training pairs. In this work, we propose an Optimal Transport-enhanced MeanFlow framework (OT-MF3D) for efficient and accurate 3D point cloud generation and completion. Beyond incorporating OT into MeanFlow, our method introduces a latent-conditioned training algorithm that decouples shape-level sampling from point-level matching. This enables exact point-level OT within each conditioned shape while performing shape-level transport in a compact latent space, avoiding the expensive nested OT computation in point-cloud space. Experiments on standard benchmarks show improved generation and completion quality over fast baselines, with substantially reduced training and inference costs compared with diffusion and other flow-based models.
Elaheh Akbari, Shansita Sharma, Ping He +5
Department of Computer Science, Vanderbilt University · Toyota InfoTech Labs · Department of Computer Science, University of California, Davis +1