Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis
Organizations: Independent Researcher
Abstract
The remarkable success of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in 2D vision has spurred significant research in extending these architectures to the complex domain of 3D analysis. Yet, a core challenge arises from a fundamental dichotomy between the regular, dense grids of 2D images and the irregular, sparse nature of 3D data such as point clouds and meshes. This survey provides a comprehensive review and a unified taxonomy of adaptation strategies that bridge this gap, classifying them into three families: (1) Data-centric methods that project 3D data into 2D formats to leverage off-the-shelf 2D models, (2) Architecture-centric methods that design intrinsic 3D networks, and (3) Hybrid methods, which synergistically combine the two modeling paradigms to benefit from both rich visual priors of large 2D datasets and explicit geometric reasoning of 3D models. Through this framework, we qualitatively analyze the fundamental trade-offs between these families concerning computational complexity, reliance on large-scale pre-training, and the preservation of geometric inductive biases. We discuss key open challenges and outline promising future research directions, including the development of 3D foundation models, advancements in self-supervised learning (SSL) for geometric data, and the deeper integration of multi-modal signals.
Figures & tables
| Representation | Description | Pros | Cons |
|---|---|---|---|
| Projective or View-based | Collection of 2D images from multiple viewpoints. | Allows direct reuse of mature 2D models; highly efficient. | Loses geometric information due to occlusion; not ”true” 3D. |
| Volumetric | 3D space discretized into a regular grid of voxels. | Regular structure enables straightforward 3D CNN extension. | Memory/compute scale cubically ( ); inefficient for sparse data. |
| Geometric Primitives | Explicit surface geometry via Point Clouds (unordered coordinates) or Meshes (vertices/edges). | Memory-efficient; captures explicit surface topology. | Irregular structure violates standard CNN assumptions. |
| Implicit | Continuous functions (e.g., NeRFs [ 44 ] , SDFs [ 47 ] ) parameterized by networks. | Resolution-free; compact encoding of complex shapes. | Slow query speeds; better for synthesis than real-time analysis. |
3D Representation Learning
3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.
See all 191 papers