3D Reconstruction

Ask about this topic

Overview

3D reconstruction studies how to infer the shape and appearance of objects or scenes, along with camera positions and motion, from partial observations. Methods range from structure-from-motion and multi-view stereo to neural radiance fields, Gaussian splatting, signed distance fields, meshes, and feed-forward models that predict geometry directly; some also use diffusion priors, active view selection, or physical sensor models. Applications include mapping and robotics, digital twins, medical imaging, and human or object modeling, with evaluations often comparing both rendered views and geometric accuracy on indoor, outdoor, and specialized datasets. A central challenge is that realistic novel-view images do not always imply accurate, metrically consistent geometry, especially with sparse or unposed views, changing scenes, difficult materials, or limited computation.

Latest papers

CardsList

957 papers121 papers in the last four weeks, up 27% on the four weeks before. 0.7% of all new papers.

Nov 12, 2025

OUGS: Active View Selection via Object-aware Uncertainty Estimation in 3DGS

Recent advances in 3D Gaussian Splatting (3DGS) have achieved state-of-the-art results for novel view synthesis. However, efficiently capturing high-fidelity reconstructions of specific objects within complex scenes remains a significant challenge. A key limitation of existing active reconstruction methods is their reliance on scene-level uncertainty metrics, which are often biased by irrelevant background clutter and lead to inefficient view selection for object-centric tasks. We present OUGS, a novel framework that addresses this challenge with a more principled, physically-grounded uncertainty formulation for 3DGS. Our core innovation is to derive uncertainty directly from the explicit physical parameters of the 3D Gaussian primitives (e.g., position, scale, rotation). By propagating the covariance of these parameters through the rendering Jacobian, we establish a highly interpretable uncertainty model. This foundation allows us to then seamlessly integrate semantic segmentation masks to produce a targeted, object-aware uncertainty score that effectively disentangles the object from its environment. This allows for a more effective active view selection strategy that prioritizes views critical to improving object fidelity. Experimental evaluations on public datasets demonstrate that our approach significantly improves the efficiency of the 3DGS reconstruction process and achieves higher quality for targeted objects compared to existing state-of-the-art methods, while also serving as a robust uncertainty estimator for the global scene.

Haiyi Li, Qi Chen, Denis Kalkofen +1

Object-Centric 3D Reconstruction3D Reconstruction3D Gaussian SplattingUQ for 3D Reconstruction
AskPDF
Nov 4, 2025

3D Cal: An Open-Source Software Library for Depth Reconstruction on Vision-Based Tactile Sensors

Tactile sensing plays a key role in enabling dexterous and reliable robotic manipulation, but realizing this capability requires substantial calibration to convert raw sensor readings into physically meaningful quantities. Despite its near-universal necessity, the calibration process remains ad hoc and labor-intensive. Here, we introduce 3D Cal, an open-source library that transforms a low-cost 3D printer into an automated probing device capable of generating large volumes of labeled training data for calibrating vision-based tactile sensors. 3D Cal also provides an end-to-end, user-friendly pipeline for training custom convolutional networks to produce high-quality depth reconstructions. Using 3D Cal, we systematically explore the relationship between training data volume and spatial reconstruction performance on two commercially available sensors, DIGIT and GelSight Mini, and derive practical, empirically-grounded guidelines for calibrating these sensors. Finally, we demonstrate depth reconstruction performance on the DIGIT and GelSight Mini comparable to state-of-the-art methods, achieving average reconstruction errors of 156 μm\mathrm{μm} and 205 μm\mathrm{μm} on unseen objects, respectively. By automating tactile sensor calibration, 3D Cal can accelerate tactile sensing research, simplify sensor deployment, and facilitate the integration of tactile sensing in robotic platforms.

Rohan Kota, Kaival Shah, J. Edward Colgate +1

McCormick School of Engineering, Center for Robotics and Biosystems, Northwestern University, Evanston, IL USA

Vision-Based Tactile SensingTactile Sensing3D ReconstructionDepth EstimationRobotic Tactile Sensing
AskPDF
Oct 21, 2025

OREN: Octree Residual Network for Real-Time Euclidean Signed Distance Mapping

Reconstructing signed distance functions (SDFs) from point cloud data benefits many robot autonomy capabilities, including localization, mapping, motion planning, and control. Methods that support online and large-scale SDF reconstruction often rely on discrete volumetric data structures, which affects the continuity and differentiability of the SDF estimates. Neural network methods have demonstrated high-fidelity differentiable SDF reconstruction but they tend to be less efficient, experience catastrophic forgetting and memory limitations in large environments, and are often restricted to truncated SDF. This work proposes OREN, a hybrid method that combines an explicit prior from octree interpolation with an implicit residual from neural network regression. Our method achieves non-truncated (Euclidean) SDF reconstruction with computational and memory efficiency comparable to volumetric methods and differentiability and accuracy comparable to neural network methods. Extensive experiments demonstrate that OREN outperforms the state of the art in terms of accuracy and efficiency, providing a scalable solution for downstream tasks in robotics and computer vision.

Zhirui Dai, Qihao Qian, Tianxing Fan +1

Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA

3D MappingImplicit Neural RepresentationsSigned Distance Functions3D Reconstruction
AskPDF
Oct 21, 2025

PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting

This paper addresses metric 3D reconstruction of indoor scenes by exploiting their inherent geometric regularities with compact representations. Using planar 3D primitives - a well-suited representation for man-made environments - we introduce PLANA3R, a pose-free framework for metric Planar 3D Reconstruction from unposed two-view images. Our approach employs Vision Transformers to extract a set of sparse planar primitives, estimate relative camera poses, and supervise geometry learning via planar splatting, where gradients are propagated through high-resolution rendered depth and normal maps of primitives. Unlike prior feedforward methods that require 3D plane annotations during training, PLANA3R learns planar 3D structures without explicit plane supervision, enabling scalable training on large-scale stereo datasets using only depth and normal annotations. We validate PLANA3R on multiple indoor-scene datasets with metric supervision and demonstrate strong generalization to out-of-domain indoor environments across diverse tasks under metric evaluation protocols, including 3D surface reconstruction, depth estimation, and relative pose estimation. Furthermore, by formulating with planar 3D representation, our method emerges with the ability for accurate plane segmentation. The project page is available at https://lck666666.github.io/plana3r

Changkun Liu, Bin Tan, Zeran Ke +6

The Hong Kong University of Science and Technology · Ant Group · Wuhan University +2

3D ReconstructionIndoor Scene ReconstructionFeed-Forward 3D ReconstructionMulti-View 3D Reconstruction
AskPDF
Oct 20, 2025

OmniRaindrop: A Comprehensive Benchmark for Dual-Focused Raindrop Removal under Day and Night Scenes

Adherent raindrops on camera lenses are a pervasive source of real-world image degradation, undermining outdoor visual perception for downstream applications such as robotics and 3D scene reconstruction. However, most existing raindrop datasets and methods target only background-focused 2D images, overlooking the equally common raindrop-focused case and the impact of raindrops on 3D scene reconstruction. Moreover, they are almost exclusively daytime, lacking the complex artificial lighting and reflections of nighttime raindrop scenes. To address these issues, we introduce OmniRaindrop, a real-world dataset and benchmark that jointly evaluates 2D raindrop removal and 3D scene reconstruction across raindrop-focused and background-focused captures, under both daytime and nighttime conditions. OmniRaindrop provides spatially aligned raindrop-focused, background-focused, and rain-free ground-truth image sets: a 2D subset of 15,186 aligned pairs/triplets and a 3D subset of 40 real-world scenes evenly split between day and night, totaling 3,822 multi-view images. On top of this data, the benchmark traces how raindrop degradation propagates from 2D restoration to 3D reconstruction by testing representative restoration methods and their effects on pose estimation, point cloud initialization, and 3DGS optimization. Comprehensive experiments reveal that raindrop-focused scenes are substantially harder than background-focused ones, and that errors in 2D restoration and pose/point cloud initialization cascade into degraded 3D reconstruction, pointing to clear directions for more robust 2D restoration and 3DGS design. Building on the dataset, we have organized two consecutive NTIRE challenges (2025 and 2026) on Day and Night Raindrop Removal for Dual-Focused Images.

Zhiqiang Teng, Tingting Chen, Beibei Lin +4

Beijing Jiaotong University · National University of Singapore

3D ReconstructionMulti-View 3D Reconstruction
AskPDF
Oct 17, 2025

Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery

Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task for immersive and embodied applications. The challenge lies in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by leveraging readily available satellite imagery for realistic coarse geometry and open-domain diffusion models for high-quality close-up appearance synthesis. We propose Skyfall-GS, a novel hybrid framework that synthesizes immersive city-block scale 3D urban scenes by combining satellite reconstruction with diffusion refinement, eliminating the need for costly 3D annotations, and also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic texture. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/

Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai +6

National Yang Ming Chiao Tung University · UIUC · University of Zaragoza +1

3D Reconstruction3D Scene GenerationImage-to-3D Scene GenerationUrban Scene Reconstruction
AskPDF
Sep 21, 2025

SPFSplatV2: Efficient Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views

We introduce SPFSplatV2, an efficient feed-forward framework for 3D Gaussian splatting from sparse multi-view images, requiring no ground-truth poses during training or inference. The framework employs a shared feature extraction backbone to jointly predict 3D Gaussian primitives and camera poses in a canonical space from unposed inputs. To enable efficient and accurate pose estimation, we introduce a masked attention mechanism for target-view pose prediction and a reprojection loss that enforces pixel-aligned Gaussian primitives, providing stronger geometric constraints. We further demonstrate the compatibility of our training framework with different reconstruction architectures, resulting in two model variants. Remarkably, despite the absence of pose supervision, our method achieves state-of-the-art performance in both in-domain and out-of-domain novel view synthesis, even under extreme viewpoint changes and limited image overlap. It also surpasses many methods that rely on geometric supervision in relative pose estimation. By eliminating dependence on ground-truth poses, our method offers the scalability to leverage larger and more diverse datasets. Code and pretrained models will be available on our project page: https://ranrhuang.github.io/spfsplatv2/.

Ranran Huang, Krystian Mikolajczyk

Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, United Kingdom

3D Gaussian Splatting3D ReconstructionCamera Pose EstimationNovel View SynthesisSparse-View 3D ReconstructionFeed-Forward 3D Reconstruction
AskPDF
Sep 16, 2025

Effective Gaussian Management for High-fidelity Scene Reconstruction

This paper proposes an effective Gaussian management framework for high-fidelity scene reconstruction of both appearance and geometry. Unlike recent Gaussian Splatting (GS) pipelines that treat all primitives uniformly during optimization, our framework explicitly manages the attribute activation, representation and pruning of Gaussian. Specifically, our framework first introduces GauSep, a novel densification strategy that selectively activates Gaussian color or normal attributes to alleviate destructive gradient conflicts arising from dual supervision. We further propose GauRep, an adaptive Gaussian representation that dynamically adjusts spherical harmonics (SHs) orders and performs task-decoupled pruning to reduce redundancy at both the individual and global levels. To provide reliable geometric supervision for above mangement process, we additionally introduce CoRe, an regularized surface reconstruction module that distills robust normal fields from an SDF branch to the Gaussian representation through a confidence mechanism. Notably, the proposed Gaussian management is compatible with various reconstruction architectures and can be seamlessly integrated to improve performance while reducing size of the model. Extensive experiments demonstrate that our approach achieves superior or comparable performance in appearance and geometry reconstruction compared with state-of-the-art methods, while using significantly fewer parameters.

Jiateng Liu, Hao Gao, Jiu-Cheng Xie +5

School of Automation, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China · Department of Computer and Information Science, University of Macau, Taipa, Macau · school of Communications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China +2

3D Gaussian Splatting3D ReconstructionAdaptive Gaussian Densification3D Scene Representation
AskPDF
Sep 15, 2025

Neural 3D Object Reconstruction with Small-Scale Unmanned Aerial Vehicles

Miniaturized Uncrewed Aerial Vehicles (UAVs) can access indoor and hard-to-reach spaces, but severe constraints on payload and autonomy have limited their use in demanding tasks such as high-quality 3D reconstruction. We introduce a novel system architecture that enables autonomous, high-fidelity 3D scanning of static objects with sub-100 gram UAVs. Our core innovation lies in a closed-loop active viewpoint selection framework specifically tailored for ultra-constrained micro-platforms, advancing beyond standard static or offline active reconstruction methods. The framework establishes a dual-reconstruction pipeline that creates a real-time (RT) feedback loop between data capture and flight control. A near-RT process uses Structure-from-Motion (SfM) to generate an instantaneous point-cloud of the object. A systematic trajectory adaptation algorithm analyzes the model quality on the fly and dynamically adapts the UAV's trajectory based on parameterized spatial partitioning to intelligently capture new images of poorly covered areas, ensuring comprehensive acquisition. For the final, high-fidelity output, a non-RT pipeline employs a Neural Radiance Fields (NeRF)-based Neural 3D Reconstruction (N3DR) approach, fusing SfM-derived camera poses with precise external location data, evaluated across both radio-based Ultra Wideband (UWB) and visual motion-capture setups, to correct sensor noise and achieve superior accuracy. We implemented and validated this architecture using Crazyflie 2.1 UAVs. Our experiments, conducted in both single- and multi-UAV configurations show that algorithmic dynamic trajectory adaptation consistently improves reconstruction quality over static flight paths. This work demonstrates a scalable and autonomous solution that unlocks the potential of miniaturized UAVs for fine-grained 3D reconstruction, a capability previously reserved for much larger platforms.

Àlmos Veres-Vitàlyos, Filip Lemic, Daniel Johannes Bugelnig +5

i2CAT Foundation, Spain · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia · University of Klagenfurt, Austria +3

Aerial Robotics3D Reconstruction
AskPDF
Sep 15, 2025

End-to-End 4D Heart Mesh Recovery Across Full-Stack and Sparse Cardiac MRI

Reconstructing cardiac motion from CMR sequences is critical for diagnosis, prognosis, and intervention. Existing methods rely on complete CMR stacks to infer full heart motion, limiting their applicability during intervention when only sparse observations are available. We present TetHeart, the first end-to-end framework for unified 4D heart mesh recovery from both offline full-stack and intra-procedural sparse-slice observations. Our method leverages deformable tetrahedra to capture shape and motion in a coherent space shared across cardiac structures. Before a procedure, it initializes detailed, patient-specific heart meshes from high-quality full stacks, which can then be updated using whatever slices can be obtained in real-time, down to a single one during the procedure. TetHeart incorporates several key innovations: (i) an attentive slice-adaptive 2D-3D feature assembly mechanism that integrates information from arbitrary numbers of slices at any position; (ii) a distillation strategy to ensure accurate reconstruction under extreme sparsity; and (iii) a weakly supervised motion learning scheme requiring annotations only at keyframes, such as the end-diastolic and end-systolic phases. Trained and validated on three large public datasets and evaluated zero-shot on additional private interventional and public datasets without retraining, TetHeart achieves state-of-the-art accuracy and strong generalization in both pre- and intra-procedural settings. Code and dataset is available at https://github.com/Scalsol/TetHeart.

Yihong Chen, Jiancheng Yang, Deniz Sayin Mercadier +3

School of Computer and Communication Science, EPFL · ELLIS Institute Finland, Aalto University, Finland · University of North Carolina at Charlotte +1

4D Reconstruction3D ReconstructionImage ReconstructionSparse-View 3D ReconstructionMagnetic Resonance ImagingCardiac MRI
AskPDF
Sep 5, 2025

CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus

Fast and efficient photorealistic 3D reconstruction with (semantic) Gaussian Splatting (GS) is crucial for time-critical robotic perception and navigation, where robots may need to rapidly reconstruct and inspect specific points of interest (POIs), such as in search-and-rescue scenarios. Existing semantic GS methods prioritize reconstruction and segmentation quality while uniformly optimizing entire scenes, often overlooking runtime constraints crucial for real-world deployment. Moreover, object extraction often suffers from floaters and incomplete reconstructions, while post-optimization approaches often produce incomplete POIs. We propose CoRe-GS, a coarse-to-refined, task-driven approach that first creates a segmentation-ready GS representation through lightweight late-stage semantic refinement, then selectively refines only the views and Gaussians associated with the POI. A color-based filtering strategy further suppresses segmentation-induced floaters. On LERF-Mask, our representation achieves competitive mIoU with 6-8x fewer optimization steps. On NeRDS 360 and SCRREAM, CoRe-GS reduces time-to-POI by 91-95% over full-scene semantic GS while improving reconstruction quality and reducing floaters. Across all benchmarks, it is 44-76% faster than post-optimization object extraction, demonstrating that task-aware refinement can deliver efficient, high-quality POI reconstruction for time-critical robotic perception and operation.

Hannah Schieber, Dominik Frischmann, Victor Schaack +4

Technical University of Munich · Human-Centered Computing and Extended Reality Lab · TUM University Hospital, Clinic for Orthopedics and Sports Orthopedics +3

3D Gaussian Splatting3D Reconstruction3D Semantic SegmentationRobotic Perception
AskPDF
Aug 29, 2025

ECHO: Ego-Centric modeling of Human-Object interactions

Modeling human-object interactions (HOI) from an egocentric perspective is a critical yet challenging task, particularly when relying on sparse signals from wearable devices like smart glasses and watches. We present ECHO, the first unified framework to jointly recover human pose, object motion, and contact dynamics solely from head and wrist tracking. To tackle the underconstrained nature of this problem, we introduce a novel tri-variate diffusion process with independent noise schedules that models the mutual dependencies between the human, object, and interaction modalities. This formulation allows ECHO to operate with flexible input configurations, making it robust to intermittent tracking and capable of leveraging partial observations. Crucially, it enables training on a combination of large-scale human motion datasets and smaller HOI collections, learning strong priors while capturing interaction nuances. Furthermore, we employ a smooth inpainting inference mechanism that enables the generation of temporally consistent interactions for arbitrarily long sequences. Extensive evaluations demonstrate that ECHO achieves state-of-the-art performance, significantly outperforming existing methods lacking such flexibility. The project page is available at https://ptrvilya.github.io/echo/.

Ilya A. Petrov, Vladimir Guzov, Riccardo Marin +5

University of Tübingen, Germany · Tübingen AI Center, Germany · Technical University of Munich, Germany +3

HOI Generation3D ReconstructionObject Pose EstimationHuman Pose EstimationDiffusion ModelsHuman-Object Interaction
AskPDF
Aug 20, 2025

GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects

Reconstructing articulated objects is essential for building digital twins of interactive environments. However, prior methods typically decouple geometry and motion by first reconstructing object shape in distinct states and then estimating articulation through post-hoc alignment. This separation complicates the reconstruction pipeline and restricts scalability, especially for objects with complex, multi-part articulation. We introduce a unified representation that jointly models geometry and motion using articulated 3D Gaussians. This formulation improves robustness in motion decomposition and supports articulated objects with up to 20 parts, significantly outperforming prior approaches that often struggle beyond 2--3 parts due to brittle initialization. To systematically assess scalability and generalization, we propose MPArt-90, a new benchmark consisting of 90 articulated objects across 20 categories, each with diverse part counts and motion configurations. Extensive experiments show that our method consistently achieves superior accuracy in part-level geometry reconstruction and motion estimation across a broad range of object types. We further demonstrate applicability to downstream tasks such as robotic simulation and human-scene interaction modeling, highlighting the potential of unified articulated representations in scalable physical modeling.

Licheng Shen, Saining Zhang, Honghan Li +4

BAAI · AIR, THU · NTU +2

Articulated Object Reconstruction3D Gaussian Splatting3D Reconstruction3D Scene Representation
AskPDF
Aug 13, 2025

Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priors

We revisit the role of texture in monocular 3D hand reconstruction, not as an afterthought for photorealism, but as a dense, spatially grounded cue that can actively support pose and shape estimation. Our observation is simple: even in high-performing models, the overlay between predicted hand geometry and image appearance is often imperfect, suggesting that texture alignment may be an underused supervisory signal. We propose a lightweight texture module that embeds per-pixel observations into UV texture space and enables a novel dense alignment loss between predicted and observed hand appearances. Our approach assumes access to a differentiable rendering pipeline and a model that maps images to 3D hand meshes with known topology, allowing us to back-project a textured hand onto the image and perform pixel-based alignment. The module is self-contained and easily pluggable into existing reconstruction pipelines. To isolate and highlight the value of texture-guided supervision, we augment HaMeR, a high-performing yet unadorned transformer architecture for 3D hand pose estimation. The resulting system improves both accuracy and realism, demonstrating the value of appearance-guided alignment in hand reconstruction.

Giorgos Karvounas, Nikolaos Kyriazis, Iason Oikonomidis +2

1ICS-FORTH · University of Crete · University of Texas at Austin

3D Hand Pose Estimation3D ReconstructionMonocular 3D Reconstruction3D Hand Reconstruction3D Mesh Reconstruction
AskPDF
Jul 29, 2025

Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videos

We present Ov3R, a novel framework for open-vocabulary semantic 3D reconstruction from RGB video streams, designed to advance Spatial AI. The system features two key components: CLIP3R, a CLIP-informed 3D reconstruction module that predicts dense point maps from overlapping clips while embedding object-level semantics; and 2D-3D OVS, a 2D-3D open-vocabulary semantic module that lifts 2D features into 3D by learning fused descriptors integrating spatial, geometric, and semantic cues. Unlike prior methods, Ov3R incorporates CLIP semantics directly into the reconstruction process, enabling globally consistent geometry and fine-grained semantic alignment. Our framework achieves state-of-the-art performance in both dense 3D reconstruction and open-vocabulary 3D segmentation, marking a step forward toward real-time, semantics-aware Spatial AI.

Ziren Gong, Xiaohan Li, Fabio Tosi +4

University of Bologna · USTC · BIT +1

Open-Vocabulary 3D Segmentation3D ReconstructionMulti-View 3D Reconstruction3D VLMs
AskPDF
Jun 24, 2025

Online camera-pose-free stereo endoscopic tissue deformation recovery with tissue-invariant vision-biomechanics consistency

Tissue deformation recovery based on stereo endoscopic images is crucial for tool-tissue interaction analysis and benefits surgical navigation and autonomous soft tissue manipulation. Previous research suffers from the problems raised from camera motion, occlusion, large tissue deformation, lack of tissue-specific biomechanical priors, and reliance on offline processing. Unlike previous studies where the tissue geometry and deformation are represented by 3D points and displacements, the proposed method models tissue geometry as the 3D point and derivative map and tissue deformation as the 3D displacement and local deformation map. For a single surface point, 6 parameters are used to describe its rigid motion and 3 parameters for its local deformation. The method is formulated under the camera-centric setting, where all motions are regarded as the scene motion with respect to the camera. Inter-frame alignment is realized by optimizing the inter-frame deformation, making it unnecessary to estimate camera pose. The concept of the canonical map is introduced to optimize tissue geometry and deformation in an online approach. Quantitative and qualitative experiments were conducted using in vivo and ex vivo laparoscopic datasets. With the inputs of depth and optical flow, the method stably models tissue geometry and deformation even when the tissue is partially occluded or moving outside the field of view. Results show that the 3D reconstruction accuracy in the non-occluded and occluded areas reaches 0.37±\pm0.27 mm and 0.39±\pm0.21 mm in terms of surface distance, respectively. The method can also estimate surface strain distribution during various manipulations as an extra modality for mechanical-based analysis.

Jiahe Chen, Naoki Tomii, Ichiro Sakuma +1

School of Engineering, The University of Tokyo, Tokyo, 113-8656, Japan

Endoscopy3D ReconstructionMedical Image AnalysisDynamic Scene ReconstructionDeformable Shape ReconstructionStereo Vision+1
AskPDF
Jun 11, 2025

MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images

We introduce MetricHMSR, a novel framework for recovering metric human meshes and 3D scenes from a single monocular image. Existing methods struggle to recover metric scale due to monocular scale ambiguity and weak-perspective camera assumptions. Moreover, their fully coupled feature representations make it difficult to disentangle local pose from global translation, often requiring multi-stage pipelines that introduce accumulated errors. To address these challenges, we propose MetricHMR (Metric Human Mesh Recovery), which incorporates a bounding camera ray map representation to provide explicit metric cues for human reconstruction,together with a Human Mixture-of-Experts (HumanMoE) that dynamically routes image features to specialized experts, enabling the disentangled perception of local human pose and global metric position. Leveraging the recovered metric human as a geometric anchor, we further refine monocular metric depth estimation to achieve more accurate 3D alignment between humans and scenes.Comprehensive experiments demonstrate that our method achieves state-of-the-art performance on both human mesh recovery and metric human-scene reconstruction. Project Page: https://Metaverse-AI-Lab-THU.github.io/MetricHMSR.

Chentao Song, He Zhang, Haolei Yuan +4

Tsinghua University · Beihang University · Beijing Normal University

Metric Depth Estimation3D ReconstructionDepth EstimationHuman Mesh RecoveryMonocular 3D Reconstruction3D Mesh Reconstruction+1
AskPDF
May 21, 2025

Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data

This paper introduces Synthetic Enclosed Echoes (SEE), a novel dataset designed to enhance robot perception and 3D reconstruction capabilities in underwater environments. SEE comprises high-fidelity synthetic sonar data, complemented by a smaller subset of real-world sonar data. To facilitate flexible data acquisition, a simulated environment has been developed, enabling the generation of additional data through modifications such as the inclusion of new structures or imaging sonar configurations. This hybrid approach leverages the advantages of synthetic data, including readily available ground truth and the ability to generate diverse datasets, while bridging the simulation-to-reality gap with real-world data acquired in a similar environment. The SEE dataset comprehensively evaluates acoustic data-based methods, including mathematics-based sonar approaches and deep learning algorithms. These techniques were employed to validate the dataset, confirming its suitability for underwater 3D reconstruction. Furthermore, this paper proposes a novel modification to a state-of-the-art algorithm, demonstrating improved performance compared to existing methods. The SEE dataset enables the evaluation of acoustic data-based methods in realistic scenarios, thereby improving their feasibility for real-world underwater applications.

Guilherme de Oliveira, Matheus M. dos Santos, Paulo L. J. Drews-Jr

NAUTEC - Intelligent robotics and automation group, FURG - Federal University of Rio Grande, Rio Grande, RS, Brazil · ICMC - Institute of Mathematics and Computer Sciences, USP - University of S˜ao Paulo, S˜ao Carlos, SP, Brazil

Synthetic-to-Real Domain AdaptationUnderwater Imaging3D ReconstructionSynthetic Data GenerationSynthetic Benchmark GenerationUnderwater 3D Reconstruction
AskPDF
May 12, 2025

TUGS: Physics-based Compact Representation of Underwater Scenes by Tensorized Gaussian

Underwater 3D scene reconstruction is crucial for multimedia applications in adverse environments, such as underwater robotic perception and navigation. However, the complexity of interactions between light propagation, water medium, and object surfaces poses significant difficulties for existing methods in accurately simulating their interplay. Additionally, expensive training and rendering costs limit their practical application. Therefore, we propose Tensorized Underwater Gaussian Splatting (TUGS), a compact underwater 3D representation based on physical modeling of complex underwater light fields. TUGS includes a physics-based underwater Adaptive Medium Estimation (AME) module, enabling accurate simulation of both light attenuation and backscatter effects in underwater environments, and introduces Tensorized Densification Strategies (TDS) to efficiently refine the tensorized representation during optimization. TUGS is able to render high-quality underwater images with faster rendering speeds and less memory usage. Extensive experiments on real-world underwater datasets have demonstrated that TUGS can efficiently achieve superior reconstruction quality using a limited number of parameters. The code is available at https://liamlian0727.github.io/TUGS

Shijie Lian, Ziyi Zhang, Hua Li +4

1Huazhong University of Science and Technology · 2Zhongguancun Academy · 3The Chinese University of Hong Kong, Shenzhen +3

3D Gaussian SplattingUnderwater Imaging3D ReconstructionUnderwater 3D ReconstructionGaussian Splatting
AskPDF
Dec 17, 2024

CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image

Recently, generalizable feed-forward methods based on 3D Gaussian Splatting have gained significant attention for their potential to reconstruct 3D scenes using finite resources. These approaches create a 3D radiance field, parameterized by per-pixel 3D Gaussian primitives, from just a few images in a single forward pass. However, unlike multi-view methods that benefit from cross-view correspondences, 3D scene reconstruction with a single-view image remains an underexplored area. In this work, we introduce CATSplat, a novel generalizable transformer-based framework designed to break through the inherent constraints in monocular settings. First, we propose leveraging textual guidance from a visual-language model to complement insufficient information from a single image. By incorporating scene-specific contextual details from text embeddings through cross-attention, we pave the way for context-aware 3D scene reconstruction beyond relying solely on visual cues. Moreover, we advocate utilizing spatial guidance from 3D point features toward comprehensive geometric understanding under single-view settings. With 3D priors, image features can capture rich structural insights for predicting 3D Gaussians without multi-view techniques. Extensive experiments on large-scale datasets demonstrate the state-of-the-art performance of CATSplat in single-view 3D scene reconstruction with high-quality novel view synthesis.

Wonseok Roh, Hwanhee Jung, Jong Wook Kim +6

Korea University · Google · Purdue University

3D Gaussian Splatting3D ViTs3D ReconstructionSingle-View 3D ReconstructionFeed-Forward 3D Reconstruction
AskPDF
Dec 15, 2024

Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals

3D human mesh reconstruction (HMR) from RGB images often degrades under poor illumination, occlusion, and non-line-of-sight conditions. Acoustic sensing provides complementary spatial cues but suffers from low spatial resolution. We propose SonicMesh, which, to the best of our knowledge, is the first acoustic--visual framework for robust 3D human mesh reconstruction. SonicMesh first converts ultrasonic echoes into range--azimuth acoustic images through an Inverse Synthetic Aperture Radar (ISAR)-based imaging process. It then introduces a cross-dimensional anatomical registration module that maps modality-specific 2D joint features into a common canonical 3D human space. The registered anatomical representations are further integrated with acoustic and visual features through a two-stage fusion network for final mesh reconstruction. Experiments demonstrate that SonicMesh achieves accurate and robust 3D human reconstruction across normal, poor-light, occluded, and non-line-of-sight environments, consistently outperforming existing RGB-, radio-frequency (RF)-, and mmWave-based approaches under challenging sensing conditions.

Xiaoxuan Liang, Hong Zhou, Zhaolong Wei +3

University of Massachusetts at Amherst, Location, Country · Trine University · University of Illinois at Chicago +1

3D ReconstructionHuman Mesh Recovery3D Mesh Reconstruction3D Human ReconstructionMultimodal Sensor Fusion
AskPDF
Dec 3, 2024

SparseLGS: Sparse View Language Embedded Gaussian Splatting

Recently, several studies have combined Gaussian Splatting to obtain scene representations with language embeddings for open-vocabulary 3D scene understanding. While these methods perform well, they essentially require very dense multi-view inputs, limiting their applicability in real-world scenarios. In this work, we propose SparseLGS to address the challenge of 3D scene understanding with pose-free and sparse view input images. Our method leverages a learning-based dense stereo model to handle pose-free and sparse inputs, and a three-step region matching approach to address the multi-view semantic inconsistency problem, which is especially important for sparse inputs. Different from directly learning high-dimensional CLIP features, we extract low-dimensional information and build bijections to avoid excessive learning and storage costs. We introduce a reconstruction loss during semantic training to improve Gaussian positions and shapes. To the best of our knowledge, we are the first to address the 3D semantic field problem with sparse pose-free inputs. Experimental results show that SparseLGS achieves comparable quality when reconstructing semantic fields with fewer inputs (3-4 views) compared to previous SOTA methods with dense input. Besides, when using the same sparse input, SparseLGS leads significantly in quality and heavily improves the computation speed (5×\timesspeedup). Project page: https://ustc3dv.github.io/SparseLGS

Jun Hu, Zhang Chen, Zhong Li +2

School of Mathematical Science, University of Science and Technology of China · Meta · Apple Inc. +1

3D Gaussian SplattingLanguage-Embedded 3DGS3D ReconstructionSparse-View 3D Reconstruction3D Scene Understanding
AskPDF
Nov 26, 2024

NumGrad-Pull: Numerical Gradient Guided Tri-plane Representation for Surface Reconstruction from Point Clouds

Reconstructing continuous surfaces from unoriented and unordered 3D points is a fundamental challenge in computer vision and graphics. Recent advancements address this problem by training neural signed distance functions to pull 3D location queries to their closest points on a surface, following the predicted signed distances and the analytical gradients computed by the network. In this paper, we introduce NumGrad-Pull, leveraging the representation capability of tri-plane structures to accelerate the learning of signed distance functions and enhance the fidelity of local details in surface reconstruction. To further improve the training stability of grid-based tri-planes, we propose to exploit numerical gradients, replacing conventional analytical computations. Additionally, we present a progressive plane expansion strategy to facilitate faster signed distance function convergence and design a data sampling strategy to mitigate reconstruction artifacts. These components are synergistically integrated into a unified tri-plane-based pulling framework, in which numerical gradients, progressive expansion, and complementary sampling jointly address the locality and sparsity challenges of learning SDFs from unoriented point clouds. Our extensive experiments across a variety of benchmarks demonstrate the effectiveness and robustness of our approach. Codes are available at: https://github.com/cuiruikai/numgrad-pull.

Ruikai Cui, Binzhu Xie, Shi Qiu +3

School of Computing, The Australian National University, Australia · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Computer Science & Software Engineering, the University of Western Australia, Australia

Implicit Neural Representations3D ReconstructionPoint Cloud LearningPoint Cloud Surface ReconstructionTri-Plane Representations3D Scene Representation
AskPDF
Nov 24, 2024

GSurf: Learning Signed Distance Fields from Splatting Opaque Gaussians for High-quality 3D Reconstruction

High-fidelity surface reconstruction from multi-view images is a core problem in 3D computer vision. While neural implicit surfaces like SDFs offer smooth geometry, they are often bottlenecked by the computational intensity of volume rendering. Conversely, 3D Gaussian Splatting (3DGS) provides rapid training but lacks geometry continuity, often leading to fragmented surfaces. This paper presents a novel framework that integrates Signed Distance Fields directly into the splatting pipeline. By leveraging the continuous nature of SDFs to regularize Gaussian primitives, our method effectively fills geometric holes and suppresses noise inherent in sparse point clouds. Unlike hybrid approaches that rely on heavy volumetric sampling, our approach utilizes the efficiency of splatting to achieve faster convergence. Extensive evaluations demonstrate that our method produces high-quality surfaces with significantly fewer primitives, offering a more compact and efficient representation for both indoor and outdoor environments.

Baixin Xu, Jiangbei Hu, Jiaze Li +1

College of Computing and Data Science, Nanyang Technological University · School of Software, Dalian University of Technology

3D Gaussian SplattingSigned Distance Functions3D ReconstructionMulti-View 3D Reconstruction
AskPDF
Sep 16, 2024

PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing

Detailed and photorealistic 3D human modeling is essential for various applications and has seen tremendous progress. However, full-body reconstruction from a monocular RGB image remains challenging due to the ill-posed nature of the problem and sophisticated clothing topology with self-occlusions. In this paper, we propose PSHuman, a novel framework that explicitly reconstructs human meshes utilizing priors from the multiview diffusion model. It is found that directly applying multiview diffusion on single-view human images leads to severe geometric distortions, especially on generated faces. To address it, we propose a cross-scale diffusion that models the joint probability distribution of global full-body shape and local facial characteristics, enabling detailed and identity-preserved novel-view generation without any geometric distortion. Moreover, to enhance cross-view body shape consistency of varied human poses, we condition the generative model on parametric models like SMPL-X, which provide body priors and prevent unnatural views inconsistent with human anatomy. Leveraging the generated multi-view normal and color images, we present SMPLX-initialized explicit human carving to recover realistic textured human meshes efficiently. Extensive experimental results and quantitative evaluations on CAPE and THuman2.1 datasets demonstrate PSHumans superiority in geometry details, texture fidelity, and generalization capability.

Peng Li, Wangguandong Zheng, Yuan Liu +9

HKUST · Southeast University · Tsinghua University +1

Multi-View Diffusion Models3D Reconstruction3D Mesh Reconstruction3D Human ReconstructionSingle-View 3D ReconstructionDiffusion Models
AskPDF
Aug 30, 2024

2DGH: 2D Gaussian-Hermite Splatting for High-quality Rendering and Better Geometry Features

2D Gaussian Splatting has recently emerged as a significant method in 3D reconstruction, enabling novel view synthesis and geometry reconstruction simultaneously. While the well-known Gaussian kernel is broadly used, its lack of anisotropy and deformation ability leads to dim and vague edges at object silhouettes, limiting the reconstruction quality of current Gaussian splatting methods. To enhance the representation power, we draw inspiration from quantum physics and propose to use the Gaussian-Hermite kernel as the new primitive in Gaussian splatting. The new kernel takes a unified mathematical form and extends the Gaussian function, which serves as the zero-rank special case in the updated general formulation. Our experiments demonstrate that the proposed Gaussian-Hermite kernel achieves improved performance over traditional Gaussian Splatting kernels on both geometry reconstruction and novel-view synthesis tasks. Specifically, on the DTU dataset, our method yields more accurate geometry reconstruction, while on datasets such as MipNeRF360 and our customized Detail dataset, it achieves better results in novel-view synthesis. These results highlight the potential of the Gaussian-Hermite kernel for high-quality 3D reconstruction and rendering.

Ruihan Yu, Tianyu Huang, Jingwang Ling +1

Department of Physics, Tsinghua University · School of software and BNRist, Tsinghua University

3D Gaussian Splatting3D ReconstructionNovel View SynthesisGaussian Splatting
AskPDF
Aug 20, 2024

Multi-view Hand Reconstruction with a Point-Embedded Transformer

This work introduces a novel and generalizable multi-view Hand Mesh Reconstruction (HMR) model, named POEM, designed for practical use in real-world hand motion capture scenarios. The advances of the POEM model consist of two main aspects. First, concerning the modeling of the problem, we propose embedding a static basis point within the multi-view stereo space. A point represents a natural form of 3D information and serves as an ideal medium for fusing features across different views, given its varied projections across these views. Consequently, our method harnesses a simple yet effective idea: a complex 3D hand mesh can be represented by a set of 3D basis points that 1) are embedded in the multi-view stereo, 2) carry features from the multi-view images, and 3) encompass the hand in it. The second advance lies in the training strategy. We utilize a combination of five large-scale multi-view datasets and employ randomization in the number, order, and poses of the cameras. By processing such a vast amount of data and a diverse array of camera configurations, our model demonstrates notable generalizability in the real-world applications. As a result, POEM presents a highly practical, plug-and-play solution that enables user-friendly, cost-effective multi-view motion capture for both left and right hands. The model and source codes are available at https://github.com/JubSteven/POEM-v2.

Lixin Yang, Licheng Zhong, Pengxiang Zhu +4

School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, Shanghai 200230, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China +1

3D ViTsMarkerless Motion Capture3D Hand Pose Estimation3D ReconstructionHuman Mesh RecoveryMulti-View 3D Reconstruction+1
AskPDF