Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China
Traditional computed tomography (CT) can capture paired 3D face and skull geometry, but it remains costly, fixed in location, and exposes patients to radiation. For applications such as remote diagnostics, digital healthcare, forensic identification, missing-person investigation, and archaeological analysis, there is an urgent need for a portable, safe, and accessible solution to capture 3D skull geometry. In this work, we introduce Vision Tomography (VT), a concept and framework capable of reconstructing paired 3D face-skull geometry from various visual modalities, and also enabling joint manipulation of 3D facial and skull geometry. To achieve this, we first construct a dataset of over 200 samples, where each case includes a CT-based skull, a CT-based face, and a high-fidelity textured face scan. Secondly, we propose a novel dense ray matching registration method that ensures topological consistency across face, skull, and their tissue correspondences. Building on this registration, we introduce the 3D Bidirectional Face-Skull Morphable Model (BFSM) as the foundation of Vision Tomography, which enables shape inference between the face and skull through a shared coefficient space, while also modeling tissue thickness variation to support one-to-many facial reconstructions from the same skull, reflecting individual facial changes such as age or body fat. Finally, we demonstrate the effectiveness of Vision Tomography across various forensic and medical applications, including paired 3D face-skull reconstruction from images or sketches and predicting surgical outcomes with joint face-skull editing. Extensive experiments confirm the robustness and accuracy of our method. Vision Tomography is available at https://github.com/wang-zidu/BFSM
Figures & tables
Fig. 1: (a) We introduce the 3D Bidirectional Face-Skull Morphable Model (BFSM) as the foundation of Vision Tomography (VT), which enables shape inference between the face and skull through a shared coefficient space, while also modeling tissue thickness variation to support one-to-many facial reconstructions from the same skull. (b) Applications of Vision Tomography, including 3D face-skull reconstruction from visual data and joint editing of face and skull geometry for surgical planning.
Fig. 2: A simple comparison between Computed Tomography (CT) and Vision Tomography (VT).
Fig. 3: (a) and (b): Overview of our face and skull data acquisition process. (c): Clinician-guided and automated alignment of CT and scanning data. (d): Visualization of paired 3D face and skull samples from our dataset.
Fig. 4: Overview of our topology-consistent deformation pipeline. (a) Dense ray matching for face mesh deformation. (b) Dense ray matching for skull mesh deformation.
Fig. 5: Anatomically aligned face-skull pairs generated by randomly sampling (βid,βalb) from FSMM and (βti,βs) from TMM, respectively.
Fig. 6: (a) Given an input face image IX , VT predicts the coefficients of the FSMM to reconstruct an anatomically aligned 3D face-skull pair. (b) Based on the reconstructed face-skull pair, the tissue thickness is estimated and the TMM coefficients are fitted, enabling facial variation by perturbing the TMM coefficients for a given skull.
Fig. 7: Paired 3D face and skull reconstruction from images.
Fig. 8: Paired 3D face and skull reconstruction from different style sketches with controllable facial texture.
Method
Frontal-view Image as Input
Side-view Image as Input
Face
Skull
Face
Skull
RMSE ↓
Recall ↑
Chamfer ↓
RMSE ↓
Recall ↑
Chamfer ↓
RMSE ↓
Recall ↑
Chamfer ↓
RMSE ↓
Recall ↑
Chamfer ↓
SMIRK+FSMM
2.681
0.622
6.591
2.444
0.688
5.445
3.007
0.555
7.099
2.604
0.661
5.871
DECA+FSMM
2.674
0.614
6.755
2.380
0.694
5.308
3.218
0.519
7.616
2.637
0.658
5.915
Deep3D+FSMM
1.908
0.788
3.994
2.044
0.758
4.395
2.311
0.712
4.768
2.179
0.732
4.762
3DDFA+FSMM
2.392
0.675
5.00
2.436
0.687
5.361
2.649
0.631
5.487
2.768
0.631
6.104
TABLE I: Quantitative comparison of 3D face-skull reconstruction from images. The best is highlighted in bold. Distance errors are measured in millimeters, and a recall threshold of 3mm is used.
Method
nid
nexp
RMSE ↓
Recall ↑
Chamfer ↓
FLAME
300
0
0.520
0.910
7.539
300
100
0.497
0.918
7.527
BFM
199
0
0.766
0.726
1.881
199
79 †
0.616
0.814
1.625
Ours
199
0
0.448
0.909
1.275
300
0
0.421
0.923
1.230
TABLE II: Quantitative comparison of the upper bounds of different face model representation capabilities. Distance errors are measured in millimeters, and a recall threshold of 1mm is used.
Nq=1
Nq=3
Nq=5
NO
RMSE ↓
NO
RMSE ↓
NO
RMSE ↓
NF/100
0.600
NF/100
0.599
NF/100
0.598
NF/10
0.596
NF/10
0.594
NF/10
0.592
NF
0.594
NF
0.588
NF
0.590
TABLE III: Ablation study for skull registration. Distance errors are measured in millimeters.
Fig. 9: Our Vision Tomography supports surgical planning prediction for aesthetic facial procedures, providing critical guidance for surgical decision-making and facilitating effective doctor-patient communication.
Fig. 10: Reconstruct paired face and skull from text prompt.
Fig. 11: Our method accurately reconstructs paired face and skull geometry from a single image. During testing, the top-left face scan of each example is rendered as image input to the Vision Tomography framework.
Computed Tomography (CT) is essential for diagnosing pediatric craniofacial abnormalities, yet poses radiation risks to developing anatomies. Reconstructing 3D CT from sparse bi-planar X-rays offers a low-dose alternative but is severely ill-posed. Existing methods employ geometry-agnostic feature lifting, naively projecting 2D features into 3D without explicit spatial modeling, causing depth ambiguity and degraded osseous boundaries. We present PSCT-Net, a geometry-aware framework with differentiable back-projection. Differentiable back-projection establishes a spatially faithful volumetric prior, alleviating depth ambiguity. An Attention-Guided Projection (AGP-3D) module then learns non-linear voxel-wise correspondences between 2D regions and 3D locations. A Bidirectional Mamba (BiM-3D) module captures long-range volumetric dependencies with linear complexity. We further curate a private institutional pediatric skull CT cohort, PedSkull-CT, comprising normal and pathological cases for internal evaluation, addressing the gap in adult-centric, trunk-focused datasets. Project page and code are available at https://dydevelop.github.io/PSCT-Net/.
Dong Yeong Kim, Jaewon Choi, Youmin Shin +5
Interdisciplinary Program in Bioengineering, Seoul National University · Department of Transdisciplinary Medicine, Seoul National University Hospital · Department of Artificial Intelligence, Yonsei University +2
Existing 3D facial-landmark methods localize points on visible skin, but whether CT-defined internal skeletal landmarks can be inferred from external soft-tissue geometry remains unclear. We formulate a coordinate-consistent surface-to-skeleton task using same-acquisition CT-derived surfaces, separating estimation from optical-to-CT registration, scanner-domain, and acquisition-state effects, with coverage analyzed separately. From 240 clinical CT scans from two hospitals, we construct a locked retrospective protocol pairing CT-derived external soft-tissue point clouds with 21 skeletal landmarks and three visible soft-tissue landmarks. An integrated hierarchical point-cloud model achieves 2.97 mm mean radial error on skeletal landmarks and 3.03 mm on deep or surface-invisible landmarks in 40 held-out patients. Patient-mismatch controls support patient-specific signal beyond a fixed population configuration or global similarity alone, while coverage ablations indicate dependence on non-anterior geometry. Optical-transfer diagnostics reveal substantial coverage-related and global-configuration components, although deployable optical inference remains unresolved. These results answer the controlled feasibility question affirmatively and provide a basis for hidden skeletal landmark inference.
Tomoki Abe, Taiki Kanaya, Kazuki Saita +4
Keio University, Yokohama, Japan · Tokyo Dental College, Tokyo, Japan
Reconstructing high-fidelity 3D head geometry from images is critical for a wide range of applications, yet existing methods face fundamental limitations. Traditional photogrammetry achieves exceptional detail but requires extensive camera arrays (25-200+ views), substantial computation, and manual cleanup in challenging areas like facial hair. Recent alternatives present a fundamental trade-off: foundation models enable efficient single-image reconstruction but lack fine geometric detail, while optimization-based methods achieve higher fidelity but require dense views and expensive computation. We bridge this gap with a hybrid approach that combines the strengths of both paradigms. Our method introduces a multi-view surface normal prediction model that extends monocular foundation models with cross-view attention to produce geometrically consistent normals in a feed-forward pass. We then leverage these predictions as strong geometric priors within an inverse rendering optimization framework to recover high-frequency surface details. Our approach outperforms state-of-the-art single-image and multi-view methods, achieving high-fidelity reconstruction on par with dense-view photogrammetry while reducing camera requirements and computational cost.
Noé Artru, Rukhshanda Hussain, Emeline Got +3
Ubisoft La Forge · University of Toronto · ÉTS Montréal +1