Vision Tomography: Towards Portable 3D Bidirectional Face-Skull Inference from Visual Data

Zidu Wang; Meng Xu; Miao Xu; Hengyuan Ma; Jiankuo Zhao; Xutao Li; Zhen Lei; Xiangyu Zhu

cs.CVOct 27, 2026

Vision Tomography: Towards Portable 3D Bidirectional Face-Skull Inference from Visual Data

Authors: Zidu Wang, Meng Xu, Miao Xu, Hengyuan Ma, Jiankuo Zhao, Xutao Li, Zhen Lei, Xiangyu Zhu

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China

Abstract

Traditional computed tomography (CT) can capture paired 3D face and skull geometry, but it remains costly, fixed in location, and exposes patients to radiation. For applications such as remote diagnostics, digital healthcare, forensic identification, missing-person investigation, and archaeological analysis, there is an urgent need for a portable, safe, and accessible solution to capture 3D skull geometry. In this work, we introduce Vision Tomography (VT), a concept and framework capable of reconstructing paired 3D face-skull geometry from various visual modalities, and also enabling joint manipulation of 3D facial and skull geometry. To achieve this, we first construct a dataset of over 200 samples, where each case includes a CT-based skull, a CT-based face, and a high-fidelity textured face scan. Secondly, we propose a novel dense ray matching registration method that ensures topological consistency across face, skull, and their tissue correspondences. Building on this registration, we introduce the 3D Bidirectional Face-Skull Morphable Model (BFSM) as the foundation of Vision Tomography, which enables shape inference between the face and skull through a shared coefficient space, while also modeling tissue thickness variation to support one-to-many facial reconstructions from the same skull, reflecting individual facial changes such as age or body fat. Finally, we demonstrate the effectiveness of Vision Tomography across various forensic and medical applications, including paired 3D face-skull reconstruction from images or sketches and predicting surgical outcomes with joint face-skull editing. Extensive experiments confirm the robustness and accuracy of our method. Vision Tomography is available at https://github.com/wang-zidu/BFSM

Figures & tables

Fig. 1: (a) We introduce the 3D Bidirectional Face-Skull Morphable Model (BFSM) as the foundation of Vision Tomography (VT), which enables shape inference between the face and skull through a shared coefficient space, while also modeling tissue thickness variation to support one-to-many facial reconstructions from the same skull. (b) Applications of Vision Tomography, including 3D face-skull reconstruction from visual data and joint editing of face and skull geometry for surgical planning.

Fig. 2: A simple comparison between Computed Tomography (CT) and Vision Tomography (VT).

Fig. 3: (a) and (b): Overview of our face and skull data acquisition process. (c): Clinician-guided and automated alignment of CT and scanning data. (d): Visualization of paired 3D face and skull samples from our dataset.

Fig. 4: Overview of our topology-consistent deformation pipeline. (a) Dense ray matching for face mesh deformation. (b) Dense ray matching for skull mesh deformation.

Fig. 5: Anatomically aligned face-skull pairs generated by randomly sampling $(\beta_{{id}},\beta_{{alb}})$ from FSMM and $(\beta_{{ti}},\beta_{s})$ from TMM, respectively.

Fig. 6: (a) Given an input face image $I_{X}$ , VT predicts the coefficients of the FSMM to reconstruct an anatomically aligned 3D face-skull pair. (b) Based on the reconstructed face-skull pair, the tissue thickness is estimated and the TMM coefficients are fitted, enabling facial variation by perturbing the TMM coefficients for a given skull.

Fig. 7: Paired 3D face and skull reconstruction from images.

Fig. 8: Paired 3D face and skull reconstruction from different style sketches with controllable facial texture.

Method	Frontal-view Image as Input						Side-view Image as Input
	Face			Skull			Face			Skull
	RMSE $\downarrow$	Recall $\uparrow$	Chamfer $\downarrow$	RMSE $\downarrow$	Recall $\uparrow$	Chamfer $\downarrow$	RMSE $\downarrow$	Recall $\uparrow$	Chamfer $\downarrow$	RMSE $\downarrow$	Recall $\uparrow$	Chamfer $\downarrow$
SMIRK+FSMM	2.681	0.622	6.591	2.444	0.688	5.445	3.007	0.555	7.099	2.604	0.661	5.871
DECA+FSMM	2.674	0.614	6.755	2.380	0.694	5.308	3.218	0.519	7.616	2.637	0.658	5.915
Deep3D+FSMM	1.908	0.788	3.994	2.044	0.758	4.395	2.311	0.712	4.768	2.179	0.732	4.762
3DDFA+FSMM	2.392	0.675	5.00	2.436	0.687	5.361	2.649	0.631	5.487	2.768	0.631	6.104

TABLE I: Quantitative comparison of 3D face-skull reconstruction from images. The best is highlighted in bold. Distance errors are measured in millimeters, and a recall threshold of

3mm

is used.

Method	$n_{id}$	$n_{exp}$	RMSE $\downarrow$	Recall $\uparrow$	Chamfer $\downarrow$
FLAME	300	0	0.520	0.910	7.539
FLAME	300	100	0.497	0.918	7.527
BFM	199	0	0.766	0.726	1.881
BFM	199	79 †	0.616	0.814	1.625
Ours	199	0	0.448	0.909	1.275
	300	0	0.421	0.923	1.230

TABLE II: Quantitative comparison of the upper bounds of different face model representation capabilities. Distance errors are measured in millimeters, and a recall threshold of

1mm

is used.

$N_{q}=1$		$N_{q}=3$		$N_{q}=5$
$N_{O}$	RMSE $\downarrow$	$N_{O}$	RMSE $\downarrow$	$N_{O}$	RMSE $\downarrow$
$N_{\mathrm{F}}/100$	0.600	$N_{\mathrm{F}}/100$	0.599	$N_{\mathrm{F}}/100$	0.598
$N_{\mathrm{F}}/10$	0.596	$N_{\mathrm{F}}/10$	0.594	$N_{\mathrm{F}}/10$	0.592
$N_{\mathrm{F}}$	0.594	$N_{\mathrm{F}}$	0.588	$N_{\mathrm{F}}$	0.590

TABLE III: Ablation study for skull registration. Distance errors are measured in millimeters.

Fig. 9: Our Vision Tomography supports surgical planning prediction for aesthetic facial procedures, providing critical guidance for surgical decision-making and facilitating effective doctor-patient communication.

Fig. 10: Reconstruct paired face and skull from text prompt.

Fig. 11: Our method accurately reconstructs paired face and skull geometry from a single image. During testing, the top-left face scan of each example is rendered as image input to the Vision Tomography framework.

3D Morphable Models

3D morphable models represent faces, bodies, and objects with compact, editable shape and appearance parameters.

See all 16 papers

PSCT-Net: Geometry-Aware Pediatric Skull CT Reconstruction via Differentiable Back-Projection and Attention-Guided Refinement
Jun 18, 2026Dong Yeong Kim, Jaewon Choi, Youmin Shin +53D Medical Imaging Image Reconstruction
Interdisciplinary Program in Bioengineering, Seoul National University · Department of Transdisciplinary Medicine, Seoul National University Hospital · Department of Artificial Intelligence, Yonsei University +2
Surface-to-Skeleton 3D Cephalometry: Estimating Hidden Skeletal Landmarks from CT-Derived External Soft-Tissue Surfaces
Aug 12, 2026Tomoki Abe, Taiki Kanaya, Kazuki Saita +43D Medical Imaging Medical Image Localization
Keio University, Yokohama, Japan · Tokyo Dental College, Tokyo, Japan
Skullptor: High Fidelity 3D Head Reconstruction in Seconds with Multi-View Normal Prediction
Feb 24, 2026Noé Artru, Rukhshanda Hussain, Emeline Got +33D Face Reconstruction Sparse-View 3D Reconstruction
Ubisoft La Forge · University of Toronto · ÉTS Montréal +1

Jun 18, 2026J/K move · Enter open · S save

PSCT-Net: Geometry-Aware Pediatric Skull CT Reconstruction via Differentiable Back-Projection and Attention-Guided Refinement

Dong Yeong Kim, Jaewon Choi, Youmin Shin, JunGyu Lee+4

Interdisciplinary Program in Bioengineering, Seoul National University · Department of Transdisciplinary Medicine, Seoul National University Hospital · Department of Artificial Intelligence, Yonsei University+2

3D Medical Imaging Image Reconstruction Computed Tomography

Open paper AskPDF arXiv

Computed Tomography (CT) is essential for diagnosing pediatric craniofacial abnormalities, yet poses radiation risks to developing anatomies. Reconstructing 3D CT from sparse bi-planar X-rays offers a low-dose alternative but is severely ill-posed. Existing methods employ geometry-agnostic feature lifting, naively projecting 2D features into 3D without explicit spatial modeling, causing depth ambiguity and degraded osseous boundaries. We present PSCT-Net, a geometry-aware framework with differentiable back-projection. Differentiable back-projection establishes a spatially faithful volumetric prior, alleviating depth ambiguity. An Attention-Guided Projection (AGP-3D) module then learns non-linear voxel-wise correspondences between 2D regions and 3D locations. A Bidirectional Mamba (BiM-3D) module captures long-range volumetric dependencies with linear complexity. We further curate a private institutional pediatric skull CT cohort, PedSkull-CT, comprising normal and pathological cases for internal evaluation, addressing the gap in adult-centric, trunk-focused datasets. Project page and code are available at https://dydevelop.github.io/PSCT-Net/.

Abstract

3D Morphable Models

Explore similar work

PSCT-Net: Geometry-Aware Pediatric Skull CT Reconstruction via Differentiable Back-Projection and Attention-Guided Refinement

Surface-to-Skeleton 3D Cephalometry: Estimating Hidden Skeletal Landmarks from CT-Derived External Soft-Tissue Surfaces

Skullptor: High Fidelity 3D Head Reconstruction in Seconds with Multi-View Normal Prediction