Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans

Eli A. Underwood; Nicolas Rondan; Angel V. Peterchev; J. Matias Di Martino

cs.CVOct 29, 2026

Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans

Authors: Eli A. Underwood, Nicolas Rondan, Angel V. Peterchev, J. Matias Di Martino

Organizations: Duke University · Universidad Cat´olica del Uruguay

Abstract

Aligning a partial, real-world 3D face scan to a canonical representation requires solving two coupled problems at once: estimating the rigid pose that brings the scan into a common frame, and completing the facial shape in regions the scan never observed. Existing pipelines typically decouple these steps, running a generic point-cloud registration front end followed by a separate non-rigid fitting or deformation stage. Building on the GeoTransformer superpoint-matching architecture, we propose a single network that predicts both a rigid pose and a shape-complete point cloud in one forward pass: a cross-attention regressor, conditioned on transformer point features and predicted coarse correspondences, maps a partial scan to per-patch coefficients of a 3D morphable face model learned offline from a corpus of 8,000 identities. The resulting morphed template is then used, in the same forward pass, as the target for fine-grained Sinkhorn matching and rigid pose estimation, supervised purely by geometric alignment to ground-truth morphed geometry, so no hand-labeled coefficients are required. Against six classical and learned non-rigid registration baselines on held-out identities, our method leads the strongest ones by 3-4 times on per-vertex error and baselines that collapse the head onto the visible region by more than an order of magnitude, and is simultaneously the most accurate and fastest method on ground-truth-free scans from two independent real-scan datasets, NoW and NPHM. Code will be made publicly available.

Figures & tables

Figure 1: Mean-face template and partial validation scan (top), with the morphed references produced by three representative non-rigid registration baselines—OAReg [ 26 ] , ClusterReg [ 25 ] , and CPD [ 11 ] —and our method (bottom). The baselines collapse the head onto the visible region, whereas our method recovers a plausible full head from the same partial input.

Figure 2: Pipeline overview. A shared KPConv (Kernel Point Convolution) backbone and GeoTransformer produce coarse superpoint correspondences between the mean-face template and a partial scan; a cross-attention regressor predicts per-patch morphable-basis coefficients $z$ (conditioned on post-transformer features, pre-transformer backbone features, and the coarse correspondences), which are decoded into a morphed template that updates the reference positions used by Sinkhorn optimal-transport (OT) fine matching and local-to-global registration (LGR) to produce the estimated pose $(\hat{R},\hat{t})$ . Bottom: detail of the coefficient regressor, showing its various input feature sources and the per-patch decoding into $z$ .

Component	Value
Backbone	voxel 0.025, 4 stages
Transformer $\sigma_{d}$ (train)	0.16
Coarse targets / correspondences	128 / 256
Coarse / fine matching radius	0.05 / 0.1
Sinkhorn iterations	100
Morphable patches $\times$ components	$32\times 36$

Table 1: Key configuration for the checkpoint used in all reported results. At real-scan (NoW and NPHM) inference only,

\sigma_{d}

is scaled by a fixed

\times 1.2

correction factor.

Method	Self-Occlusion	Face-Cut	Self-Occlusion	Face-Cut	Time (s) $\downarrow$
	GT err (mm) $\downarrow$		CD (mm) $\downarrow$
NRICP [ 1 ]	$31.86\,\pm\,16.56$	$29.62\,\pm\,10.36$	$3.87\,\pm\,9.27$	$1.71\,\pm\,6.63$	64.45
CPD [ 11 ]	$46.65\,\pm\,16.51$	$44.97\,\pm\,10.16$	$1.55\,\pm\,0.19$	$1.47\,\pm\,0.15$	39.95
NSFP [ 10 ]	$9.99\,\pm\,4.27$	$8.88\,\pm\,3.92$	$0.50\,\pm\,0.25$	$0.31\,\pm\,0.20$	15.40
GraphSCNet [ 16 ]	$8.56\,\pm\,3.75$	$7.77\,\pm\,3.63$	$2.06\,\pm\,1.42$	$1.35\,\pm\,0.82$	40.60
ClusterReg [ 25 ]	$42.71\,\pm\,17.03$	$38.22\,\pm\,11.02$	$0.87\,\pm\,0.28$	$0.91\,\pm\,0.18$	41.40

Table 2: Val set (250 identities

\times

2 views = 500 pairs: 266 Self-Occlusion / 234 Face-Cut pairs), GT vertex error and 1-way CD (partial scan

\to

deformed-template, visible region only, same definition as Tab. 4 ), median

\pm

std over pairs, all methods timed on the same hardware. Ours leads every column. Four of six baselines (NRICP, CPD, ClusterReg, OAReg) have GT error an order of magnitude worse than Ours despite ClusterReg posting some of the best CD in the table alongside 38–47mm GT error: CD’s blind spot to implausible completions, visible in single rows ( Sec. 4.3 ).

Method	CD (mm) $\downarrow$	Time (s) $\downarrow$
NRICP [ 1 ]	$1.69\,\pm\,0.18$	31.35
CPD [ 11 ]	$1.50\,\pm\,0.12$	22.50
NSFP [ 10 ]	$0.61\,\pm\,0.05$	11.80
GraphSCNet [ 16 ]	$1.61\,\pm\,0.09$	37.45
ClusterReg [ 25 ]	$1.05\,\pm\,0.06$	23.05
OAReg [ 26 ]	$1.61\,\pm\,0.14$	10.90

Table 3: NoW benchmark test split, no ground truth: 1-way CD (partial scan

\to

deformed-template) and end-to-end wall-clock time, median

\pm

std over the test scans, all methods timed on the same cluster hardware. CD uses each scan’s own recovered scale factor rather than a fixed dataset-wide constant ( Sec. 4.3 ). Ours leads on both CD and time.

Figure 3: Accuracy (1-way CD, partial scan $\to$ deformed-template, median) vs. wall-clock time (log scale, includes rigid-ICP pre-alignment for every baseline that requires it) across each method’s own iteration/step budget, NoW benchmark test split. Baselines: NRICP [ 1 ] , CPD [ 11 ] , NSFP [ 10 ] , GraphSCNet [ 16 ] , ClusterReg [ 25 ] , OAReg [ 26 ] .

Figure 4: Qualitative results on Val (left), NoW (middle), and NPHM (right) examples. Rows: Self-Occlusion / Face-Cut crops (Val) and two representative scans each for NoW and NPHM. Columns: the partial input scan; our morphed reference alone, viewed from the side to show its departure from the fixed template (dark blue); and the same morphed reference (light blue) overlaid with the input scan (orange) to show surface-fit quality.

3D Face Reconstruction

3D face reconstruction estimates facial shape and appearance from images or video to create usable models and avatars.

See all 56 papers

MeshLoom: Feed-Forward Non-Rigid Registration of Mesh Sequences
Jun 15, 2026Jianqi Chen, Jiraphon Yenphraphai, Xiangjun Tang +4Non-Rigid Shape Correspondence 3D Mesh Deformation
KAUST, Saudi Arabia · Snap Inc., United States of America · Purdue University, United States of America
TokenMatch: 3D Mesh Correspondence Transformer with Curvature-Guided Tokenisation
Sep 3, 2026Adeela Islam, Zorah Lähner, Vittorio Murino +13D ViTs Non-Rigid Shape Correspondence
1Italian Institute of Technology · University of Genoa · 6Max Planck Institute for Informatics, SIC +4
Ordered Diffusion for 3D Human Registration
Aug 6, 2026Mattia Masiero, Ilya A. Petrov, Daniel Cremers +2Point Cloud Registration 3D Diffusion Models
University of Tübingen, Germany · Tübingen AI Center, Germany · Technical University of Munich, Germany +2

Jun 15, 2026J/K move · Enter open · S save

MeshLoom: Feed-Forward Non-Rigid Registration of Mesh Sequences

Jianqi Chen, Jiraphon Yenphraphai, Xiangjun Tang, Sergey Tulyakov+3

KAUST, Saudi Arabia · Snap Inc., United States of America · Purdue University, United States of America

Non-Rigid Shape Correspondence 3D Mesh Deformation

Open paper AskPDF arXiv

We present MeshLoom, a feed-forward registration network that directly reconstructs vertex deformations across mesh sequences. Our approach advances non-rigid registration beyond existing models, which are typically constrained by costly per-instance optimization, narrow object categories, pairwise-only inputs, or merely intermediate outputs. The network is simple and efficient, registering multiple meshes within seconds. At its core lies a topology-aware encoder--decoder design. Specifically, we first introduce a topology-aware point representation that encodes the anchor (reference) mesh's topology into its per-vertex features. This representation strengthens the network's understanding of the anchor-mesh geometry and disambiguates points that are Euclidean-close yet geodesically distant. We then propose a multi-modal encoder that fuses this anchor-mesh representation with complementary cues from each frame, such as shape latents and image features. These multi-source signals are compressed into a compact global motion embedding that captures dense inter-frame correspondence. A lightweight decoder then queries this global embedding with the anchor-mesh point representation, retrieving per-vertex deformations at target timestamps. Through extensive experiments across diverse motions and object categories, we show that MeshLoom achieves state-of-the-art results on non-rigid registration. In addition, we find that our global embedding-then-query paradigm naturally enables the network to generate deformations at intermediate timestamps, which extends MeshLoom to motion interpolation and mesh morphing. Project page: https://meshloom.github.io/ .

Abstract

3D Face Reconstruction

Explore similar work

MeshLoom: Feed-Forward Non-Rigid Registration of Mesh Sequences

TokenMatch: 3D Mesh Correspondence Transformer with Curvature-Guided Tokenisation

Ordered Diffusion for 3D Human Registration