Mesh-Agnostic Refinement Models for Latent Space Encoding of 3D Human Body Shapes
Authors: Chaima Zouaoui, Jonathan Cerqueira, Sylvain Arguillere, Martin Bauer, Nicolas Charon, Mohamed Daoudi, Emmanuel Hartman
Organizations: Univ. Lille, CNRS, Centrale Lille, Institut Mines-Télécom, UMR 9189 CRIStAL, F-59000 Lille, France · IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Dihal Systems, F-59000 Lille, France · Florida State University, Department of Mathematics, 208 Love Building, 1017 Academic Way, Tallahassee, 32306-4510, Florida, United States · Université de Lille, Département de Mathématiques, Bâtiment M2, Avenue Carl Gauss, Cité Scientifique, Villeneuve-d’Ascq, 59655, Nord, France · University of Houston, Department of Mathematics, 3551 Cullen Blvd, PGH Room 641, Houston, 77204-3008, Texas, United States
Parametric human body models such as the Skinned Multi-Person Linear (SMPL) model provide compact representations of human shape and pose, but their reconstructions remain affected by both errors in estimated latent parameters and limitations in the decoder's expressive capacity. We propose a framework for refining these reconstructions by exploiting the geometric discrepancy between an initial decoded surface and a target mesh, without requiring vertex correspondences between them. The framework comprises three complementary approaches. Latent-VaRefine uses varifold distance gradients to correct the estimated latent code while preserving the original parametric representation. Vertex-VaRefine processes multiscale varifold-gradient features with a DiffusionNet backbone to predict vertex displacements in a canonical pose, yielding an approximately pose-equivariant refinement pipeline. ElaOpt directly optimizes surface deformations under elastic regularization and normal direction constraints. On AMASS, Vertex-VaRefine reduces the mean vertex-to-vertex error of OmniFit reconstructions from 1.181 to 0.342 cm, a 71.0 percent reduction, while combining Latent-VaRefine with ElaOpt reduces the asymmetric Chamfer distance of ArtEq reconstructions from 1.681 to 0.717 cm, a 57.3 percent reduction. Evaluations with SMPL-H and SMPL-X demonstrate the framework's adaptability to different body models, while additional experiments on remeshed targets and raw scans assess its robustness to changes in discretization and geometry beyond the parametric model. These results highlight the complementary strengths of latent space correction, learned vertex refinement, and elastic optimization, offering different trade-offs between preserving the model parameterization and improving geometric fidelity.
Figures & tables
Figure 1: Overview of our refinement framework. (1) Given a target mesh T of arbitrary topology, (2) an existing encoder provides an initial reconstruction, the source mesh S=Dec(C) , with the fixed topology of the body model. (3) Our three methods refine this reconstruction: Latent-VaRefine corrects the latent code, Vertex-VaRefine predicts a per-vertex correction in the canonical T-pose that is then re-posed by linear blend skinning, and ElaOpt deforms the mesh vertices by elastic optimization.
Angle interval
Invariant proj.
Raw VariGrad
Reduction
(deg)
mean
median
mean
median
(median)
[10,20]
22.99
4.37
32.84
17.99
×4.12
[10,40]
37.29
7.17
53.44
27.89
×3.89
[10,60]
49.62
9.98
71.66
38.50
×3.86
Table 1: Numerical validation of the approximate pose-invariance of the canonical-coordinate Varigrad representation VariGradσ0 (Eq. ( 7 )), on 100 real AMASS source/target pairs. For each pair, a random number of joints (between 1 and 10) are rotated by a random angle within the given interval, identically for both meshes. We report the per-vertex relative error (%) between the VariGrad representation recomputed at this new pose and its value at the reference pose θ=0 , for both the approximately invariant and raw versions of VariGrad. The reduction is the ratio of the raw median error to the median error between the two versions.
Figure 2: Overview of Latent-VaRefine: Given a source mesh of fixed SMPL topology S with coefficients C and a target mesh of arbitrary topology T , we encode the reconstruction error by the low-dimensional VariGrad v with respect to the coefficients C (as defined in the last paragraph of Section 4.2 ). The pair (C,v) is then used to find ΔC via a fully connected MLP. The coefficient correction ΔC is then added to the initial coefficients C to produce a latent code C+ΔC yielding a better mesh reconstruction.
Figure 3: Overview of Vertex-VaRefine. Given a source mesh S=DecL(C) of prescribed topology and a target mesh T of arbitrary topology, we compute multi-scale VariGrad descriptors v∈RV×3×ℓ on S and project them onto the per-vertex rest-pose frames, ϕj=Rj(θ)⊤vj , giving approximately pose-invariant features on the neutral-pose shape Sˉ . A DiffusionNet backbone ( N blocks of learned diffusion, spatial gradient features and a shared pointwise MLP) maps ϕ to a displacement D∈RV×3 in T-pose space. The corrected shape M=Sˉ+D is re-posed with the source pose, S^=LBS(M,θ) .
Figure 4: Schematic of the optimization based approach for vertex based mesh correction. The correction vector field is estimated by minimizing the sum of a similarity and a regularization energy.
Method
#Params (M)
Train Time/Epoch(s)
Inference (ms)
Latent-VaRefine
21.43
8.2
73.1
Vertex-VaRefine
0.46
260
82.4
ElaOpt
n/A
n/A
2006.8
Table 2: Computational cost of the proposed refinement frameworks. We report the number of trainable parameters, the average training time per epoch, and the average inference time per sample including VariGrad calculations.
Method
Metric
AMASS
GRAB
TCD
HumanEva
EKUT
SFU
ArtEq
V2V
3.168
3.163
5.637
2.566
1.938
4.188
CD
1.681
1.708
2.689
1.327
1.087
2.223
ArtEq +
V2V
2.401
1.994
3.092
2.257
2.043
3.031
CD
1.376
1.190
1.655
1.294
1.162
1.748
Latent-VaRefine
V2V
2.511
2.499
3.620
2.305
1.678
3.356
Gain (%)
20.7
21.0
35.8
10.2
13.4
19.9
Table 3: Comparison of reconstruction performance on representative AMASS sub-datasets. ArtEq + denotes ArtEq retrained with the additional BMLmovi training frames. We report the vertex-to-vertex error (V2V) and Asymmetric Chamfer Distance (CD), both expressed in centimeters (cm), together with the relative improvement with respect to ArtEq. The column labeled AMASS does not include the BMLmovi sub-dataset as that was used as the training set for the two VaRefine methods.
Figure 5: Qualitative comparison between ArtEq, latent-VaRefine, Vertex-VaRefine, ElaOpt, Latent-VaRefine + ElaOpt, and Latent-VaRefine + Vertex-VaRefine on different body shapes and poses. Color maps show the per-vertex reconstruction error with respect to the ground-truth surface in centimeters (cm).
Metric
Latent-VaRefine
Vertex-VaRefine
ElaOpt
Hausdorff
43.5
57.1
60.1
V2V supremum
41.3
55.9
18.3
Varifold (oriented)
32.7
59.0
75.4
Varifold (unoriented)
32.7
57.2
75.2
Table 4: Relative improvement (%) over the ArtEq baseline on the SSM_synced subset of AMASS, under metrics independent of the training objective.
Model
Configuration
Metric
AMASS
GRAB
EKUT
SFU
ArtEq
V2V
3.168
3.163
1.938
4.188
CD
1.681
1.708
1.087
2.223
Latent- VaRefine
1 scale
V2V
2.511
2.499
1.678
3.356
CD
1.414
1.407
1.013
1.899
1 scale (MSE loss only)
V2V
2.625
2.577
1.722
3.525
CD
1.470
1.466
1.024
1.965
Table 5: Ablation studies on the number of VariGrad scales for Latent-VaRefine and Vertex-VaRefine, and on the coefficient loss term of Latent-VaRefine. We report vertex-to-vertex (V2V) and asymmetric Chamfer Distance (CD) errors, both in cm, on representative datasets and on the entire AMASS benchmark (excluding BMLmovi). Best results are in bold, second-best are underlined.
Method
Metric
AMASS
GRAB
TCD
HumanEva
EKUT
SFU
ArtEq
V2V
3.168
3.163
5.637
2.566
1.938
4.188
CD
1.681
1.708
2.689
1.327
1.087
2.223
Vertex-VaRefine (w/ equiv.)
V2V
1.552
1.356
2.457
1.423
1.094
2.012
Gain (%)
51.0
57.1
56.4
44.5
43.6
52.0
CD
0.943
0.870
1.300
0.856
0.746
1.211
Gain (%)
43.9
49.1
51.7
35.5
31.4
45.5
Table 6: Ablation of the pose-equivariant construction in Vertex-VaRefine (DiffusionNet backbone), evaluated on the same representative AMASS sub-datasets as Table 3 . Both variants share identical backbone, VariGrad features, training data, and schedule; the only difference is the canonical T-pose projection and re-posing step of Section 6.1 . We report the vertex-to-vertex error (V2V) and Asymmetric Chamfer Distance (CD), in centimeters (cm), together with the relative gain (%) over the ArtEq baseline. The AMASS column is the average over the AMASS sub-datasets, excluding BMLmovi, which was used for training. Removing the equivariant construction degrades both metrics on every sub-dataset.
Subset
Metric
ArtEq
DiffusionNet (Vertex-VaRefine)
SpiralNet
PoissonNet
AMASS
V2V
3.168
1.552
2.019
2.118
CD
1.681
0.943
1.162
1.205
SFU
V2V
4.188
2.012
2.612
2.823
CD
2.223
1.211
1.486
1.592
Table 7: Ablation on the backbone processing the frame-projected VariGrad features ϕ . All three variants share the same descriptors, rest-pose frame projection, T-pose prediction, re-posing, loss and training schedule; only the backbone (and, for SpiralNet++, the decoding head it requires) changes. V2V and asymmetric Chamfer Distance in centimeters.
ArtEq
ElaOpt
Nothing
+Masks
+ H1
+ H2
+SRNF
V2V
4.188
3.330
3.270
3.314
3.265
3.268
3.314
CD
2.223
0.885
0.848
0.861
0.850
0.908
0.851
Table 8: Numerical evaluation on the effect of the ElaOpt terms on V2V and Chamfer. All computations are on the SFU dataset and reported in cm.
SMPL-H
SMPL-X
Method
Metric
AMASS
TCD
SFU
AMASS
TCD
SFU
ArtEq
V2V
3.167
5.637
4.188
2.575
4.282
3.805
CD
1.681
2.689
2.223
1.296
1.908
1.807
Latent- VaRefine
V2V
2.675
4.103
3.600
2.138
2.923
3.072
Gain (%)
15.5
27.2
14.0
17.0
31.7
19.3
CD
1.480
2.035
2.012
1.119
1.361
1.546
Table 9: Extension to SMPL-H and SMPL-X (neutral models). V2V and asymmetric Chamfer Distance (CD) in cm, with relative improvement (Gain) over the initial source reconstruction. The AMASS column does not include BMLmovi, which is used for training.
Method
Metric
AMASS
GRAB
TCD
HumanEva
EKUT
SFU
OmniFit
V2V
1.181
1.130
1.112
1.167
1.592
1.162
CD
0.649
0.641
0.706
0.616
0.727
0.655
Latent-VaRefine
V2V
0.822
0.825
1.092
0.853
0.822
0.868
Gain (%)
30.4
27.0
1.8
26.9
48.4
25.3
CD
0.533
0.529
0.700
0.482
0.520
0.572
Gain (%)
17.9
17.5
0.8
21.8
28.5
12.7
Table 10: Per-dataset evaluation on the AMASS subsets. OmniFit gives the initial reconstruction; we report the error after each refinement method and the relative gain over OmniFit. Lower is better; all values in cm.
Method
Original
Downsampled
Upsampled
Latent-VaRefine
17.8
18.5
17.5
Vertex-VaRefine
54.7
46.3
46.9
ElaOpt
65.7
66.1
65.8
Table 11: Robustness to mesh resolution. Relative improvement (%) in asymmetric Chamfer Distance (CD) over ArtEq on 100 target meshes from the SFU dataset, for the original, downsampled and upsampled targets. The methods are applied as trained on SMPL targets.
Metric
Latent-VaRefine
Vertex-VaRefine
ElaOpt
CD Gain (%)
3.9
29.1
65.2
Table 12: Generalization to non-SMPL targets. Relative improvement (%) in asymmetric Chamfer Distance (CD) over the ArtEq prediction on 10 raw scans from FAUST. The scans were chosen so that ArtEq gives a relatively accurate pose, before evaluating any of the proposed methods.
Figure 6: Qualitative comparison on 10 raw FAUST scans. Color maps show the per-vertex asymmetric Chamfer distance to the ground-truth scan surface, in centimeters (cm).
Figure 7: Comparison of the methods used in the ElaOpt term-wise ablation on four meshes. On the left in gray is the target to be optimized to and on the right, in green, is the base ArtEq mesh each method will be applied to. In red, from left to right are: ElaOpt, plain normal-constrained optimization, normal-constrained optimization with masks, with H1 , with H2 , and with SNRF . Note the numerical instability artifacts in the plain optimization as well as the distorted faces in those methods not using masks.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The above is an SMPL mesh (left) in gray along with the results of naive parameter reuse into SMPL-X (center) and an SMPL-X mesh (right) that is the result of an optimization on the SMPL-X parameters to minimize MSE with the ground truth. As SMPL and SMPL-X have different topologies, this MVE and V2V distance are calculated after first applying SMPL’s supplied matrix to the mesh to convert its mesh topology to that of SMPL. The plotted error is V2V distance in meters under a log of base 10.
Figure 9: Overview of the autoencoder’s architecture. The upper left subnetwork is for β conversion, the center left subnetwork is for hands. Together their outputs form an initial guess, with the pose and orientation retained naively. The pose parameters are then corrected by the larger network in the center of the figure.
Figure 10: The above is an SMPL mesh (left) in gray along with the results of naive parameter reuse in SMPL-X (center) and the SMPL-X mesh (right) that produced by the autoencoder. The result here is nearly as good as the optimized result above in Figure 8 . The plotted error is V2V distance in meters under a log of base 10.
Figure 11: Similar architecture may be trained to allow for conversion to SMPL-H. Note here the naive result is much closer to correct and the converter largely serves to fix the incorrect hands.
Figure 12: In a), we present six examples from the CAPE Dataset. The ground truth clothed meshes are in red, the SMPL approximations are presented in yellow, the refined meshes via Vertex-Varefine are presented in blue, and the refined meshes produced by ElaOpt are presented in green. In b), we present two examples from the THumans dataset. Again, the raw scans are displayed in red, the SMPL approximations are displayed in yellow, and the reconstruction using ElaOpt is displayed in green. In c), we display the transfer of a motions from the CMU dataset to the clothed meshes using the outputs of the ElaOpt method from b).