We present CUBE (Control-based Unified B-spline Encoding), a new geometric representation for human faces that combines B-spline volumes with learned features, and demonstrate its use as a decoder for 3D scan registration and monocular 3D face reconstruction. Unlike existing B-spline representations with 3D control points, CUBE is parametrized by a lattice (e.g., 8 x 8 x 8) of high-dimensional control features, increasing the model's expressivity. These features define a continuous, two-stage mapping from a 3D parametric domain to 3D Euclidean space via an intermediate feature space. First, high-dimensional control features are locally blended using the B-spline bases, yielding a high-dimensional feature vector whose first three values define a 3D base mesh. A small MLP then processes this feature vector to predict a residual displacement from the base shape, yielding the final refined 3D coordinates. To reconstruct 3D surfaces in dense semantic correspondence, CUBE is queried at 3D coordinates sampled from a fixed template mesh. Crucially, CUBE retains the local support property of traditional B-spline representations, enabling local surface editing by updating individual control features. We demonstrate the strengths of this representation by training transformer-based encoders to predict CUBE's control features from unstructured point clouds and monocular images, achieving state-of-the-art scan registration results compared to recent baselines.
Figures & tables
Figure 2: Defined by a lattice of high-dimensional control features, CUBE reconstructs a 3D face in a two-stage process. First, 3D coordinates are sampled from a fixed template mesh. These coordinates are then mapped to high-dimensional features via B-spline interpolation using the lattice of control features. The first three values of the resulting high-dimensional feature vector form a coarse base shape. Finally, the full feature vector is input to a small MLP, which predicts coordinate offsets (residuals) from the base shape, resulting in the refined 3D point coordinates.
Figure 3: We register scans to a common mesh topology by directly predicting the control features of CUBE. For this, input scan vertices are tokenized, concatenated with trainable control tokens, and then passed through multiple transformer layers. The resulting control token embeddings are extracted and reshaped to form the feature lattice of CUBE. Querying CUBE with a fixed template mesh points then reconstructs the registered 3D mesh.
Figure 4: Starting from a randomly sampled mesh, we accessorize it with skin appearance, clothing, and hair and render it from 16 cameras using Blender. The synthetic images along with the cameras are then used to reconstruct a scan [ 44 ] to provide paired training data. We show some examples of mesh-scan pairs generated using this approach in the last two rows.
Figure 5: For a test scan shown in the top left, we evaluate CUBE models with encoders of different sizes and a varying number of control points. For each model, we show the predicted mesh and a corresponding error map visualizing the point-to-scan distance. The first columns contain shapes without applying the residuals predicted by the CUBE residual MLP, while the third and fourth columns are of shapes with the residuals applied. The last column visualizes the magnitude of the residuals across the face. The first three rows show models with 16x16x16 control points with encoders of different sizes (small, medium and large), while the last two rows are from a CUBE-Large model with 8×8×8 and 4×4×4 control points respectively.
Encoder Size
mc
MLP
PTS (mm)
V2V (mm)
Mean
Median
Std
Mean
Median
Std
CUBE-S ( d=384 )
43
w/o
2.59
1.70
2.77
3.66
3.04
2.34
w/
1.77
0.99
2.32
2.19
1.92
1.24
83
w/o
2.16
1.35
2.49
2.70
2.30
1.66
w/
1.73
0.94
2.33
2.04
1.78
1.18
163
w/o
1.74
0.95
2.31
2.09
1.81
1.24
Table 1: Impact of model sizes, number of control points mc , and residual offsets. Ablation study showing the performance of our different model configurations on our test set of synthetic scans. We report point-to-scan (PTS) and vertex-to-vertex (V2V) distances.
Method
PTS (mm)
V2V (mm)
Mean
Median
Std
Mean
Median
Std
Ground Truth
0.60
0.40
0.72
n/a
n/a
n/a
BPS [ 40 ]
512 points
4.90
4.33
3.35
14.84
14.72
3.92
1024 points
4.66
4.01
3.30
15.16
15.03
3.81
2048 points
5.03
4.21
3.69
17.78
17.79
3.87
Table 2: Comparison with State-of-the-Art Methods. Quantitative results for BPS [ 40 ] , TEMPEH [ 6 ] , and CUBE across different model sizes and residual configurations. We report the point-to-scan (PTS) and vertex-to-vertex (V2V) distances.
Figure 6: We show qualitative comparisons of BPS [ 40 ] , TEMPEH [ 6 ] and CUBE for predicting registered meshes from three different real scans in various expressions. BPS struggles with scans of varying orientations and collapses to predicting an average face shape that only rigidly matches the location of scan vertices. While TEMPEH-Refined improves on the predictions of its Global stage, it produces noisy reconstructions with geometric artifacts (last example). In contrast, CUBE produces reconstructions with the lowest errors, while matching the identity and expression of the scan closely even for challenging expressions (ex: a person having their tongue out in the first row).
Figure 7: Control features derived from a source and a target scan are linearly interpolated and then evaluated through our CUBE decoder. The shapes resulting from the interpolated control features transition smoothly from the source to the target.
Figure 8: We compute the difference between the control features of a neutral and expression scan of a source subject (first column). This feature difference is then added to control features derived from neutral scans obtained of different target subjects (columns 2-6). In this case, CUBE control arithmetic transfers facial expressions from the source to the target subjects.
Figure 9: The CUBE scan encoder generalizes to scans captured by different setups. Here we show scans from two different datasets CoMA [ 45 ] (columns 1-3) and FaMoS [ 6 ] (columns 4-6). The input scans are shown in the first row, the predictions from our CUBE-L model are shown in the second row, and an overlay of the predictions with the input scan is shown in the third row.
Figure 10: CUBE can also be leveraged for image understanding tasks like in-the-wild face reconstruction. CUBE is versatile and expressive enough to capture a wide range of identities and expressions without relying on an underlying 3DMM.
Figure S1 : CUBE control features can be regressed from images by concatenating mc learnable control embeddings to image patch tokens, and processing them with a standard vision transformer. With the exception of the patchify layer before the transformer encoder, our architecture for image-based regression is identical to the model we introduced for scan registration in the main paper. For image-based regression, we used a ViT-Large backbone with 8×8×8 control tokens.
Figure S2 : Given a high resolution scan, we randomly sample 25K, 50K and 75K points from the scan to result in point clouds of different sizes as seen in the first row. These sampled point clouds are independently processed by our CUBE-L model to result in the predictions seen in the second row. The third row overlays the predicted meshes on the original scan. The last row shows the mesh to scan distance for the predicted meshes. The predicted shapes remain plausible and capture the geometry of the scan in all cases demonstrating that our model is able to gracefully handle point clouds of different sizes.
Figure S3 : Position encoding (PE) the input scan points has a positive effect on the convergence of our CUBE encoder. While the difference in final training error is not significant, PE did seem to accelerate convergence in the earlier iterations of training.
Figure S4 : We show an example where by sampling CUBE at different locations on the template mesh surface, we can dictate the topology of the output shape. When combined with our transformer based encoder for scan registration, this allows CUBE to be used not only with input scans with a varying number of points, but also to represent output meshes in different topologies.
Figure S5 : Left: We show how CUBE can be optimized to match the shape of a target mesh with proxy surfaces for the scalp and facial hair. The optimized control points along with the final mesh and the approximation error are also shown. Right: We show the result of fitting CUBE to match detected dense landmarks on an input image. CUBE can reasonably approximate the facial geometry with a small number of 4×4×4 control features in both cases.
Figure S6 : The CUBE scan encoder produces temporally smooth results for input scan sequences and can be used for performance registration. In this figure, we show the input scan, the prediction of our model and an overlay of the prediction on the input scan for a sequence of scans reconstructed from a facial performance of a subject. Note that the scans were processed independently for each frame.
Figure S8 : Additional qualitative results from our CUBE-Large 16×16×16 model for scan registration. For each input scan, we visualize the prediction from the model and the point to scan distance (mm). Our method can handle input scans in varying topologies acquired from a diverse collection of subjects in various expressions and accessories.
Figure S9 : Additional qualitative results from our CUBE-Large 8×8×8 model for face reconstruction from monocular images. For each input image, we visualize the predicted from the front and the side.
Recent frameworks like ToFu and TEMPEH provide an automated alternative to classical registration pipelines by predicting 3D meshes in dense semantic correspondence directly from calibrated multi-view images. However, these learning-based methods rely on the slow, manual registration pipelines they aim to replace for their training supervision. We overcome this limitation with MOCHI (Multi-view Optimizable Correspondence of Heads from Images), a multi-view 3D face prediction framework trained without requiring registered training data. MOCHI eliminates the registration data dependency by enforcing topological consistency through a pseudo-linear inverse kinematic solver. Semantic alignment is guided by dense keypoints from a 2D landmark predictor trained exclusively on synthetic data. Our analysis further reveals that standard point-to-surface distances induce training instabilities and visual artifacts in registration-free settings. We propose pointmap- and normal-based losses instead, which provide smoother gradients and superior reconstruction fidelity. Finally, we introduce a test-time optimization scheme that refines network weights over a few dozen iterations. This approach bridges the gap between feed-forward efficiency and iterative optimization precision, allowing MOCHI to outperform traditional labor-intensive pipelines in both reconstruction accuracy and visual quality. Code and model are public at: https://filby89.github.io/mochi.
Panagiotis P. Filntisis, George Retsinas, Radek Daněček +3
Institute of Robotics, Athena Research Center, 15125 Maroussi, Greece · HERON – Hellenic Robotics Center of Excellence, Athens, Greece · School of ECE, NTUA, Greece +3
Nowadays as convolution neural networks demonstrate its powerful problem-solving ability in the area of image processing, efforts have been made to reconstruct detailed face shapes from 2D face images or videos. However, to make the full use of CNN, a large number of labeled data is required to train the network. Coarse morphable face model has been used to synthesize labeled data. However, it is hard for coarse morphable face models to generate photo-realistic data with detail such as wrinkles. In this project, we present a pipeline that reconstructs a human face 3D model from a single RGB image. The pipeline includes face detection, landmark detection, regression of 3DMM model parameters, and soft rendering. Mentor: Zhipeng Fan (Email: [email protected]) Code Repository: https://github.com/SeVEnMY/3d-face- reconstruction Code Reference: https://github.com/sicxu/Deep3DFaceRecon pytorch
Recent monocular 3D face reconstruction methods achieve high fidelity by fitting a 3D Morphable Model (3DMM) to dense priors predicted by networks, but the optimization stage is computationally expensive, often taking tens of seconds per image. We present RealDenseFace, a real-time optimization-based 3D face reconstruction method with dense UV-space network predictions. Our key idea is to formulate 3DMM fitting as a nonlinear least-squares problem and solve it with a tailored Gauss-Newton solver that converges in only a few iterations. The reconstruction is conducted in two stages. In the first stage, the network predicts two dense UV-space maps from a single RGB image: a correspondence map for UV-to-image alignment, and a relative-depth map for geometric constraints along the viewing direction. In the second stage, the solver fits per-vertex targets sampled from these maps at the vertex UV coordinates. The solver supports all three reconstruction settings: single-image fitting, offline sequence reconstruction, and online tracking. Our method achieves state-of-the-art accuracy on the NeRSemble SVFR benchmark. The online tracker runs at 80+ FPS, and the offline sequence reconstruction is over 20 times faster than previous optimization-based baselines.