cs.CVApr 14, 2026
SaveRepresenting 3D Faces with Learnable B-Spline Volumes
Organizations: Google
Abstract
We present CUBE (Control-based Unified B-spline Encoding), a new geometric representation for human faces that combines B-spline volumes with learned features, and demonstrate its use as a decoder for 3D scan registration and monocular 3D face reconstruction. Unlike existing B-spline representations with 3D control points, CUBE is parametrized by a lattice (e.g., 8 x 8 x 8) of high-dimensional control features, increasing the model's expressivity. These features define a continuous, two-stage mapping from a 3D parametric domain to 3D Euclidean space via an intermediate feature space. First, high-dimensional control features are locally blended using the B-spline bases, yielding a high-dimensional feature vector whose first three values define a 3D base mesh. A small MLP then processes this feature vector to predict a residual displacement from the base shape, yielding the final refined 3D coordinates. To reconstruct 3D surfaces in dense semantic correspondence, CUBE is queried at 3D coordinates sampled from a fixed template mesh. Crucially, CUBE retains the local support property of traditional B-spline representations, enabling local surface editing by updating individual control features. We demonstrate the strengths of this representation by training transformer-based encoders to predict CUBE's control features from unstructured point clouds and monocular images, achieving state-of-the-art scan registration results compared to recent baselines.
Figures & tables
Figure 2: Defined by a lattice of high-dimensional control features, CUBE reconstructs a 3D face in a two-stage process. First, 3D coordinates are sampled from a fixed template mesh. These coordinates are then mapped to high-dimensional features via B-spline interpolation using the lattice of control features. The first three values of the resulting high-dimensional feature vector form a coarse base shape. Finally, the full feature vector is input to a small MLP, which predicts coordinate offsets (residuals) from the base shape, resulting in the refined 3D point coordinates.
Figure 3: We register scans to a common mesh topology by directly predicting the control features of CUBE. For this, input scan vertices are tokenized, concatenated with trainable control tokens, and then passed through multiple transformer layers. The resulting control token embeddings are extracted and reshaped to form the feature lattice of CUBE. Querying CUBE with a fixed template mesh points then reconstructs the registered 3D mesh.
Figure 4: Starting from a randomly sampled mesh, we accessorize it with skin appearance, clothing, and hair and render it from 16 cameras using Blender. The synthetic images along with the cameras are then used to reconstruct a scan [ 44 ] to provide paired training data. We show some examples of mesh-scan pairs generated using this approach in the last two rows.
Figure 5: For a test scan shown in the top left, we evaluate CUBE models with encoders of different sizes and a varying number of control points. For each model, we show the predicted mesh and a corresponding error map visualizing the point-to-scan distance. The first columns contain shapes without applying the residuals predicted by the CUBE residual MLP, while the third and fourth columns are of shapes with the residuals applied. The last column visualizes the magnitude of the residuals across the face. The first three rows show models with 16x16x16 control points with encoders of different sizes (small, medium and large), while the last two rows are from a CUBE-Large model with and control points respectively.
| Encoder Size | MLP | PTS (mm) | V2V (mm) | |||||
|---|---|---|---|---|---|---|---|---|
| Mean | Median | Std | Mean | Median | Std | |||
| CUBE-S ( ) | w/o | 2.59 | 1.70 | 2.77 | 3.66 | 3.04 | 2.34 | |
| w/ | 1.77 | 0.99 | 2.32 | 2.19 | 1.92 | 1.24 | ||
| w/o | 2.16 | 1.35 | 2.49 | 2.70 | 2.30 | 1.66 | ||
| w/ | 1.73 | 0.94 | 2.33 | 2.04 | 1.78 | 1.18 | ||
| w/o | 1.74 | 0.95 | 2.31 | 2.09 | 1.81 | 1.24 | ||
Table 1: Impact of model sizes, number of control points , and residual offsets. Ablation study showing the performance of our different model configurations on our test set of synthetic scans. We report point-to-scan (PTS) and vertex-to-vertex (V2V) distances.
| Method | PTS (mm) | V2V (mm) | ||||
|---|---|---|---|---|---|---|
| Mean | Median | Std | Mean | Median | Std | |
| Ground Truth | 0.60 | 0.40 | 0.72 | n/a | n/a | n/a |
| BPS [ 40 ] | ||||||
| 512 points | 4.90 | 4.33 | 3.35 | 14.84 | 14.72 | 3.92 |
| 1024 points | 4.66 | 4.01 | 3.30 | 15.16 | 15.03 | 3.81 |
| 2048 points | 5.03 | 4.21 | 3.69 | 17.78 | 17.79 | 3.87 |
Table 2: Comparison with State-of-the-Art Methods. Quantitative results for BPS [ 40 ] , TEMPEH [ 6 ] , and CUBE across different model sizes and residual configurations. We report the point-to-scan (PTS) and vertex-to-vertex (V2V) distances.
Figure 6: We show qualitative comparisons of BPS [ 40 ] , TEMPEH [ 6 ] and CUBE for predicting registered meshes from three different real scans in various expressions. BPS struggles with scans of varying orientations and collapses to predicting an average face shape that only rigidly matches the location of scan vertices. While TEMPEH-Refined improves on the predictions of its Global stage, it produces noisy reconstructions with geometric artifacts (last example). In contrast, CUBE produces reconstructions with the lowest errors, while matching the identity and expression of the scan closely even for challenging expressions (ex: a person having their tongue out in the first row).
Figure 7: Control features derived from a source and a target scan are linearly interpolated and then evaluated through our CUBE decoder. The shapes resulting from the interpolated control features transition smoothly from the source to the target.
Figure 8: We compute the difference between the control features of a neutral and expression scan of a source subject (first column). This feature difference is then added to control features derived from neutral scans obtained of different target subjects (columns 2-6). In this case, CUBE control arithmetic transfers facial expressions from the source to the target subjects.
Figure 9: The CUBE scan encoder generalizes to scans captured by different setups. Here we show scans from two different datasets CoMA [ 45 ] (columns 1-3) and FaMoS [ 6 ] (columns 4-6). The input scans are shown in the first row, the predictions from our CUBE-L model are shown in the second row, and an overlay of the predictions with the input scan is shown in the third row.
Figure 10: CUBE can also be leveraged for image understanding tasks like in-the-wild face reconstruction. CUBE is versatile and expressive enough to capture a wide range of identities and expressions without relying on an underlying 3DMM.
Figure S1 : CUBE control features can be regressed from images by concatenating learnable control embeddings to image patch tokens, and processing them with a standard vision transformer. With the exception of the patchify layer before the transformer encoder, our architecture for image-based regression is identical to the model we introduced for scan registration in the main paper. For image-based regression, we used a ViT-Large backbone with control tokens.
Figure S2 : Given a high resolution scan, we randomly sample 25K, 50K and 75K points from the scan to result in point clouds of different sizes as seen in the first row. These sampled point clouds are independently processed by our CUBE-L model to result in the predictions seen in the second row. The third row overlays the predicted meshes on the original scan. The last row shows the mesh to scan distance for the predicted meshes. The predicted shapes remain plausible and capture the geometry of the scan in all cases demonstrating that our model is able to gracefully handle point clouds of different sizes.
Figure S3 : Position encoding (PE) the input scan points has a positive effect on the convergence of our CUBE encoder. While the difference in final training error is not significant, PE did seem to accelerate convergence in the earlier iterations of training.
Figure S4 : We show an example where by sampling CUBE at different locations on the template mesh surface, we can dictate the topology of the output shape. When combined with our transformer based encoder for scan registration, this allows CUBE to be used not only with input scans with a varying number of points, but also to represent output meshes in different topologies.
Figure S5 : Left: We show how CUBE can be optimized to match the shape of a target mesh with proxy surfaces for the scalp and facial hair. The optimized control points along with the final mesh and the approximation error are also shown. Right: We show the result of fitting CUBE to match detected dense landmarks on an input image. CUBE can reasonably approximate the facial geometry with a small number of control features in both cases.
Figure S6 : The CUBE scan encoder produces temporally smooth results for input scan sequences and can be used for performance registration. In this figure, we show the input scan, the prediction of our model and an overlay of the prediction on the input scan for a sequence of scans reconstructed from a facial performance of a subject. Note that the scans were processed independently for each frame.
Figure S8 : Additional qualitative results from our CUBE-Large model for scan registration. For each input scan, we visualize the prediction from the model and the point to scan distance (mm). Our method can handle input scans in varying topologies acquired from a diverse collection of subjects in various expressions and accessories.
Figure S9 : Additional qualitative results from our CUBE-Large model for face reconstruction from monocular images. For each input image, we visualize the predicted from the front and the side.
More in this topic
3D Representation Learning
3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.
See all 191 papers