We propose a remeshing-enhanced method for generatively deforming shapes with visual losses. It is intuitive that sufficiently drastic deformations of a mesh without changing its triangulation can easily compromise element quality, even if such large geometry changes may be semantically desired. Shape deformation methods could thus benefit from changing the triangulation; however, this is not done by most generative, text-based, visually-supervised mesh deformation methods. Remeshing is a discrete operation, proven to be especially challenging to couple with the notoriously noisy supervision signal provided by visual losses. We propose a vertex-based deformation optimization quantity capable of large deformations and robustness to such noise; we periodically remesh using an isotropic remesher that interpolates and carries forward the deformation optimization state. This enables continuous, geometry-informed progress in coarse-to-fine addition of resolution. The resulting shapes' triangulations fit their optimized geometry and have neat isotropic elements. Further, our method is localizable, able to grow new features on a base shape with expressive detail, leaving the rest unchanged. We showcase the effectiveness of our method on a variety of shapes and prompts, both local and global deformations, and demonstrate its superior visual quality and triangle efficiency. Our project page is at https://threedle.github.io/radmesh.
Figures & tables
Figure 2: Gallery of local growths and global detailing deformations . Top row: our method is capable of growing large parts out of small selected regions on diverse shapes, organic and manufactured. The results show symmetry, high detail even at extremities, and clean geometry at selection boundaries. Bottom row: we can also perform global “detailization” deformations, where the whole shape is allowed to change. This is useful on simple base shapes and demonstrates the ability to change both high-level structure and finer details.
Figure 3: Overview of our optimization loop. Geometry is optimized via gradient descent of our deformation quantity Q , with associated optimizer state, with respect to a base mesh. At epoch i , we solve for the deformation using Qi ( Sec. 3.2 ), render, take visual loss, and update the quantity. Once every N epochs, we remesh the last deformed shape, interpolate the optimizer state onto the result, and use it as the new source mesh until the next scheduled remesh. (Note that the deformation in this figure is exaggerated for illustrative purposes; in practice, the remeshes are frequent enough that the deformation between each remesh is more subtle.)
Figure 4: Evolution. We visualize the geometric evolution through the course of two example optimizations, one localized and the other a global deformation. The local growth example involves disjoint selected components (the wings, the horn), which all grow simultaneously. The global deformation shows an ability to form complex geometric detail (starting from a shape as basic as a sphere) and appendages while maintaining clean isotropic triangulation throughout.
Figure 5: Global detailization . Our method is able to take basic shapes and globally deform & remesh them expressively towards prompts. The results keep high-level structure ( e.g . four similar legs) while exhibiting expressive geometric detail: the folds on the flower-themed flower-shaped backrest, the tapered lobes of the chair made of balloons , the wheelchair ’s wheels.
Figure 6: More local growth examples. (a) Growing garments: Our method is able to generate thin parts and garments from small selection regions on human shapes, extending considerably outwards from the original extents with high detail at the generated extremities. (b) Same selection, different growths: On the same selection and mesh, our method can deform & remesh to produce varied growths, such as the person riding (including a detailed hat), the cargo with geometric texture on the surface, and the robotic saddle with sharp features.
Figure 7: Generated triangulation tailored for the generated geometry. Taking global deformation of a sphere as an example, we show that each final triangulation, applied to the original geometry, encodes the structures and appendages requiring high vertex density in the result shape. This shows the appropriateness and adaptivity of the generated triangulation for the generated geometry, not merely a uniform upsample of the original triangulation.
Method
CLIP ↑
VQA ↑
FQ ↑
#F
MagicClay [ 8 ]
29.821
57.150
86.784
17203.55
Instant3dit [ 7 ]
28.399
52.111
80.132
40034.69
Geometry in Style [ 16 ]
29.279
52.120
94.146
11308.06*
MeshUp [ 27 ]
30.725
66.093
84.831
11308.06*
MeshUp + our remesh
31.148
69.360
96.310
29461.34
RADmesh (ours)
31.416
72.930
96.516
16963.75
Table 1: Quantitative comparison : CLIP and VQA scores ( ×100 , higher is better), average face quality (FQ) ( ×100 , higher is better), average face counts (#F). We evaluate on n=64 shape-prompt pairs (see supplementary material for details). Our method achieves superior CLIP and VQA score, suggesting better perceptual quality and adherence to prompts, as well as superior triangle quality while being efficient with face budget among the methods that remesh. (*Geometry in Style and MeshUp do not remesh; their average face counts thus reflect the resolutions of the source meshes.)
Figure 9: Ablations. (a) No scale parameter: Without the optimizable scale parameter, growths fail to attain large extents and have blunted, less sharp detail at the extremities. (b) No optimizer state interpolation: Without optimizer state interpolation, the optimization loses all gradient history upon each remesh. This leads to less stable deformations, asymmetries, exaggerated view-dependent artifacts, and, in some cases, excessive resolution or scale-up without better geometry.
Figure 10: Ablations on remeshing. (a) No coarse-to-fine remeshing: Simply finely remeshing the region only once at the start , even to higher resolutions than the final resolution of our coarse-to-fine schedule, fails to produce quality growths. This shows the significance of a coarse-to-fine remeshing schedule. (b) Adaptive remeshing: Using adaptive remeshing (giving more resolution to high-curvature areas) rather than uniform isotropic remeshing also breaks the coarse-to-fine property, compromising the ability to grow large geometries.
Figure 11: Iterative editing. We progressively add parts to the torso mesh by iterative selections (including on top of previously grown parts) and prompts.
Figure 12: Workflow on textured meshes. Despite changing connectivity, our method interpolates vertex attributes inside, and fully preserves correspondence, geometry, and data of mesh elements outside the region selected for deforming and remeshing. This includes e.g . corner UVs. We demonstrate the workflow of deforming and remeshing a region on a mesh with existing UVs and texture: the rest of the mesh is unchanged, and the user can choose to unwrap and texture the newly grown geometry.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12Figure 13
Figure 13: Using inverse rendering methods with generative supervision. We add our CSD-based generative semantic loss (replacing L1 supervision against ground truth target images) to the deformation-and-remeshing pipelines of the two methods Continuous Remeshing for Inverse Rendering [ 46 ] and Large Steps in Inverse Rendering of Geometry [ 44 ] . Both methods, even granted some bespoke hyperparameter tuning to accommodate the new task, fail to achieve the large-scale geometric changes required to form not just high-level structure but also clear new parts e.g . the octopus tentacles, the snowman’s hat. Our method achieves these structures while maintaining smoothness, unlike the noisy surface elements of Continuous Remeshing . Our method is also triangle-efficient, unlike the resolution induced by the remeshing requirements of Large Steps.
Meshes remain the most practical representation for geometry reasoning and integration into graphics pipelines, yet existing reconstruction methods struggle to produce high-quality meshes. Most state-of-the-art approaches initially learn an intermediate representation (NeRF/3DGS) and treat mesh extraction as a post-processing step, which often leads to oversmoothed surfaces or poor quality meshes with excessive triangle counts.Existing mesh-native optimization methods alleviate some of these issues but suffer from fixed-resolution discretizations and unstable optimization behavior. In this paper, we introduce an adaptive mesh-based optimization framework and a practical mesh rendering technique to address these challenges. Our representation combines an optimizable Delaunay-triangulated tetrahedral grid with a multi-resolution hash grid. The former is refined through point pruning and insertion, while the latter provides latent features for SDF/appearance value predictions. We use volumetric rendering to bootstrap a coarse geometry while leveraging mesh-based rendering for recovering fine-grained details. Additionally, we propose a differentiable, rasterization-based depth-offset rendering formulation, reducing geometric artifacts and improving reconstruction quality. Our method significantly outperforms existing mesh optimization approaches across a variety of object-centric benchmarks while being competitive with state-of-the-art NeRF/3DGS methods.
Johannes Weidenfeller, Shaofei Wang, Philipp Fürnstahl +1
ETH Zurich, Zurich, Switzerland · Beijing Institute for General Artificial Intelligence, Beijing, China · University of Zurich, Zurich, Switzerland +1
We consider the problem of regenerating 3D objects from 2D images and initial 3D shapes. Most 3D generators operate in a one-shot fashion, converting text or images to a 3D object with limited controllability. We introduce instead MeshReGen, a 3D regenerator that is conditioned on an initial 3D shape. This conceptually simple formulation allows us to support numerous useful tasks, including 3D enhancement, reconstruction, and editing. MeshReGen uses a new conditioning mechanism based on VecSet, which allows the regenerator to update or improve the input geometry with consistent fine-grained details. MeshReGen learns a widely applicable regeneration prior from off-the-shelf 3D datasets via self-supervised pretext tasks and augmentations, without additional annotations. We evaluate both the geometric consistency and fine-grained quality of MeshReGen, achieving state-of-the-art performance in controllable 3D generation across several tasks.
Geon Yeong Park, Roman Shapovalov, Rakesh Ranjan +3
KAIST · ⋆Work done during an internship at Meta Reality Labs. · Meta Reality Labs
This work proposes an adaptation of the attention mechanism for triangle meshes. The core observation is that endowing the attention mechanism with critical properties for learning over meshes -- intrinsicality and triangulation-agnosticism -- enables it to attain state-of-the-art results over several learning-based tasks in geometry-processing. The above is achieved by modifying the attention mechanism from the bottom up based on simple principles from geometry-processing. Namely, the quantities used within attention -- queries, keys and values -- are created by an intrinsic, triangulation-agnostic network, and treated as discretizations of continuous functions. From that, we devise an appropriate attention mechanism that operates over triangle meshes through standard FEM discretization of the resulting integrals of the above functions. Surprisingly, as far as we know, this straightforward approach has not been utilized for learning over meshes. Experiments show our method exceeds current state of the art, including both mesh-based architectures as well as point cloud transformers. Namely, we show significant improvements on several common benchmarks and tasks -- predicting canonical high-frequency signals; predicting deformations; computing dense correspondences, both between full shapes and partial ones; and predicting feature descriptors.
Ashwath Shetty, Zihan Zhu, Soeren Pirk +1
Université de Montréal & Mila, Canada · Kiel University, Germany