Semantic Segmentation of Textured Non-manifold 3D Meshes using Transformers

Mohammadreza Heidarianbaei; Max Mehltretter; Franz Rottensteiner

cs.CVApr 2, 2026

Semantic Segmentation of Textured Non-manifold 3D Meshes using Transformers

Authors: Mohammadreza Heidarianbaei, Max Mehltretter, Franz Rottensteiner

Organizations: Institute of Photogrammetry and GeoInformation, Leibniz University Hannover, Germany

Abstract

Textured 3D meshes jointly represent geometry, topology, and appearance, yet their irregular structure poses significant challenges for deep-learning-based semantic segmentation. While a few recent methods operate directly on meshes without imposing geometric constraints, they typically overlook the rich textural information also provided by such meshes. We introduce a texture-aware transformer that learns directly from raw pixels associated with each mesh face, coupled with a new hierarchical learning scheme for multi-scale feature aggregation. A texture branch summarizes all face-level pixels into a learnable token, which is fused with geometrical descriptors and processed by a stack of Two-Stage Transformer Blocks (TSTB), which allow for both a local and a global information flow. We evaluate our model on the Semantic Urban Meshes (SUM) benchmark and a newly curated cultural-heritage dataset comprising textured roof tiles with triangle-level annotations for damage types. Our method achieves 81.9% mF1 and 94.3% OA on SUM and 49.7% mF1 and 72.8% OA on the new dataset, substantially outperforming existing approaches.

Figures & tables

Figure 1: Overview of the network architecture. The feature extraction branch generates one feature vector per face. The faces are clustered using K-means clustering. The tensor containing the feature vectors is reshaped so that there is one sequence of feature vectors per cluster, and a learnable cluster token is prepended to each sequence. These feature vectors are processed by a sequence of $N_{Bl}$ TSTB. The output of the final block is used to predict the class scores and labels. Numbers in brackets denote tensor dimensions.

Figure 2: Architecture of a TSTB. It consists of two sub-blocks: a local block in which the feature vectors within each face cluster can interact with each other, and a cross-cluster block designed to capture global context by allowing the cluster tokens of different clusters to interact. Numbers in brackets are dimensions of tensors.

Class	Training	Validation	Testing	Total
ND	1,445,547	348,638	530,823	2,325,008
J	778,955	182,090	234,293	1,195,338
St	337,499	48,771	119,975	506,245
D	8,185	1,342	2,837	12,364
LI	67,742	19,573	22,448	109,763
BC	575,191	144,778	202,052	922,021

Table 1: Number of triangles per class and number of tiles in the CH dataset and the training, validation and test subsets.

Class	T	V	Bld	W	C	B
$F1$ [%]	92.3	94.5	95.6	78.8	62.9	64.2

Table 2: Class-specific F1 scores of our method achieved on the SUM dataset. The

mF1

score is 81.9%, the

OA

is 94.3%.

Class	$ND$	$J$	$St$	$D$	$LI$	$BC$	$SE$
$F1$ [%]	78.3	81.8	50.1	0.0	62.9	63.4	11.7

Table 3: Class-specific F1 scores of our method achieved on the CH dataset. The

mF1

score is 49.7%, the

OA

is 72.8%.

${N_{F}}$	${OA}$ [%]	${mF1}$ [%]
32	90.6	74.9
64	90.8	75.4
128	92.9	78.6
256	94.3	81.9

Table 4: Effect of the feature dimension

{N_{F}}

on model performance on the SUM dataset.

Modality	${OA}$ [%]	${mF1}$ [%]
Geometry	84.1	66.9
Texture	88.1	70.2
Both	94.3	81.9

Table 5: Comparison of quality metrics achieved on the SUM dataset when using different input modalities.

Method	SUM		CH
Method	${OA}$	${mF1}$	${OA}$	${mF1}$
	[%]	[%]	[%]	[%]
C-A	93.8	81.2	70.0	46.3
S-A (ours)	94.3	81.9	72.8	49.7

Table 6: Comparison of the results achieved on the two test datasets using two architectures of the TSTBs. C-A: global interactions are considered by cross-attention, following Heidarianbaei et al. (2025) . S-A: our proposed method using self-attention in the cross-cluster blocks.

Method	SUM		CH
Method	$OA$	$mF1$	$OA$	$mF1$
DiffusionNet	80.4	60.1	63.8	25.9
NoMeFormer	84.3	66.6	67.9	34.1
Ours	94.3	81.9	72.8	49.7

Table 7: Overall Accuracy (

OA

) and mean F1 score (

mF1

), both in [%], achieved by our method and two baselines on the SUM and CH datasets.

Figure 3: Qualitative results achieved by our method and by the best-performing baseline, NoMeFormer. From left to right: Input, reference, our method, and NoMeFormer. Top: an example from SUM. Bottom: an example from CH. Colour code — SUM: $T$ (blue), $V$ (dark green), $Bld$ (green), $W$ (yellow), $C$ (orange), $B$ (red). Colour code — CH: $ND$ (greenish blue), $J$ (red), $St$ (green), $D$ (chartreuse green), $LI$ (gold), $BC$ (yellow), $SE$ (orange), unlabelled (blue).

Method	$OA$	$mF1$
RF-MRF	91.2	68.1
KPConv	93.3	76.7
DiffusionNet	81.0	61.3
NoMeFormer	84.9	68.6
Ours	93.3	78.4

Table 8: Area-weighted

OA

and

mF1

scores [%] achieved by our method and four baselines on the SUM dataset.

3D Semantic Segmentation

3D semantic segmentation assigns meaning to points, voxels, meshes, or reconstructed scene elements for spatial understanding.

See all 137 papers

Advances in Neural 3D Mesh Texturing: A Survey
May 28, 2026Sai Raj Kishore Perla, Hao Zhang, Ali Mahdavi-Amiri3D Asset Generation Texture Synthesis
Simon Fraser University
Intrinsic and Triangulation-Agnostic Attention: A Simple and Powerful Approach for Learning on Meshes
Jul 27, 2026Ashwath Shetty, Zihan Zhu, Soeren Pirk +13D ViTs Geometric Representation Learning
Université de Montréal & Mila, Canada · Kiel University, Germany
Edge Prediction for Roof Wireframe Reconstruction with Transformers
Jun 1, 2026Gustav Hanning, Ludvig Dillén, Jonathan Astermark +23D ViTs 3D Reconstruction
Centre for Mathematical Sciences, Lund University

May 28, 2026J/K move · Enter open · S save

Advances in Neural 3D Mesh Texturing: A Survey

Sai Raj Kishore Perla, Hao Zhang, Ali Mahdavi-Amiri

Simon Fraser University

3D Asset Generation Texture Synthesis 3D Texture Generation

Open paper AskPDF arXiv

Texturing 3D meshes plays a vital role in determining the visual realism of digital objects and scenes. Although recent generative 3D approaches based on Neural Radiance Fields and Gaussian Splatting can produce textured assets directly, polygonal meshes remain the core representation across modeling, animation, visual effects, and gaming pipelines. Neural 3D mesh texturing therefore continues to be an essential and active area of research. In this survey, we present a comprehensive review of recent advances in neural 3D mesh texturing, covering methods for texture synthesis, transfer, and completion. We first summarize key foundations in mesh geometry, texture mapping, differentiable rendering, and neural generative models, and then organize the literature into a unified taxonomy spanning early GAN-based methods to modern diffusion-based pipelines. We further analyze common architectures and supervision strategies, review datasets and evaluation protocols, and discuss emerging applications, practical/commercial systems, and open challenges. Together, these insights provide a structured perspective on the current landscape and help guide future developments in learning-based 3D mesh texturing.

Abstract

3D Semantic Segmentation

Explore similar work

Advances in Neural 3D Mesh Texturing: A Survey

Intrinsic and Triangulation-Agnostic Attention: A Simple and Powerful Approach for Learning on Meshes

Edge Prediction for Roof Wireframe Reconstruction with Transformers