MME: Mixture of Mesh Experts with Random Walk Transformer Gating

Amir Belder; Ayellet Tal

cs.CVFeb 28, 2026

MME: Mixture of Mesh Experts with Random Walk Transformer Gating

Authors: Amir Belder, Ayellet Tal

Organizations: Technion - Israel Institute of Technology, Israel

Abstract

In recent years, various methods have been proposed for mesh analysis, each offering distinct advantages and often excelling on different object classes. We present a novel Mixture of Experts (MoE) framework designed to harness the complementary strengths of these diverse approaches. We propose a new gate architecture that encourages each expert to specialise in the classes it excels in. Our design is guided by two key ideas: (1) random walks over the mesh surface effectively capture the regions that individual experts attend to, and (2) an attention mechanism that enables the gate to focus on the areas most informative for each expert's decision-making. To further enhance performance, we introduce a dynamic loss balancing scheme that adjusts a trade-off between diversity and similarity losses throughout the training, where diversity prompts expert specialization, and similarity enables knowledge sharing among the experts. Our framework achieves state-of-the-art results in mesh classification, retrieval, and semantic segmentation tasks. Our code is available at: https://github.com/amirbelder/MME-Mixture-of-Mesh-Experts.

Figures & tables

Panel of Figure 1 — Figure 1: Model. At each iteration ( $t$ ), the expert environment receives a batch of meshes along with a weighting factor ( $\lambda_{t}$ ) that balances the loss terms. It outputs a weight for each expert ( $s_{t}$ ) and the batch’s accuracy ( $r_{t}$ ), which are then passed to the agent as input for the next iteration ( $t+1$ ).

Panel of Figure 2 — Figure 2: Expert environment. At each iteration $t$ , all experts receive the same batch of meshes and independently produce their predictions. The same meshes are also fed into the random walk extractor, which generates walk-based representations. The extracted random walks are then passed to the Transformer gate as input, which assigns weights to the experts. The experts’ predictions, along with the gate’s weights, are passed to the expert chooser, that selects which expert’s prediction to use for each mesh, specifically, by choosing the one with the highest assigned weight. The similarity loss, weighted by $\lambda_{t}$ , is applied to the experts’ predictions ( $V_{1},\dots,V_{J}$ ), while the diversity loss is applied to the final prediction, $V_{\text{chosen}}$ . The gate’s weightings form the next state ( $s_{t}$ ) and the batch’s accuracy forms the next reward ( $r_{t}$ ).

Panel of Figure 3 — Figure 3: Transformer gate. We zoom into the gate model. The gate’s input is comprised of a random walk which is extracted from a given input mesh. The gate outputs a weight for each of the $J$ experts, which is the key to selecting the prediction of the most suitable expert for the mesh. The gate is comprised of an encoder and a decoder. The encoder processes the walk, and its output serves as input to the decoder. It consist of a single embedding layer followed by $8$ Multi-Head Attention (MHA) layers. The decoder outputs a single weight for each expert (leading to an output at the size of $J$ ). It consists of a single embedding layer, followed by $8$ MHA layers, and a FC layer at the size of the number of experts ( $J$ ).

Model	3D-Future	ModlNet40
MeshNet [ FFY∗19 ]	$64.1\%$	$91.9\%$
MeshWalker [ LT20 ]	$70.2\%$	$92.3\%$
AttWalk [ BILT22 ]	$72.1\%$	$92.5\%$
MeshNet++ [ SSK21 ]	$71.4\%$	$91.6\%$
WalkFormer [ GHF∗23 ]	-	$91.1\%$
Ensemble	$78.0\%$	$92.3\%$

Table 1: Classification results. In this experiment, we used three experts: MeshWalker, AttWalk, and MeshNet. Our results outperform those of other models and, notably, surpass the individual performance of each expert. It is also more effective than combining these experts using an ensemble approach.

Model	SHREC11	Cube Engraving
MeshWalker [ LT20 ]	$97.1\%$	$98.6\%$
AttWalk [ BILT22 ]	$99.7\%$	-
MeshNet++ [ SSK21 ]	$99.8\%$	$98.5\%$
MeshCNN [ HHF∗19 ]	$91.0\%$	$92.2\%$
GWCNN [ ESKBC17 ]	$90.3\%$	-
PD-MeshNet [ MLR∗20 ]	$99.1\%$	$94.4\%$

Table 2: Classification results. In this experiment, we used three experts: MeshWalker, MeshCNN, and PD-MeshNet. Our results not only outperform those of other models but also exceed the individual performance of each expert and the ensemble.

Method	ShapeNet-Core55		ModelNet40
	mAP	NDCG	mAP
AttWalk [ BILT22 ]	$81.1\%$	$86.7\%$	$91.2\%$
MeshWalker [ BILT22 ]	$79.2\%$	$84.5\%$	$89.9\%$
DALN [ SYS∗17 ]	$66.3\%$	$76.2\%$	-
MeshNet [ FFY∗19 ]	$70.2\%$	$75.3\%$	$81.9\%$
Ensemble	$84.3\%$	$85.6\%$	${89.2\%}$

Table 3: Retrieval results. In this setting, we use MeshWalker, AttWalk, and MeshNet as expert models. We note that the results of MeshNet on ShapeNet-Core55 are from our own reproduction, as they are not reported in the original paper. Our results outperform those of other models and surpass the individual performance of each expert, as well as the ensemble.

Model	Human body		COSEG	PartNet
Model	Face	Edge	Edge	Face	Edge
MeshWalker	$92.7\%$	$94.8\%$	$99.1\%$	-	-
WalkFormer	-	$98.0\%$	$99.4\%$	-	-
MeshCNN	$89.0\%$	$92.3\%$	$98.2\%$	$61.0\%$	$62.1\%$
MeT	-	$93.6\%$	$99.3\%$	-	-
PD-MeshNet	$85.6\%$	-	$96.9\%$	$58.7\%$	$59.8\%$

Table 4: Semantic segmentation accuracy. In this experiment, we used three experts: MeshFormer, MeshCNN, and PD-MeshNet. Our results outperform those of other models and surpass the individual performance of each expert as well as the ensemble. We note that since the competing methods do not report results on PartNet, we ran their publicly available code to obtain the results used as our expert baselines.

Panel of Figure 4 — Figure 4: Qualitative results. Body parts are clearly segmented and consistently labeled across poses.

Panel of Figure 5 — Figure 5: Qualitative results of segmentation (COSEG). Our segmentations closely match the ground truth, whereas PD-MeshNet produces errors on these objects. This is because our model selected MeshCNN as the expert for these cases, rather than PD-MeshNet.

Gate	Classi.	Seg.	Retrieval
$2$ FC	$74.2\%$	$92.0\%$	$83.4\%$
3D conv. + $2$ FC	$76.4\%$	$92.2\%$	$84.5\%$
MeshNet [ FFY∗19 ]	$80.0\%$	$92.7\%$	$85.1\%$
PD-MeshNet [ MLR∗20 ]	$81.7\%$	$93.5\%$	$85.5\%$
MeshWalker [ LT20 ]	$83.8\%$	$93.8\%$	$86.9\%$
AttWalk [ BILT22 ]	$84.6\%$	$94.0\%$	$90.5\%$

Table 5: Different gates. Our method’s gating mechanism achieves the best results compared to other gating alternatives. In this experiment, we evaluated it on 3D-FUTURE classification, Human Body segmentation, and ShapeNet-Core55 retrieval.

Panel of Figure 6 — Figure 6: $\lambda$ ’s dynamic value . The value of $\lambda$ changes dynamically along 3D-FUTURE’s training, transitioning between positive values that promote similarity and negative values that encourage diversity, in order to achieve optimal accuracy.

$\lambda$ ’s Value	Accuracy
$-1$	$70.1\%$
$0$	$74.8\%$
$0.01$	$82.3\%$
$0.1$	$76.3\%$
$0.5$	$76.7\%$
$1$	$77.1\%$

Table 6: Comparison with static

\lambda

values. Our dynamic

\lambda

strategy achieves the highest accuracy

Loss	Accuracy
Diversity	$82.3\%$
Diversity + Cosine	$84.0\%$
Diversity + MSE	$82.9\%$
Diversity + KLD (Ours)	$\mathbf{86.1\%}$

Table 7: Accuracy with different losses. KLD as the similarity loss achieves the highest accuracy, outperforming alternative similarity losses and diversity-only training.

Class	Accuracy			Expert choice
Class	AttWalk	MeshWalker	MeshNet	Expert choice
Armchair	$\mathbf{8}7.6\%$	$78.3\%$	$70.7\%$	$90.9\%$ AttWalk
Bar stool	$74.6\%$	$\mathbf{8}3.2\%$	$64.8\%$	$80.0\%$ MeshWalker
Kids Bed	$70.5\%$	$68.3\%$	$\mathbf{8}0.2\%$	$91.7\%$ MeshNet

Table 8: Expert selection at inference. Each expert excelled on different classes (3D-FUTURE): AttWalk on Armchair, MeshWalker on Bar stool, and MeshNet [ FFY∗19 ] on Kids Bed. During inference the gate selected the best model for each class, e.g. AttWalk’s classification for

90.9\%

of the Armchair.

Class	AttWalk (Armchair)		MeshWalker (Bar stool)		MeshNet (Kids Bed)
Class	Before	After	Before	After	Before	After
Armchair	$80.4\%$	$\mathbf{87.6}\%$	$79.4\%$	$78.3\%$	$71.5\%$	$70.7\%$
Bar stool	$76.1\%$	$74.6\%$	$75.2\%$	$\mathbf{83.2}\%$	$66.0\%$	$64.8\%$
Kids Bed	$71.7\%$	$70.5\%$	$69.3\%$	$68.3\%$	$72.7\%$	$\mathbf{80.2}\%$

Table 9: Individual improvement. Our MoE training process improves each expert’s performance on its designated classes; as expected, the performance of the other experts on those classes correspondingly decreases.

Experts	Accuracy
AttWalk [ BILT22 ]	$72.1\%$
$3$ AttWalk	$83.3\%$
MeshWalker [ LT20 ]	$70.2\%$
$3$ MeshWalker	$82.9\%$
MeshNet [ FFY∗19 ]	$64.1\%$
$3$ MeshNet	$75.6\%$

Table 10: Comparison of homogeneous and heterogeneous configurations. While multiple identical experts can still specialise and improve performance, the heterogeneous setting yields the best results due to architectural diversity.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Rethinking Routing Exploration in Modern Mixture-of-Experts
Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts Models Expert Routing
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
Mar 20, 2026Abien Fred Agarap, Arnulfo P. AzcarragaMixture of Experts Mixture-of-Experts Models
De La Salle University, Manila, Philippines
Structure Aware Neural Architecture Search for Mixture of Experts
Aug 30, 2026Petr Babkin, Oleg BakhteevExpectation-Maximization Neural Architecture Search

Oct 15, 2026J/K move · Enter open · S save

Rethinking Routing Exploration in Modern Mixture-of-Experts

Bingcong Li, Hao Ma, Michael Muehlebach, Niao He+1

ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems+1

Mixture-of-Experts Models Expert Routing Sparse Mixture-of-Experts

Open paper AskPDF arXiv

Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.

Abstract

Mixture-of-Experts Models

Explore similar work

Rethinking Routing Exploration in Modern Mixture-of-Experts

Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement

Structure Aware Neural Architecture Search for Mixture of Experts