MPM: Mutual Pair Merging for Efficient Vision Transformers

Simon Ravé; Pejman Rasti; David Rousseau

cs.CVApr 7, 2026

MPM: Mutual Pair Merging for Efficient Vision Transformers

Authors: Simon Ravé, Pejman Rasti, David Rousseau

Organizations: LARIS University of Angers Angers, France · UMR INRAe-IRHS Angers, France

Abstract

Decreasing sequence length is a common way to accelerate transformers, but prior token reduction work often targets classification and reports proxy metrics rather than end-to-end latency. For semantic segmentation, token reduction is further constrained by the need to reconstruct dense, pixel-aligned features, and on modern accelerators the overhead of computing merge maps can erase expected gains. We propose Mutual Pair Merging (MPM), a training-free token aggregation module that forms mutual nearest-neighbor pairs in cosine space, averages each pair, and records a merge map enabling a gather-based reconstruction before the decoder so that existing segmentation heads can be used unchanged. MPM introduces no learned parameters and no continuous compression knob (no keep-rate or threshold). The speed-accuracy trade-off is set by a discrete insertion schedule. We benchmark end-to-end latency on an NVIDIA H100 GPU (with and without FlashAttention-2) and a Raspberry Pi 5 across standard segmentation datasets. On ADE20K, MPM reduces per-image latency by up to 60% for ViT-Tiny on Raspberry Pi 5, and increases throughput by up to 20% on H100 with FlashAttention-2 while keeping the mIoU drop below 3%. These results suggest that simple, reconstruction-aware, training-free token merging can translate into practical wall-clock gains for segmentation when overhead is explicitly accounted for.

Figures & tables

Figure 1: Visual abstract of Mutual Pair Merging (MPM). Similar tokens are matched by mutual pairs and averaged together. Tokens without a mutual relationship remain as singletons. A merge map is kept in memory to enable reconstruction of the entire token sequence. MPM has no learned parameters and no continuous compression knob; the speed-accuracy trade-off is controlled by the choice of insertion blocks.

Figure 2: Visualization of MPM on the same image during daytime and nighttime. Nighttime is simulated by reducing the image’s luminosity and adding a small amount of thermal and shot noise modeled by Gaussian and Poisson distributions. Approximately 6% fewer tokens are merged at night than during the day.

Model	mIoU	GFLOPs	Latency (FPS)
Seg-T/16	38.1	25	660
ALGM*	38.9	$\sim$ 16.7	665
CTS*	38.2	-	547
ToMe	38.1	$\sim$ 19	751
MPM (2,5)	37.6	$\sim$ 17.6	831
Seg-S/16	45.3	76.9	333

Table 1: Comparison of main token reduction methods on ADE20K; images are 512x512 pixels. Results were recorded on a single H100 GPU in full precision without FlashAttention. * indicates methods that require training or fine-tuning, and

\sim

denotes the average recorded FLOPs per image over the entire dataset with random batches.

Model	mIoU	GFLOPs	FPS (B=32)
ViT-S/16	53.0	64	426
CTS*	52.7	-	438
ALGM*	53.2	$\sim$ 52	430
ToMe	53	$\sim$ 45	505
MPM (2,5)	52.5	$\sim$ 44	561
ViT-B/16	55.0	219	159

Table 2: Results on Pascal Context dataset (H100, Float 32) with Mask Transformer segmentation head [ 26 ] , 480x480px per image, base sequence length of 900 tokens for patch size 16.

Model	mIoU	GFLOPs	FPS (B=32)
ViT-S/16	76.3	116	106
CTS*	76.6	-	169
ALGM*	76.9	79	132
ToMe	76.3	$\sim$ 97	105
MPM (2,5)	75.3	$\sim 81$	158
ViT-B/16	77.3	348	44

Table 3: Results on the Cityscapes dataset (H100, float32) with the Mask Transformer segmentation head [ 26 ] . * indicates methods that require training or fine-tuning. Input images were 768x768, corresponding to 2304 tokens in the base sequence.

Model	FPS B=1	FPS B=2
ViT-T/16	1.06	1.05
ALGM*	1.64	1.47
MPM (2,5)	1.71	1.75
ViT-S/16	0.49	0.44
ALGM*	0.69	0.64
MPM (2,5)	0.73	0.71

Table 4: Comparison of latency for merging methods on a Raspberry Pi 5 on the ADE20K dataset.

Model	Latency Batch Size = 32 (FPS)
ViT-B/16	735
ALGM*	643
ToMe	753
MPM (2,5)	780
ViT-L/16	375
ALGM*	351

Table 5: Latency comparison (FPS) for different merging methods at batch size 32 on H100 GPU in half precision (BFloat16) using FlashAttention-2 [ 6 ] .

Model	mIoU	GFLOPs	FPS (B=32)
DeiT-B Mask	46.2	259	130
MPM (2,5)	45.6	$\sim 187$	174
ViT-B/16 Linear	47.8	214	159
MPM (2,5)	46.7	$\sim 141$	228

Table 6: Results using a DeiT backbone [ 27 ] with a Mask Transformer head and a base ViT with a linear head. We observe trends similar to those in Tab. 1 .

Model	mIoU	FPS (B=4)
EVA-01	61.5	2.61
MPM (2,5)	61.4	3.70

Table 7: Results on ADE20K using EVA-01 [ 9 ] backbone with ViT Adapter [ 3 ] and Mask2Former [ 4 ] head. Due to memory limitations, latency is measured at batch size 4 on a single A100 GPU.

Figure 3: Accuracy vs. FPS for MPM using different insertion layers in ViT-Base [ 7 ] on ADE20K [ 38 ] .

Explore similar work

Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
Sep 16, 2026Victor Bercy, Martyna Poreba, Michal Szczepanski +1Visual Token Merging Efficient ViTs
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Université Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea
MergeOver: Post-Training Token Merging for Recursive Vision Transformers
Aug 13, 2026Junseo Kim, Uraz Odyurt, Amirreza YousefzadehVision Transformer Efficient ViTs
Computer Architecture for Embedded Systems, University of Twente Enschede, The Netherlands

Sep 16, 2026J/K move · Enter open · S save

Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM

Victor Bercy, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Université Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France

Visual Token Merging Efficient ViTs Vision Transformer Semantic Segmentation

Open paper AskPDF arXiv

Vision Transformers (ViTs) have achieved state-of-the-art performance across a range of computer vision tasks, mainly thanks to the self-attention mechanism. However, its complexity, increasing quadratically with the number of tokens, remains the major obstacle to ViT efficiency and deployment at scale. Token merging reduces this cost by aggregating redundant tokens. Yet existing methods are typically evaluated within a single architecture, leaving open whether their effectiveness stems from the merging mechanism itself or from the specific decoder they are paired with. We extend Graph-Guided Token Merging (G2TM), a single module inserted early in a ViT-based network, beyond its original Segmenter setting. We evaluate G2TM across three semantic segmentation frameworks (Segmenter, SETR, EoMT) and three decoder families (Linear, Transformer-, convolution-based), as well as standard ViT image classification. Our results show that G2TM's behavior and accuracy-efficiency trade-off are consistent across every tested architecture for a given backbone size, indicating that its effectiveness is a property of the encoder rather than the decoder. G2TM also generalizes well to image classification, achieving an even smaller degradation in accuracy compared to semantic segmentation. We further find that G2TM's optimal hyperparameters, resulting in a consistent drop in GFLOPs of 22-47% and an increase in throughput by up to 74% for segmentation models on ADE20K dataset, depend primarily on the backbone's pre-training recipe and on the target dataset, rather than on the decoder choice.

Abstract

Explore similar work

Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

MergeOver: Post-Training Token Merging for Recursive Vision Transformers