Token compression is crucial for mitigating the quadratic complexity of self-attention mechanisms in Vision Transformers (ViTs), which often involve numerous input tokens. Existing methods, such as ToMe, rely on GPU-inefficient operations (e.g., sorting, scattered writes), introducing overheads that limit their effectiveness. We introduce MaMe, a training-free, differentiable token merging method based entirely on matrix operations, which is GPU-friendly to accelerate ViTs. Additionally, we present MaRe, its inverse operation, for token restoration, forming a MaMe+MaRe pipeline for image synthesis. When applied to pre-trained models, MaMe doubles ViT-B throughput with a 2% accuracy drop. Notably, fine-tuning the last layer with MaMe boosts ViT-B accuracy by 1.0% at 1.1x speed. In SigLIP2-B@512 zero-shot classification, MaMe provides 1.3x acceleration with negligible performance degradation. In video tasks, MaMe accelerates VideoMAE-L by 48.5% on Kinetics-400 with only a 0.84% accuracy loss. Furthermore, MaMe achieves simultaneous improvements in both performance and speed on some tasks. In image synthesis, the MaMe+MaRe pipeline enhances quality while reducing Stable Diffusion v2.1 generation latency by 31%. Collectively, these results demonstrate MaMe's and MaRe's effectiveness in accelerating vision models. The code is available at https://github.com/cominder/mame}{https://github.com/cominder/mame.
Figures & tables
Figure 1: The visualization illustrates the progression of token count reduction in the first 8 blocks of the AugReg ViT-B/16 with MaMe. Each color square represents a distinct type of token. More results are in the supplementary material.
Model
Method
FLOPs
Throughput
Top-1 Acc
(G)
(img/s)
(%)
Training Free on ImageNet-1K (224 × 224)
ViT-S (DeiT)
Baseline
4.6
5039
79.82
EViT
2.3
8950
73.83
ToMe
2.3
8874
77.99
DiffRate
2.3
8875
78.75
Table 1: The Results of Token compression on the off-the-shelf models. Throughput is measured on an A100 GPU with a batchsize of 1024 using FP16 precision.
Model
Param
FLOPs
Throughput
Top-1 Acc
(M)
(G)
(img/s)
(%)
ImageNet-1K Results (224 × 224)
ViT-T
5.72
1.3
2291
72.2
ViT-T †
5.72
0.6
4462
70.9
ViT-T ‡
5.72
1.0
4462
71.2
ViT-T ⋆
5.72
1.2
2468
72.4
Table 2: The Results of Vision Architectures with MaMe Compression. The mark † denotes MaMe compression applied to three blocks (layers 3/6/9) with training from scratch; ‡ indicates MaMe applied to three blocks (layers 3/6/9) followed by fine-tuning; ⋆ denotes MaMe applied only to the last transformer block followed by fine-tuning. Throughput is measured on an A100 GPU with a batch size of 64 using FP32 precision.
Model
Method
Input Size
Throughput
Top-1 Acc
(px)
(img/s)
(%)
Zero-Shot Classification on ImageNet-1K
CLIP (ViT-L/14)
Baseline
224
51.22
70.34
ToMe(r=8)
224
51.33
68.98
ToMe(r=12)
224
52.86
66.00
MaMe( τ=0.7 )
224
69.09
67.60
Table 3: Zero-shot image classification results. Inference throughput measured on a 3090 GPU with FP32 precision.
Method
Latency(s)
Bleu-1
Bleu-2
Bleu-3
Bleu-4
ROUGE_L
CIDEr
LLaVA-1.5-7B
3.12
20.72
13.28
8.08
4.93
20.94
0.71
+ ToMe
2.30
20.15
12.90
7.90
4.89
21.67
1.60
+ MaMe
2.61
20.10
12.87
7.87
4.83
21.69
2.71
Table 4: COCO caption task performance of the LLaVA-1.5-7B model and its modified versions by ToMe and MaMe.
COCO Caption Task
Question:
Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:
The image features a white plate with a variety of food items on it. There are two pieces of broccoli, one on the left side and the other on the right side of the plate. Additionally, there are two pieces of fish, one on the left side and the other on the right side of the plate. A fork is placed on the right side of the plate, ready to be used for eating. The plate is placed on a dining table, which is the main focus of the image.
+ToMe:
The image features a white plate with a variety of food items on it. There are two pieces of fish, one on the left side and the other on the right side of the plate. A generous portion of broccoli is spread across the plate, with some pieces located near the fish and others scattered around the plate. A fork and a knife are placed on the table, ready for use. The dining table occupies most of the image, with the plate and utensils placed on it.
+MaMe:
The image features a white plate with a variety of food items on it. There are two pieces of broccoli, one on the left side and the other on the right side of the plate. Two pieces of fish are also present on the plate, located towards the center . In addition to the food, there are utensils on the table. A fork is placed on the left side of the plate, while a knife is located on the right side. A cup can be seen on the left side of the table , and a bowl is placed on the right side. The dining table occupies the majority of the image, with the plate and utensils placed on top of it .
Table 5: MaMe correctly identified core components (plate, broccoli, fish, fork, knife) and their spatial arrangements, plus additional items (cup) that the other models miss. While it includes an erroneous "bowl," its output is structured with greater descriptive precision compared to the more generic and less detailed descriptions from LLaVA and ToMe. More results are in the supplementary material.
Model
Method
Input
Throughput
Top-1 Acc
(FxHW)
(videos/s)
(%)
Action Recognition on Kinetics-400
VideoMAE-B
Baseline
16x224
13.24
76.81
ToMe(r=96)
16x224
13.76
75.54
ToMe(r=128)
16x224
14.06
73.34
MaMe( τ=0.8 )
16x224
13.81
74.23
Table 6: Results of VideoMAE on action recognition benchmarks. Inference throughput is measured on a 3090 GPU with FP16 precision.
Figure 2: Qualitative Results. Images from left to right generated by Stable Diffusion v2.1, ToMeSD and MaMe+MaRe. MaMe+MaRe retains high-frequency details (left) and produce images with higher aesthetic quality (right). More results are in the supplementary material.
Figure 3: Qualitative Results. The image generated by the model with MaMe. The clarity of the picture changed with the similarity threshold. A higher similarity threshold brings a more detailed and clearer painting. More results are in the supplementary material.
Speed
Image Quality
Latency (s) ↓
FID ↓
IS ↑
LPIPS ↓
PSNR ↑
SSIM ↑
CLIP ↑
baseline
17.26
1.93
22.75
0.68
8.80
0.17
23.41
ToMeSD(25% ratio)
14.81
2.04
23.47
0.64
9.07
0.19
23.48
ToMeSD(50% ratio)
12.67
3.68
22.21
0.65
9.13
0.21
23.51
MaMe+MaRe( τ=0.7 )
11.87
2.54
20.84
0.63
9.77
0.20
24.15
MaMe+MaRe( τ=0.8 )
12.45
2.28
23.31
0.63
9.79
0.21
24.18
Table 7: Text-to-image generation performance comparison results with SD2.1 (baseline) in generating 768×768 images, based on 50 sampling steps.
Figure 4: Ablation experiments using AugReg ViT-B/16. Our default settings are marked in Gray. We report Top-1 accuracy (acc) and model inference throughput (im/s) with FP32 precision and batchsize 1 on a 3090 GPU. The visualization results of different methods are shown in Figure 5 .
Figure 6: The accuracy and throughput change with the similarity threshold under both alternating and random partitions. num_block denotes the number of blocks used for merging. There is a non-linear, saturating relationship between the similarity threshold and accuracy across ViT architectures. Five different random seeds are employed to conduct five experiments under different ratios of tokens as source tokens as shown in the box in the Figures (c) and (d), illustrating that the default, determined alternating partition curves represent a Pareto frontier.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The visualization in the 8th block of the AugReg ViT-B/16 using MaMe with different settings. Each color square represents a distinct type of token. Default is our default method.
Figure 8: The visualization illustrates the progression of token count reduction in the first 8 blocks of the AugReg ViT-B/16 with MaMe. Each color square represents a distinct type of token.
COCO Caption Task
Question:
Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:
2 people are standing in front of a table with a variety of wine bottles and a plate of food. The man is holding a wine bottle and the woman is holding a wine glass. They are both smiling and enjoying their time together. The table is filled with wine bottles and a few wine glasses, and there is a cake on the table as well.
+ToMe:
2 people are standing at a table with a variety of wine bottles and glasses. The man is wearing a yellow shirt and the woman is wearing a black shirt. They are both holding wine glasses and appear to be tasting different wines. There are several bottles of wine on the table, some of which are placed closer to the man and others near the woman. A plate of food is also present on the table, adding to the ambiance of the wine tasting event.
+MaMe:
2 people are standing at a table with a variety of food and drinks. The man is wearing a yellow shirt and the woman is wearing a black shirt . They are both holding wine glasses and appear to be enjoying a meal together. The table is filled with several bottles of wine, a plate of food, and a fork . The scene is set in a restaurant with a window view .
Appendix
Table 8: MaMe provides the most accurate and comprehensive description, correctly identifying clothing details (yellow shirt, black shirt) and contextual elements (restaurant setting, window view). ToMe captures basic elements accurately but lacks environmental context. LLaVA omitting important visual details like clothing colors. MaMe demonstrates superior visual grounding and contextual understanding.
COCO Caption Task
Question:
Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:
A man wearing a green and white ski suit is skiing down a snowy hill. He is holding ski poles in his hands and appears to be enjoying the activity. The man is the main focus of the image, and the snowy hill and trees in the background provide a picturesque winter scene.
+ToMe:
24 is the number on the man’s outfit.
+MaMe:
A man wearing a green and white ski suit is skiing down a snowy hill. He is holding ski poles in his hands and appears to be enjoying the activity. The man is wearing a hat and goggles, which are essential for protecting his eyes from the cold and snow . The scene captures the excitement and thrill of skiing in a beautiful winter landscape.
Appendix
Table 9: MaMe delivers the most comprehensive and context-aware description, accurately identifying all visual elements (ski suit, poles, hat, goggles) and providing relevant contextual reasoning about protective gear. LLaVA covers basics but omits key details like headgear, while ToMe fails entirely, producing only an unrelated fragment ("number 24").
Figure 9: Images from left to right generated by Stable Diffusion v2.1, ToMeSD and MaMe+MaRe. MaMe+MaRe retains high-frequency details, making the picture more realistic.
Figure 10: Images generated by the model with MaMe+MaRe. The clarity of the painting changed with the similarity threshold. A higher similarity threshold brings a more detailed and clearer painting.
The quadratic cost of self-attention in Vision Transformers (ViTs) constitutes a fundamental bottleneck for practical deployment, motivating a vibrant line of research on token reduction. Among existing approaches, token merging (ToMe) has emerged as an elegant training-free solution; yet its design rests on an unspoken premise of token equality, which contravenes the well-documented non-uniformity of self-attention and leads to information loss in high-salience tokens under aggressive compression. We address this limitation with AdaMerge, a token-merging framework based on two complementary mechanisms. First, salience-weighted similarity leverages column-wise feature-affinity centrality as a token-importance proxy and incorporates the resulting salience scores into the bipartite matching score, ensuring that pivotal tokens contribute more strongly to the merged representation. Second, adaptive merging intensity uses pre-computed layer-wise similarity statistics to dynamically modulate the per-layer reduction count in accordance with input-specific redundancy. On ImageNet-1k with ViT-B/16, AdaMerge consistently outperforms ToMe, PiToMe, and DSM across all FLOPs-matched regimes. The accuracy gap widens monotonically with compression: at the 13.4G FLOPs operating point, AdaMerge sustains a Top-1 degradation of only -1.06%, compared to -1.45% for PiToMe and -4.62% for DSM. To our knowledge, AdaMerge is the first to combine salience-weighted similarity and adaptive per-layer reduction into a single training-free token merging framework, advancing the accuracy-FLOPs Pareto frontier of ViT acceleration.
Semi Lee, Hyejin Go, Hyesong Choi
Electronic Engineering · Soongsil University · Seoul, South Korea
Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.
Vatsal, Soumil Malhotra, Priyanshu Chahal +1
Department of Mathematics, Indian Institute of Technology Roorkee
Vision Transformers (ViTs) face severe computational bottlenecks due to the quadratic complexity of self-attention at high resolutions. Existing token reduction methods rely on local metrics - such as single-layer attention scores - that are inherently vulnerable to the attention sink phenomenon, where uninformative tokens are paradoxically preserved over salient foreground objects. We propose ASAP (Attention Sink Anchored Pruning), a training-free framework that recasts this sink as a feature. Modeling ViT information flow as a Lazy Random Walk, ASAP identifies the sink as a dominant accumulator of probability mass. By computing the diffusion distance to the sink within the cumulative transition matrix, ASAP partitions tokens via Radial Diffusion Clustering and compresses background redundancy through Transition Weight Pooling in a single shot. Extensive experiments across image, video, and vision-language tasks demonstrate ASAP outperforms state-of-the-art methods, accelerating throughput by up to 48% while maintaining - or even exceeding - baseline accuracy.
Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea