MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis

Simin Huo; Ning Li

cs.CVApr 15, 2026

MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis

Authors: Simin Huo, Ning Li

Organizations: Shanghai Jiao Tong University, , Shanghai, 200240, Shanghai, China

Abstract

Token compression is crucial for mitigating the quadratic complexity of self-attention mechanisms in Vision Transformers (ViTs), which often involve numerous input tokens. Existing methods, such as ToMe, rely on GPU-inefficient operations (e.g., sorting, scattered writes), introducing overheads that limit their effectiveness. We introduce MaMe, a training-free, differentiable token merging method based entirely on matrix operations, which is GPU-friendly to accelerate ViTs. Additionally, we present MaRe, its inverse operation, for token restoration, forming a MaMe+MaRe pipeline for image synthesis. When applied to pre-trained models, MaMe doubles ViT-B throughput with a 2% accuracy drop. Notably, fine-tuning the last layer with MaMe boosts ViT-B accuracy by 1.0% at 1.1x speed. In SigLIP2-B@512 zero-shot classification, MaMe provides 1.3x acceleration with negligible performance degradation. In video tasks, MaMe accelerates VideoMAE-L by 48.5% on Kinetics-400 with only a 0.84% accuracy loss. Furthermore, MaMe achieves simultaneous improvements in both performance and speed on some tasks. In image synthesis, the MaMe+MaRe pipeline enhances quality while reducing Stable Diffusion v2.1 generation latency by 31%. Collectively, these results demonstrate MaMe's and MaRe's effectiveness in accelerating vision models. The code is available at https://github.com/cominder/mame}{https://github.com/cominder/mame.

Figures & tables

Panel of Figure 1 — Figure 1: The visualization illustrates the progression of token count reduction in the first 8 blocks of the AugReg ViT-B/16 with MaMe. Each color square represents a distinct type of token. More results are in the supplementary material.

Model	Method	FLOPs	Throughput	Top-1 Acc
Model	Method	(G)	(img/s)	(%)
Training Free on ImageNet-1K (224 $\times$ 224)
ViT-S (DeiT)	Baseline	4.6	5039	79.82
	EViT	2.3	8950	73.83
	ToMe	2.3	8874	77.99
	DiffRate	2.3	8875	78.75

Table 1: The Results of Token compression on the off-the-shelf models. Throughput is measured on an A100 GPU with a batchsize of 1024 using FP16 precision.

Model	Param	FLOPs	Throughput	Top-1 Acc
Model	(M)	(G)	(img/s)	(%)
ImageNet-1K Results (224 $\times$ 224)
ViT-T	5.72	1.3	2291	72.2
ViT-T †	5.72	0.6	4462	70.9
ViT-T ‡	5.72	1.0	4462	71.2
ViT-T ⋆	5.72	1.2	2468	72.4

Table 2: The Results of Vision Architectures with MaMe Compression. The mark † denotes MaMe compression applied to three blocks (layers 3/6/9) with training from scratch; ‡ indicates MaMe applied to three blocks (layers 3/6/9) followed by fine-tuning; ⋆ denotes MaMe applied only to the last transformer block followed by fine-tuning. Throughput is measured on an A100 GPU with a batch size of 64 using FP32 precision.

Model	Method	Input Size	Throughput	Top-1 Acc
Model	Method	(px)	(img/s)	(%)
Zero-Shot Classification on ImageNet-1K
CLIP (ViT-L/14)	Baseline	224	51.22	70.34
	ToMe(r=8)	224	51.33	68.98
	ToMe(r=12)	224	52.86	66.00
	MaMe( $\tau=0.7$ )	224	69.09	67.60

Table 3: Zero-shot image classification results. Inference throughput measured on a 3090 GPU with FP32 precision.

Method	Latency(s)	Bleu-1	Bleu-2	Bleu-3	Bleu-4	ROUGE_L	CIDEr
LLaVA-1.5-7B	3.12	20.72	13.28	8.08	4.93	20.94	0.71
+ ToMe	2.30	20.15	12.90	7.90	4.89	21.67	1.60
+ MaMe	2.61	20.10	12.87	7.87	4.83	21.69	2.71

Table 4: COCO caption task performance of the LLaVA-1.5-7B model and its modified versions by ToMe and MaMe.

COCO Caption Task

Question:	Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:	The image features a white plate with a variety of food items on it. There are two pieces of broccoli, one on the left side and the other on the right side of the plate. Additionally, there are two pieces of fish, one on the left side and the other on the right side of the plate. A fork is placed on the right side of the plate, ready to be used for eating. The plate is placed on a dining table, which is the main focus of the image.
+ToMe:	The image features a white plate with a variety of food items on it. There are two pieces of fish, one on the left side and the other on the right side of the plate. A generous portion of broccoli is spread across the plate, with some pieces located near the fish and others scattered around the plate. A fork and a knife are placed on the table, ready for use. The dining table occupies most of the image, with the plate and utensils placed on it.
+MaMe:	The image features a white plate with a variety of food items on it. There are two pieces of broccoli, one on the left side and the other on the right side of the plate. Two pieces of fish are also present on the plate, located towards the center . In addition to the food, there are utensils on the table. A fork is placed on the left side of the plate, while a knife is located on the right side. A cup can be seen on the left side of the table , and a bowl is placed on the right side. The dining table occupies the majority of the image, with the plate and utensils placed on top of it .

Table 5: MaMe correctly identified core components (plate, broccoli, fish, fork, knife) and their spatial arrangements, plus additional items (cup) that the other models miss. While it includes an erroneous "bowl," its output is structured with greater descriptive precision compared to the more generic and less detailed descriptions from LLaVA and ToMe. More results are in the supplementary material.

Model	Method	Input	Throughput	Top-1 Acc
Model	Method	(FxHW)	(videos/s)	(%)
Action Recognition on Kinetics-400
VideoMAE-B	Baseline	16x224	13.24	76.81
	ToMe(r=96)	16x224	13.76	75.54
	ToMe(r=128)	16x224	14.06	73.34
	MaMe( $\tau=0.8$ )	16x224	13.81	74.23

Table 6: Results of VideoMAE on action recognition benchmarks. Inference throughput is measured on a 3090 GPU with FP16 precision.

Panel of Figure 2 — Figure 2: Qualitative Results. Images from left to right generated by Stable Diffusion v2.1, ToMeSD and MaMe+MaRe. MaMe+MaRe retains high-frequency details (left) and produce images with higher aesthetic quality (right). More results are in the supplementary material.

Panel of Figure 3 — Figure 3: Qualitative Results. The image generated by the model with MaMe. The clarity of the picture changed with the similarity threshold. A higher similarity threshold brings a more detailed and clearer painting. More results are in the supplementary material.

	Speed	Image Quality
	Latency (s) $\downarrow$	FID $\downarrow$	IS $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	CLIP $\uparrow$
baseline	17.26	1.93	22.75	0.68	8.80	0.17	23.41
ToMeSD(25% ratio)	14.81	2.04	23.47	0.64	9.07	0.19	23.48
ToMeSD(50% ratio)	12.67	3.68	22.21	0.65	9.13	0.21	23.51
MaMe+MaRe( $\tau=0.7$ )	11.87	2.54	20.84	0.63	9.77	0.20	24.15
MaMe+MaRe( $\tau=0.8$ )	12.45	2.28	23.31	0.63	9.79	0.21	24.18

Table 7: Text-to-image generation performance comparison results with SD2.1 (baseline) in generating 768×768 images, based on 50 sampling steps.

Panel of Figure 4 — Figure 4: Ablation experiments using AugReg ViT-B/16. Our default settings are marked in Gray. We report Top-1 accuracy (acc) and model inference throughput (im/s) with FP32 precision and batchsize 1 on a 3090 GPU. The visualization results of different methods are shown in Figure 5 .

Panel of Figure 6 — Figure 6: The accuracy and throughput change with the similarity threshold under both alternating and random partitions. $num\_block$ denotes the number of blocks used for merging. There is a non-linear, saturating relationship between the similarity threshold and accuracy across ViT architectures. Five different random seeds are employed to conduct five experiments under different ratios of tokens as source tokens as shown in the box in the Figures (c) and (d), illustrating that the default, determined alternating partition curves represent a Pareto frontier.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: The visualization in the 8th block of the AugReg ViT-B/16 using MaMe with different settings. Each color square represents a distinct type of token. Default is our default method.

Panel of Figure 8 — Figure 8: The visualization illustrates the progression of token count reduction in the first 8 blocks of the AugReg ViT-B/16 with MaMe. Each color square represents a distinct type of token.

COCO Caption Task

Question:	Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:	2 people are standing in front of a table with a variety of wine bottles and a plate of food. The man is holding a wine bottle and the woman is holding a wine glass. They are both smiling and enjoying their time together. The table is filled with wine bottles and a few wine glasses, and there is a cake on the table as well.
+ToMe:	2 people are standing at a table with a variety of wine bottles and glasses. The man is wearing a yellow shirt and the woman is wearing a black shirt. They are both holding wine glasses and appear to be tasting different wines. There are several bottles of wine on the table, some of which are placed closer to the man and others near the woman. A plate of food is also present on the table, adding to the ambiance of the wine tasting event.
+MaMe:	2 people are standing at a table with a variety of food and drinks. The man is wearing a yellow shirt and the woman is wearing a black shirt . They are both holding wine glasses and appear to be enjoying a meal together. The table is filled with several bottles of wine, a plate of food, and a fork . The scene is set in a restaurant with a window view .

Appendix

Table 8: MaMe provides the most accurate and comprehensive description, correctly identifying clothing details (yellow shirt, black shirt) and contextual elements (restaurant setting, window view). ToMe captures basic elements accurately but lacks environmental context. LLaVA omitting important visual details like clothing colors. MaMe demonstrates superior visual grounding and contextual understanding.

COCO Caption Task

Question:	Please describe this image in general. Directly provide the description, do not include prefix like "This image depicts".
LLaVA:	A man wearing a green and white ski suit is skiing down a snowy hill. He is holding ski poles in his hands and appears to be enjoying the activity. The man is the main focus of the image, and the snowy hill and trees in the background provide a picturesque winter scene.
+ToMe:	24 is the number on the man’s outfit.
+MaMe:	A man wearing a green and white ski suit is skiing down a snowy hill. He is holding ski poles in his hands and appears to be enjoying the activity. The man is wearing a hat and goggles, which are essential for protecting his eyes from the cold and snow . The scene captures the excitement and thrill of skiing in a beautiful winter landscape.

Appendix

Table 9: MaMe delivers the most comprehensive and context-aware description, accurately identifying all visual elements (ski suit, poles, hat, goggles) and providing relevant contextual reasoning about protective gear. LLaVA covers basics but omits key details like headgear, while ToMe fails entirely, producing only an unrelated fragment ("number 24").

Panel of Figure 9 — Figure 9: Images from left to right generated by Stable Diffusion v2.1, ToMeSD and MaMe+MaRe. MaMe+MaRe retains high-frequency details, making the picture more realistic.

Panel of Figure 10 — Figure 10: Images generated by the model with MaMe+MaRe. The clarity of the painting changed with the similarity threshold. A higher similarity threshold brings a more detailed and clearer painting.

Visual Token Merging

Visual token merging combines redundant image or video tokens to reduce model computation while preserving task-relevant information.

See all 36 papers

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea
Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
Oct 13, 2026Vatsal, Soumil Malhotra, Priyanshu Chahal +1Vision Transformer Efficient ViTs
Department of Mathematics, Indian Institute of Technology Roorkee
ASAP: Attention Sink Anchored Pruning
May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-Attention Efficient ViTs
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea

May 26, 2026J/K move · Enter open · S save

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

Semi Lee, Hyejin Go, Hyesong Choi

Electronic Engineering · Soongsil University · Seoul, South Korea

Token Merging Self-Attention Vision Transformer Efficient ViTs Visual Token Merging Visual Token Compression

Open paper AskPDF arXiv

The quadratic cost of self-attention in Vision Transformers (ViTs) constitutes a fundamental bottleneck for practical deployment, motivating a vibrant line of research on token reduction. Among existing approaches, token merging (ToMe) has emerged as an elegant training-free solution; yet its design rests on an unspoken premise of token equality, which contravenes the well-documented non-uniformity of self-attention and leads to information loss in high-salience tokens under aggressive compression. We address this limitation with AdaMerge, a token-merging framework based on two complementary mechanisms. First, salience-weighted similarity leverages column-wise feature-affinity centrality as a token-importance proxy and incorporates the resulting salience scores into the bipartite matching score, ensuring that pivotal tokens contribute more strongly to the merged representation. Second, adaptive merging intensity uses pre-computed layer-wise similarity statistics to dynamically modulate the per-layer reduction count in accordance with input-specific redundancy. On ImageNet-1k with ViT-B/16, AdaMerge consistently outperforms ToMe, PiToMe, and DSM across all FLOPs-matched regimes. The accuracy gap widens monotonically with compression: at the 13.4G FLOPs operating point, AdaMerge sustains a Top-1 degradation of only -1.06%, compared to -1.45% for PiToMe and -4.62% for DSM. To our knowledge, AdaMerge is the first to combine salience-weighted similarity and adaptive per-layer reduction into a single training-free token merging framework, advancing the accuracy-FLOPs Pareto frontier of ViT acceleration.

Abstract

Visual Token Merging

Explore similar work

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

ASAP: Attention Sink Anchored Pruning