Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

Ying Liu; Yudong Han; Kean Shi; Liyuan Pan

cs.CVFeb 28, 2026

Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

Authors: Ying Liu, Yudong Han, Kean Shi, Liyuan Pan

Organizations: Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.

Figures & tables

Figure 1: Comparison of different strategies for enhancing visual representations in MLLMs. (a) Single- or Multi- vision encoders (VE) rely on representations from fixed layers. (b) Multi-layer fusion methods aggregate features from multiple internal layers. (c) Mema (denoted as A) introduces dynamic, explicit cross-layer interaction via a stateful memory, enabling progressive refinement across layers.

Panel of Figure 2 — Figure 2: Layer-wise visualization of attention patterns and feature embedding distributions in vision encoder. (a) “Text-Guided” fusion uses textual representations as queries to select visual features from different layers, while “Last-layer-Guide” fusion employs final-layer visual representations as queries. Attention analysis reveals both “Text-Guided” and “Last-layer-Guided” visual feature aggregation consistently prioritize deeper encoder layers over shallow ones. (b) t-SNE visualization of visual features from different layers of the vision encoder. Features from shallow and intermediate layers exhibit distinct distributions compared to deeper layers, indicating significant semantic misalignment across layers.

Figure 3: Overview of Mema. Mema maintains a stateful memory that accumulates hierarchical visual representations via HME, and injects it back into token representations through MVA.

Figure 4: Illustration of (a) HME and (b) MVA.

Mema trained on LLaVA-v1.5-7B
Model	Vision Encoder	LLM	TextVQA	GQA	MME P	MME C	SQA	POPE	MMB	MMB CN	SEED I
LLaVA-v1.5	CLIP ViT-L/14-336	Vicuna-7B	58.2	62.0	1510.7	355.7	66.8	85.9	64.3	58.3	66.0
+ Mema	CLIP ViT-L/14-336	Vicuna-7B	58.2	61.8	1520.6	372.2	70.1	86.7	64.7	58.3	66.0
LLaVA-v1.5	OpenCLIP ViT-L/14	Vicuna-7B	53.9	58.5	1332.0	321.1	71.0	84.1	60.8	52.3	64.2
+ Mema	OpenCLIP ViT-L/14	Vicuna-7B	54.1	60.9	1418.1	311.0	70.8	85.3	65.5	57.8	64.4
MGM	CLIP ViT-L/14-336 + ConvNeXt-L	Vicuna-7B	65.2	62.6	1523.0	316.0	71.5	87.0	69.3	57.4	69.4

Table 1: Plug-and-play transferability of Mema across different multimodal backbones. Mema is trained once on LLaVA-v1.5 [ 31 ] and directly inserted into other models with compatible hidden dimensions. ∗ indicates a CLIP ViT-L/14-336 [ 37 ] vision tower further fine-tuned on ShareGPT4V [ 5 ] data.

Method	Trainable (M)	Total (B)	FLOPs (T)	Data (K)
+ Full FT	6628.8	6.932	457.5	665
+ Proj Only	21.0	6.932	307.4	558
+ Mema	11.0	6.943	315.9	20

Table 2: Parameter and training cost comparison of Mema on LLaVA-1.5-7B [ 31 ] . “Trainable” denotes the number of trainable parameters, “Total” refers to the overall model size, “FLOPs” indicates the forward computation per batch, and “Data” denotes the number of training samples. “+ Full FT” denotes fine-tuning both the LLM and the multimodal projector, while “+ Proj Only” denotes fine-tuning only the multimodal projector. Total training FLOPs can be computed as FLOPs

\times

Data.

Model	Vision Encoder	LLM	TextVQA	GQA	MMMU	VQAv2	MME P	SQA	POPE	MMB	MM-Vet
Mipha	SigLIP-SO400M/14-384	Phi-1.5-1.3B	45.6	62.7	29.9	77.5	1203.1	58.3	86.9	57.7	23.5
+ Mema	SigLIP-SO400M/14-384	Phi-1.5-1.3B	49.3	62.5	29.3	79.2	1285.5	61.2	86.9	60.7	26.7
Mipha	SigLIP-SO400M/14-384	Phi-2-2.7B	56.6	63.9	25.2	81.3	1488.9	70.9	86.7	69.7	31.3
+ Mema	SigLIP-SO400M/14-384	Phi-2-2.7B	56.2	64.0	26.8	81.3	1494.7	74.6	87.8	70.0	33.1
LLaVA-v1.5	SigLIP-B/16-224	Vicuna-7B	53.4	61.0	33.8	76.0	1427.4	70.2	84.2	63.1	26.7
+ Mema	SigLIP-B/16-224	Vicuna-7B	52.4	61.0	34.4	75.0	1451.6	70.3	83.9	62.0	27.1

Table 3: Generalization of Mema across different vision encoders with heterogeneous feature dimensions.

Model	MME P	SQA	MMB	POPE	Avg
LLaVA-1.5-7B	1510.7	66.8	64.3	85.9	73.1
Dense Connector [ 43 ]	–	69.5	66.8	86.6	–
MMFuser [ 4 ]	1479.7	68.7	67.5	86.3	74.1
Instruction-Guided Fusion [ 26 ]	1519.8	70.2	66.9	87.8	75.2
TGIF [ 30 ]	–	70.1	66.4	87.9	–
+ Mema	1520.6	70.1	64.7	86.7	74.4

Table 4: Comparison with representative multi-layer visual fusion methods.

Panel of Figure 5 — Figure 5: Visualization analysis of behavior of our method. (a) Cross-layer similarity difference bewteen Mema and vanilla LLaVA-1.5-7B across vision layers. Red indicates increased similarity under Mema, while blue indicates decreased similarity. Mema reshapes the interaction pattern across depth. (b) The cosine similarity between the memory state and the current-layer features, the earliest-layer features, and the mean of features from all preceding layers, indicating progressive alignment and effective cross-layer information aggregation.

Method	TextVQA	MME P	MME C	MMB	Avg
LLaVA-v1.5-7B	58.20	1510.7	355.7	64.30	60.63
Last layer	58.00	1517.9	353.2	64.30	60.60
Layers {8,16,24}	58.00	1516.2	350.4	64.50	60.53
Layers {4,8,12,16,20,24}	58.00	1514.5	343.6	64.40	60.28
Odd Layers	58.20	1500.5	357.5	65.00	60.73
All Layers (Ours)	58.20	1520.6	372.2	64.70	61.35

Table 5: Effect of progressive cross-layer interaction. MVA is applied at the last layer (Last layer), selected layer subsets, odd-numbered layers (Odd Layers), or all layers (Ours).

Method	TextVQA	MME P	MME C	MMB	Avg
LLaVA-v1.5-7B	58.2	1510.7	355.7	64.3	60.63
+ Mema w/o Text	58.4	1489.4	371.1	64.2	60.88
+ Mema w/o HME	58.2	1510.0	355.7	64.6	60.70
+ Mema w/o MVA	29.9	1513.0	355.7	20.5	42.65
+ Mema	58.2	1520.6	372.2	64.7	61.35

Table 6: Ablation study of different components in Mema.

$r$	$\lambda$	MME P	MME C	SQA	POPE	MMB	Avg
LLaVA-v1.5-7B		1,510.7	355.7	66.8	85.9	64.3	67.4
1	0.1	1,494.1	382.6	70.0	86.5	64.3	68.7
2	0.1	1,504.0	341.1	70.3	86.7	64.4	67.9
4	0.05	1,484.0	357.5	70.1	86.9	64.8	68.1
4	0.1	1,520.6	372.2	70.1	86.7	64.7	68.8
4	0.15	1,497.8	357.1	70.2	86.6	64.9	68.2

Table 7: Sensitivity analysis of Mema with respect to the reduction ratio

r

and the alignment weight

\lambda

. The blue row denotes our default setting. We report performance across various benchmarks, with the best results in bold.

Data Size	MME P	MME C	MMB	POPE	TextVQA
LLaVA-v1.5-7B	1510.7	355.7	64.3	85.9	58.2
5K	1506.5	349.6	64.6	86.8	58.0
10K	1497.2	355.7	64.6	86.7	58.1
20K	1520.6	372.2	64.7	86.7	58.2
30K	1487.0	364.6	64.3	86.8	58.3
40K	1506.1	365.0	64.3	87.0	58.1

Table 8: Effect of training data scale for Mema. We vary the number of training samples while keeping all other settings fixed. The blue row indicates our default setting.

Figure 6: Effect of additional training data on the LLaVA-v1.5-7B. “+ Full FT” denotes fine-tuning both the LLM and the multimodal projector, while “+ Proj Only” denotes fine-tuning only the multimodal projector.

Figure 7: Qualitative comparison between LLaVA-v1.5-7B and Mema across diverse tasks (examples drawn from MME [ 32 ] , POPE [ 28 ] , and TextVQA [ 38 ] ), including hallucination, commonsense reasoning, location, and OCR.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Oct 8, 2026Yicheng Xue, Han Wu, Jufeng Yang +4VLM Reasoning Efficient Multimodal Inference
City University of Hong Kong · Zhejiang University · Peking University +2
Dual-Latent Memory Routing for Vision-Language Reasoning
Sep 2, 2026Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao +1Memory-Augmented VLMs VLM Reasoning
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Mixture of Layers: Dynamic Layer Routing for Visual Reasoning
Oct 7, 2026Jeonghwan Kim, Sofia Stoica, Jiwan Chung +5VLM Reasoning Large Vision-Language Models
University of Illinois Urbana-Champaign · Yonsei University · Meta Reality Labs

Oct 8, 2026J/K move · Enter open · S save

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Yicheng Xue, Han Wu, Jufeng Yang, Minjing Dong+3

City University of Hong Kong · Zhejiang University · Peking University+2

VLM Reasoning Efficient Multimodal Inference Memory-Augmented VLMs Visual Token Compression Visual Token Pruning

Open paper AskPDF arXiv

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

Abstract

Memory-Augmented VLMs

Explore similar work

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Dual-Latent Memory Routing for Vision-Language Reasoning

Mixture of Layers: Dynamic Layer Routing for Visual Reasoning