V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Miso Choi; Daekeun Kim; Eunji Kim; Jungbeom Lee

cs.LGOct 27, 2026

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Authors: Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee

Organizations: Dept. of Computer Science and Engineering, Korea University · Amazon

Abstract

Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.

Figures & tables

Figure 1: Overview of V-Lumen . V-Lumen retrieves visual memory embeddings using spatially aggregated and text-conditioned hash keys, then fuses them into the frozen VLM through gated residual updates.

Rows / cell	Table params	Memory	Lookup total (ms)	Gather only (ms)
16,384 (0.2 $\times$ )	8,433,472	16.9 MB	11.22 $\pm$ 0.15 (1.005 $\times$ )	0.463 (0.983 $\times$ )
81,920 (1 $\times$ )	41,976,448	84.0 MB	11.16 $\pm$ 0.11 (1.000 $\times$ )	0.471 (1.000 $\times$ )
327,680 (4 $\times$ )	167,825,792	335.7 MB	11.16 $\pm$ 0.11 (1.000 $\times$ )	0.462 (0.981 $\times$ )
655,360 (8 $\times$ )	335,605,248	671.2 MB	11.23 $\pm$ 0.21 (1.006 $\times$ )	0.491 (1.042 $\times$ )

Table 1: Lookup latency versus memory table size for one fusion layer (64 visual tokens, bf16, RTX 6000 Ada; mean over 50 runs). Values in parentheses are relative to the default setting (1

\times

Setting	ChartMus.	Frieda	VisRes	MMMU	MMMU-Pro	Avg	Avg $\Delta$
Vanilla	1.69	2.27	5.00	1.63	0.30	2.18	–
V-Lumen -only	16.70 $\pm$ 0.79	4.77 $\pm$ 0.23	24.50 $\pm$ 2.07	33.38 $\pm$ 0.95	20.50 $\pm$ 0.82	19.97 $\pm$ 0.47	+17.79 $\pm$ 0.47
LoRA-only	17.87 $\pm$ 0.28	5.53 $\pm$ 0.13	22.03 $\pm$ 0.50	41.87 $\pm$ 1.47	26.50 $\pm$ 1.93	22.76 $\pm$ 0.85	+20.58 $\pm$ 0.85
LoRA + V-Lumen	18.24 $\pm$ 0.41	5.61 $\pm$ 0.35	21.70 $\pm$ 2.39	41.91 $\pm$ 1.30	29.20 $\pm$ 0.66	23.33 $\pm$ 0.02	+21.15 $\pm$ 0.02
MoE-only	13.48 $\pm$ 0.90	4.85 $\pm$ 0.35	23.53 $\pm$ 3.87	33.21 $\pm$ 2.22	20.10 $\pm$ 0.26	19.03 $\pm$ 0.95	+16.85 $\pm$ 0.95
MoE + V-Lumen	14.19 $\pm$ 0.49	4.70 $\pm$ 1.02	25.07 $\pm$ 1.43	32.54 $\pm$ 1.17	19.00 $\pm$ 2.59	19.10 $\pm$ 0.91	+16.92 $\pm$ 0.91

Table 2: Accuracy (%) on five reasoning-intensive vision-language benchmarks, reported as mean

\pm

std over three seeds. Avg

\Delta

is the average gain over the vanilla backbone (pp). Within each LoRA and MoE pair, both settings are matched in total parameter budget (Appendix G ): LoRA-only and MoE-only correspond to LoRA(

1{+}k

) and MoE(

1{+}k

), and the

+

V-Lumen variants combine LoRA(

k

) or MoE(

k

) with V-Lumen (1), where

k=0.5

. Training schedules are listed in Appendix E.2 .

Benchmark	Vanilla	+ V-Lumen
TextVQA Singh et al. (2019)	11.00	37.17 $\pm$ 3.42 (+26.17)
OCR-VQA Mishra et al. (2019)	13.80	47.67 $\pm$ 0.76 (+33.87)
VizWiz-VQA Gurari et al. (2018)	2.00	9.97 $\pm$ 1.84 (+7.97)

Table 3: Zero-shot accuracy (%) on benchmarks outside the training curriculum, using the frozen Qwen3-VL-2B backbone (mean

\pm

std over three seeds). No in-domain few-shot samples are used, and none of the benchmarks shares image sources with our training data (Appendix F ). Gains over the vanilla backbone are shown in green .

Figure 2: Layer-wise logit-lens analysis on VisRes (MoE, left) and MMMU-Pro (LoRA, right), comparing each adapter-only model with its + V-Lumen counterpart (MoE: budget-matched pair of Tab. 2 ; LoRA: rank-matched LoRA(0.5) pair of Tab. 5 ). Top: JSD

(p_{\ell},p_{\text{final}})

to the model’s own final prediction. Bottom: top-1 accuracy of the gold answer. Layers 12–27 are shown; below layer 19, both models stay at the maximum JSD and zero top-1 accuracy. Mean

\pm

1 std over three seeds.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Mid-training steps	SFT steps
V-Lumen -only (frozen vanilla backbone)	10,000	2,000
LoRA-only / LoRA + V-Lumen	2,000	1,000
MoE-only / MoE + V-Lumen	2,000	1,000

Appendix

Table 4: Training steps for each setting.

Setting	ChartMus.	Frieda	VisRes	MMMU	MMMU-Pro	Avg	Avg $\Delta$
Vanilla	1.69	2.27	5.00	1.63	0.30	2.18	–
LoRA(0.5)	18.24 $\pm$ 0.70	5.53 $\pm$ 0.69	23.63 $\pm$ 1.40	43.37 $\pm$ 0.82	28.20 $\pm$ 1.21	23.79 $\pm$ 0.33	+21.62 $\pm$ 0.33
LoRA(0.5) + V-Lumen	18.24 $\pm$ 0.41	5.61 $\pm$ 0.35	21.70 $\pm$ 2.39	41.91 $\pm$ 1.30	29.20 $\pm$ 0.66	23.33 $\pm$ 0.02	+21.15 $\pm$ 0.02

Appendix

Table 5: Accuracy (%) under LoRA Design 1 (rank-matched) on Qwen3-VL-2B, mean

\pm

std over three seeds. Both settings share the same LoRA(0.5) adapter (rank 107; Tab. 6 ), and V-Lumen is added on top. Avg

\Delta

is the average gain over the vanilla backbone (pp). Training schedules follow Tab. 4 .

Configuration	LoRA rank	LoRA params	V-Lumen params	Total trainable
LoRA(0.5)	107	42,950,656	–	42,950,656
LoRA(0.5) + V-Lumen	107	42,950,656	86,132,038	129,082,694

Appendix

Table 6: LoRA Design 1 (rank-matched). The LoRA component is shared, and V-Lumen is added on top.

Configuration	LoRA rank	LoRA params	V-Lumen params	Total trainable	Ratio
LoRA(0.5) + V-Lumen	107	42,950,656	86,132,038	129,082,694	1.000 $\times$
LoRA(1.5)	321	128,851,968	–	128,851,968	0.998 $\times$

Appendix

Table 7: LoRA Design 2 (total-budget-matched). The LoRA-only baseline is scaled to match the combined trainable parameters of LoRA(0.5) + V-Lumen .

Configuration	Accessible params	Activated params / token
MoE only	131,170,304	4,358,144
MoE + V-Lumen	129,574,214 (0.988 $\times$ )	4,334,592 (0.995 $\times$ )
MoE component	43,442,176	2,154,496
V-Lumen component	86,132,038	2,180,096

Appendix

Table 8: MoE budget matching. Accessible parameters include all parameters the model can route to; activated parameters are those used per token.

Configuration	# experts	Expert hidden dim	V-Lumen memory table
MoE only	31	344	–
MoE + V-Lumen	21	168	32 cells $\times$ 81,920 rows $\times$ 32 dims

Appendix

Table 9: Per-side MoE configurations. Both use top-1 routing at decoder layers 2 and 8 without a shared expert. The V-Lumen memory table consists of 32 cells, one for each combination of fusion layer (2),

n

-gram order (2), and head (8).

Setting	Prefill	$\Delta$ prefill	Decode / token	$\Delta$ decode
Vanilla	66.0	–	31.90	–
V-Lumen	91.3	+25.3 (+38%)	31.93	+0.03
LoRA(0.5)	85.2	–	49.07	–
LoRA(1.5)	85.4	–	49.29	–
LoRA(0.5) + V-Lumen	110.2	+24.8 (+29%)	49.03	$-$ 0.26
MoE(1.5)	84.2	–	31.96	–

Appendix

Table 10: Prefill and per-token decode latency (ms).

\Delta

is the change from adding V-Lumen to the corresponding budget-matched baseline (Vanilla, LoRA(1.5), or MoE(1.5)). Relative to the rank-matched LoRA(0.5), LoRA(0.5) + V-Lumen adds

+25.0

ms prefill and

-0.04

ms decode.

Component	V-Lumen	LoRA(0.5) + V-Lumen	MoE(0.5) + V-Lumen	MoE(1.5)
Quantizer (patch $\rightarrow$ code)	0.3	0.3	0.3	–
TCH bins (PRH / DRH)	1.1	1.1	1.1	–
Key construction	18.2	17.3	16.6	–
Gather (table read)	1.8	1.8	1.8	–
Fusion (gate, value projection, norm)	3.2	3.2	3.1	–
V-Lumen total	24.6	23.7	22.8	–

Appendix

Table 11: Breakdown of V-Lumen prefill overhead (ms per sample, summed over both fusion layers).

Setting	Memory-table params	After load	Peak inference
Vanilla	–	4.256	4.311
V-Lumen	83,952,896 (335.8 MB, fp32)	4.603	4.658
LoRA(0.5)	–	4.428	4.483
LoRA(1.5)	–	4.772	4.829
LoRA(0.5) + V-Lumen	83,952,896 (335.8 MB, fp32)	4.775	4.830
MoE	–	4.781	4.836

Appendix

Table 12: GPU memory overhead (GB). Memory-table parameters are summed over both fusion layers.

Codebook	ChartMus.	Frieda	VisRes	MMMU	MMMU-Pro	Avg $\Delta$
Vanilla	1.69	2.27	5.00	1.63	0.30	–
LaVIT (16,384 codes)	16.70 $\pm$ 0.79	4.77 $\pm$ 0.23	24.50 $\pm$ 2.07	33.38 $\pm$ 0.95	20.50 $\pm$ 0.82	+17.79 $\pm$ 0.47
SEED (8,192 codes)	16.33 $\pm$ 0.52	5.83 $\pm$ 0.66	27.17 $\pm$ 1.01	34.00 $\pm$ 2.63	20.10 $\pm$ 0.44	+18.51 $\pm$ 1.00

Appendix

Table 13: V-Lumen -only accuracy (%) with different frozen codebooks, mean

\pm

std over three seeds. Avg

\Delta

is the average gain over the frozen backbone (pp). The LaVIT row is the V-Lumen -only setting of Tab. 2 .

Axis	Setting	Avg $\Delta$
— (default)	layers (2, 8), $K{=}8$ , 8 heads, orders $\{2,3\}$	+17.79 $\pm$ 0.47
Fusion layers	(2, 8) $\rightarrow$ (4, 10)	+17.20 $\pm$ 0.35
Bins $K$	8 $\rightarrow$ 4	+18.33 $\pm$ 0.57
	8 $\rightarrow$ 16	+17.81 $\pm$ 0.27
Memory heads	8 $\rightarrow$ 4	+18.64 $\pm$ 0.77
	8 $\rightarrow$ 16	+17.99 $\pm$ 0.83

Appendix

Table 14: Single-axis sensitivity of V-Lumen -only. Avg

\Delta

is the average gain over the frozen backbone across the five benchmarks (pp, mean

\pm

std over three seeds). Changing the number of heads keeps the total memory dimension fixed (512), so

d_{\text{head}}

becomes 64 with 4 heads and 16 with 16 heads while the table size stays about 86M parameters; this differs from the width axis in Appendix J , where the total memory dimension changes. Adding a third

n

-gram order (n=4) keeps

d_{\text{head}}=32

and adds 16 memory cells.

Setting	ChartMus.	Frieda	VisRes	MMMU	MMMU-Pro	Avg $\Delta$
Vanilla	5.41	1.36	0.00	2.26	1.30	–
V-Lumen -only	14.49 $\pm$ 0.86	4.62 $\pm$ 0.35	22.27 $\pm$ 0.06	29.11 $\pm$ 4.79	14.33 $\pm$ 3.95	+14.90 $\pm$ 1.82
LoRA(0.5)	18.62 $\pm$ 0.28	6.59 $\pm$ 0.23	21.40 $\pm$ 0.61	45.17 $\pm$ 0.88	27.57 $\pm$ 0.65	+21.80 $\pm$ 0.28
LoRA(0.5)+ V-Lumen	18.32 $\pm$ 0.17	5.23 $\pm$ 0.39	21.80 $\pm$ 0.72	44.63 $\pm$ 2.07	26.60 $\pm$ 0.87	+21.25 $\pm$ 0.49
MoE(1.5)	15.47 $\pm$ 0.36	6.21 $\pm$ 1.05	22.53 $\pm$ 0.40	34.92 $\pm$ 1.05	18.73 $\pm$ 0.81	+17.51 $\pm$ 0.25
MoE(0.5)+ V-Lumen	16.33 $\pm$ 0.92	5.68 $\pm$ 0.99	21.70 $\pm$ 0.75	33.29 $\pm$ 0.96	19.27 $\pm$ 1.76	+17.19 $\pm$ 0.36

Appendix

Table 15: Accuracy (%) on InternVL3.5-2B, mean

\pm

std over three seeds. Avg

\Delta

is the average gain over the frozen backbone (pp). The MoE pair is matched in total parameter budget; the LoRA pair is rank-matched (LoRA(0.5) with and without V-Lumen ; Design 1 in Appendix G ).

		Perturbed patches
Variant	Unperturbed	50%	100%
Rolling hash only	+18.67 $\pm$ 0.70	+14.01 $\pm$ 1.38 ( $-$ 4.67)	0 +3.98 $\pm$ 0.21 ( $-$ 14.69)
SAH	+18.19 $\pm$ 0.90	+15.09 $\pm$ 0.29 ( $-$ 3.10)	+13.94 $\pm$ 0.68 ( $-$ 4.25)
SAH + PRH	+18.51 $\pm$ 0.02	+15.03 $\pm$ 1.36 ( $-$ 3.48)	+13.83 $\pm$ 0.96 ( $-$ 4.68)
SAH + DRH	+18.32 $\pm$ 0.41	+15.14 $\pm$ 0.44 ( $-$ 3.17)	+14.55 $\pm$ 1.00 ( $-$ 3.77)
SAH + PRH + DRH	+17.79 $\pm$ 0.47	+16.19 $\pm$ 0.31 ( $-$ 1.61)	+15.62 $\pm$ 0.11 ( $-$ 2.17)

Appendix

Table 16: Component ablation of V-Lumen on the frozen vanilla backbone. Values are the average gain over the frozen backbone across five benchmarks (pp, mean

\pm

std over three seeds). Perturbed columns discretize 50% or 100% of the image patches with a re-drawn projection at evaluation time; the drop from the unperturbed setting is shown in parentheses, and the smallest drop at each level is bold .

Fusion layers	ChartMus.	Frieda	VisRes	MMMU	MMMU-Pro	Avg $\Delta$
Vanilla	1.69	2.27	5.00	1.63	0.30	–
(2, 6)	15.92 $\pm$ 0.34	5.53 $\pm$ 1.07	25.83 $\pm$ 1.45	34.42 $\pm$ 1.43	19.10 $\pm$ 0.92	+17.98 $\pm$ 0.28
(2, 8)	16.70 $\pm$ 0.79	4.77 $\pm$ 0.23	24.50 $\pm$ 2.07	33.38 $\pm$ 0.95	20.50 $\pm$ 0.82	+17.79 $\pm$ 0.47
(2, 10)	16.07 $\pm$ 0.33	5.30 $\pm$ 0.13	26.13 $\pm$ 1.18	27.44 $\pm$ 0.51	16.53 $\pm$ 1.72	+16.12 $\pm$ 0.31
(2, 15)	16.03 $\pm$ 0.13	5.30 $\pm$ 0.47	21.67 $\pm$ 1.52	29.57 $\pm$ 0.92	17.67 $\pm$ 0.91	+15.87 $\pm$ 0.14
(3, 15)	14.86 $\pm$ 0.79	4.92 $\pm$ 0.47	23.57 $\pm$ 0.51	26.68 $\pm$ 1.70	16.27 $\pm$ 1.48	+15.08 $\pm$ 0.56

Appendix

Table 17: Fusion-layer ablation of V-Lumen -only on the frozen backbone (accuracy %, mean

\pm

std over three seeds). Avg

\Delta

denotes the average gain over the frozen backbone (pp). The default configuration is (2, 8).

Axis	Setting	V-Lumen params	Read / token	Avg $\Delta$
Depth	$0.5\times$ rows (40,960)	44.2M	$1\times$	+18.14 $\pm$ 0.95
	$1\times$ rows (81,920) ∗	86.1M	$1\times$	+17.79 $\pm$ 0.47
	$2\times$ rows (163,840)	170.0M	$1\times$	+18.36 $\pm$ 0.84
	$4\times$ rows (327,680)	337.8M	$1\times$	+18.33 $\pm$ 0.78
	$8\times$ rows (655,360)	673.4M	$1\times$	+17.79 $\pm$ 0.46
Width	$d_{\text{head}}=8$	21.6M	$0.25\times$	+16.44 $\pm$ 0.46

Appendix

Table 18: Capacity scaling of V-Lumen -only on the frozen backbone. Read / token is the amount of memory read per visual token relative to the default. Avg

\Delta

is the average gain over the frozen backbone across five benchmarks (pp, mean

\pm

std over three seeds). ∗ Default configuration (Tab. 2 ).

Figure 3: Layer-wise

\mathrm{JSD}(p_{\ell},p_{\text{final}})

on the two open-ended probe pairs, ChartMuseum (MoE, left) and Frieda (LoRA, right), comparing each adapter-only model with its + V-Lumen counterpart (MoE: budget-matched; LoRA: rank-matched LoRA(0.5)). Mean

\pm

1 std over three seeds.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Oct 1, 2026

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.

Xinye Zhao, Yunkai Dang, Yunchen Wu +1

School of Intelligence Science and Technology, Nanjing University

Vision-Language Models Efficient VLM Inference Language Model Scaling Laws

Ask PDF

Jul 30, 2026

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Yao Xiao, Reuben Tan, Zhen Zhu +3

University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind

Vision-Language Models Vision-Language Retrieval Visual Token Pruning Long-Context VLMs

Ask PDF

Jun 10, 2026

AVIS: Adaptive Test-Time Scaling for Vision-Language Models

Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free

O(N)

key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.

Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8

AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1

Visual Token Pruning Test-Time Adaptation of VLMs Efficient VLM Inference Test-Time Scaling for VLMs Test-Time Scaling VLM Reasoning+2

Ask PDF

Abstract

Memory-Augmented VLMs

Explore similar work

Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

AVIS: Adaptive Test-Time Scaling for Vision-Language Models