Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Shang-Jui Ray Kuo; Paola Cascante-Bonilla

cs.CVMar 19, 2026

Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Authors: Shang-Jui Ray Kuo, Paola Cascante-Bonilla

Organizations: Stony Brook University

Abstract

Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.

Figures & tables

Figure 1: Overview of our controlled vision encoder study. We follow a LLaVA-style VLM design where an input image is encoded by a frozen vision encoder into visual tokens, which are then processed by the LLM; this modular setup enables controlled swaps of vision encoders from different architecture families under a fixed training recipe. Right: Each marker summarizes one evaluated vision-backbone checkpoint plugged into the same VLM setting. Colors denote the backbone family; marker shapes denote the pretraining objective (classification, detection, or segmentation); marker size reflects encoder scale. Gray markers indicate configurations that exhibit collapse; arrows point to the corresponding stabilized variants after applying our stabilizations.

Visual	Encoder	IN1K	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Encoder	Size	Acc	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
ViT-S	22M	78.8	59.86	50.48	46.69	44.02	12.30	77.76	48.96	57.25
ViT-B	87M	81.1	59.63	49.19	45.63	44.51	12.07	75.90	50.57	57.17
MaxViT-T	31M	83.41	61.08	49.43	47.58	44.31	12.02	76.77	53.85	58.75
MaxViT-S	69M	84.43	59.90	48.94	41.50	44.51	11.82	76.29	51.42	57.42
MaxViT-B	119M	84.85	60.10	49.72	45.00	44.59	12.29	76.26	50.94	57.60

Table 1: Matched IN1K/224 backbone swaps (VQA). ImageNet-1K supervised vision encoders plugged into the same VLM under a strictly matched setting with 224

\times

224 inputs and

L{=}196

visual tokens. We report per-benchmark VQA scores and the weighted average VQA score. Across all tables, the best results are shown in bold, and the second best are underlined.

Visual	Encoder	IN1K	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Encoder	Size	Acc	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
ViT-S	22M	78.8	32.32	21.77	24.80	5.08	17.82	51.95
ViT-B	87M	81.1	26.66	15.41	19.12	4.14	13.92	51.36
MaxViT-T	31M	83.41	29.44	17.29	22.10	5.17	15.79	52.98
MaxViT-S	69M	84.43	25.57	14.41	17.28	4.38	13.32	51.49
MaxViT-B	119M	84.85	22.02	12.67	15.28	3.36	11.41	51.39

Table 2: Matched IN1K/224 backbone swaps (localization and overall). Same setting as 1 , reporting per-benchmark localization/grounding scores, the weighted average localization score, and the weighted overall average across all benchmarks.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K $\to$ COCO	ViTDet-B	111M	1024x1024	4096	65.83	53.61	50.58	43.75	11.60	84.46	55.96	63.00
IN1K $\to$ COCO	ViTDet-L	331M	1024x1024	4096	60.00	48.59	44.78	43.88	11.78	78.14	51.94	57.64
IN1K $\to$ COCO	ViTDet-H	662M	1024x1024	4096	64.43	52.62	48.99	43.93	11.50	84.29	55.95	61.89
IN1K $\to$ COCO	VMamba-T	30M	1333x800	4150	60.24	49.18	49.04	44.06	11.81	79.48	52.65	58.05
IN1K $\to$ COCO	VMamba-S	50M	1333x800	4150	64.87	53.98	47.73	44.24	11.93	86.47	59.21	62.78

Table 3: Dense-objective checkpoints at pretraining resolution (VQA). We compare vision encoders adapted with dense objectives, including detection-pretrained (IN1K

\rightarrow

COCO) ViTDet as a Transformer baseline and VMamba as the SSM backbone, and segmentation-pretrained (IN1K

\rightarrow

ADE20K) DeiT baselines and VMamba. All checkpoints are evaluated at their pretraining input geometries and we report per-benchmark VQA scores and the average VQA score. Because input geometry (and thus token length) differs across entries, these results reflect the impact of the dense pretraining objective rather than perfectly matched architectural swaps.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K $\to$ COCO	ViTDet-B	111M	1024x1024	4096	66.03	51.17	58.17	22.37	43.74	60.42
IN1K $\to$ COCO	ViTDet-L	331M	1024x1024	4096	24.62	13.79	17.44	4.62	13.05	51.65
IN1K $\to$ COCO	ViTDet-H	662M	1024x1024	4096	56.41	40.48	50.25	16.01	35.38	58.33
IN1K $\to$ COCO	VMamba-T	30M	1333x800	4150	29.10	16.83	19.89	3.95	14.86	52.25
IN1K $\to$ COCO	VMamba-S	50M	1333x800	4150	69.52	52.87	63.50	28.15	47.94	60.78

Table 4: Dense-objective checkpoints at pretraining resolution (localization and averages). Same settings as the VQA table, reporting localization/grounding benchmarks, the average localization score, and the overall average across all benchmarks. Checkpoints are evaluated at their pretraining input geometries; since geometry (and token length) differs across entries, the comparisons primarily isolate the effect of dense objectives rather than matched backbone architecture.

Figure 2: (Top) Grounding examples. Under the matched IN1K/224 setting, VMamba-T (blue) predicts boxes closer to ground truth (green) than ViT-S (red). (Bottom) Token–region similarity. Similarity maps between the visual feature map and the corresponding text tokens from an intermediate LLM layer show VMamba-T yields sharper, more spatially localized text–vision alignment than ViT-S, indicating better preservation and utilization of spatial information.

Visual	Encoder	Image	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Encoder	Size	Size	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
ViTDet-B	111M	1024x1024	65.83	53.61	50.58	43.75	11.60	84.46	55.96	63.00
ViTDet-L	331M	1024x1024	60.00	48.59	44.78	43.88	11.78	78.14	51.94	57.64
ViTDet-L(f)	331M	1024x1024	66.43 ( +6.43 )	53.54 ( +4.95 )	45.52 ( +0.74 )	43.76 ( -0.12 )	11.58 ( -0.20 )	84.82 ( +6.68 )	57.24 ( +5.30 )	63.55 ( +5.91 )
ViTDet-H	662M	1024x1024	64.43	52.62	48.99	43.93	11.50	84.29	55.95	61.89
ViTDet-H(f)	662M	1024x1024	66.89 ( +2.46 )	54.22 ( +1.60 )	41.01 ( -7.98 )	44.04 ( +0.11 )	11.82 ( +0.32 )	87.03 ( +2.74 )	58.08 ( +2.13 )	64.05 ( +2.16 )

Table 5: Stabilizing localization collapse: VQA benchmarks. We test two interface strategies on the collapse cases: increasing connector capacity with a stronger MLP connector (marked with (f)) and changing the input geometry to a square 512

\times

512 setting while keeping the vision checkpoint fixed. Values in parentheses denote the change relative to the corresponding collapsed variant, which is highlighted in blue.

Visual	Encoder	Image	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Encoder	Size	Size	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
ViTDet-B	111M	1024x1024	66.03	51.17	58.17	22.37	43.74	60.42
ViTDet-L	331M	1024x1024	24.62	13.79	17.44	4.62	13.05	51.65
ViTDet-L(f)	331M	1024x1024	65.73 ( +41.11 )	52.09 ( +38.30 )	58.86 ( +41.42 )	23.87 ( +19.25 )	44.58 ( +31.53 )	61.00 ( +9.35 )
ViTDet-H	662M	1024x1024	56.41	40.48	50.25	16.01	35.38	58.33
ViTDet-H(f)	662M	1024x1024	69.37 ( +12.96 )	54.27 ( +13.79 )	61.97 ( +11.72 )	26.51 ( +10.50 )	47.40 ( +12.02 )	61.81 ( +3.48 )

Table 6: Stabilizing localization collapse: localization benchmarks. Same settings as Table 5 , reporting grounding metrics and aggregate scores.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	MaxViT-T(S4)	31M	224x224	49	58.36	48.75	47.29	44.03	12.23	72.87	41.22	54.89
IN1K	MaxViT-S(S4)	69M	224x224	49	59.05	49.68	41.13	44.45	12.16	75.25	49.23	56.50
IN1K	MaxViT-B(S4)	119M	224x224	49	58.61	49.33	44.53	43.63	11.84	76.33	45.19	55.68
IN1K	MaxViT-L(S4)	212M	224x224	49	57.66	48.86	45.95	43.91	11.93	75.34	47.76	55.29
IN1K	MaxViT-T	31M	224x224	196	61.08	49.43	47.58	44.31	12.02	76.77	53.85	58.75 ( +3.86 )

Appendix

Table 7: MaxViT feature-stage ablation on VQA benchmarks. Rows marked (S4) use stage-4 features, while unmarked rows use stage-3 features. Stage 3 matches the ViT/16 token count at the same input resolution, whereas stage 4 uses fewer vision tokens.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	MaxViT-T(S4)	31M	224x224	49	13.80	9.33	10.68	4.89	8.74	48.68
IN1K	MaxViT-S(S4)	69M	224x224	49	12.63	8.60	10.07	4.27	7.96	49.97
IN1K	MaxViT-B(S4)	119M	224x224	49	12.05	8.10	9.13	4.73	7.79	49.24
IN1K	MaxViT-L(S4)	212M	224x224	49	12.26	7.98	8.82	4.11	7.52	48.87
IN1K	MaxViT-T	31M	224x224	196	29.44	17.29	22.10	5.17	15.79 ( +7.05 )	52.98 ( +4.30 )

Appendix

Table 8: MaxViT feature-stage ablation on localization benchmarks. Rows marked (S4) use stage-4 features, while unmarked rows use stage-3 features. Using the lower-token stage-4 features substantially degrades grounding and localization performance.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	MambaVision-T(S4)	32M	224x224	49	60.84	51.77	45.76	43.86	12.06	79.70	53.51	58.68
IN1K	MambaVision-T2(S4)	35M	224x224	49	60.85	51.26	44.02	44.11	12.54	79.18	49.56	58.11
IN1K	MambaVision-S(S4)	50M	224x224	49	61.34	51.32	42.71	44.52	12.35	79.77	53.49	59.00
IN1K	MambaVision-B(S4)	98M	224x224	49	61.87	52.16	45.17	43.43	12.63	80.47	52.45	59.34
IN1K	MambaVision-L(S4)	228M	224x224	49	61.52	52.35	42.86	44.16	12.46	80.32	50.53	58.80

Appendix

Table 9: MambaVision feature-stage ablation on VQA benchmarks. Rows marked (S4) use stage-4 features, while unmarked rows use stage-3 features. We report both settings to study the trade-off between token count and semantic abstraction.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	MambaVision-T(S4)	32M	224x224	49	33.88	23.69	28.72	7.40	20.04	53.49
IN1K	MambaVision-T2(S4)	35M	224x224	49	30.11	20.31	23.84	7.20	17.70	52.68
IN1K	MambaVision-S(S4)	50M	224x224	49	32.87	21.65	26.06	5.95	18.42	53.55
IN1K	MambaVision-B(S4)	98M	224x224	49	36.33	25.48	31.58	9.01	22.03	54.33
IN1K	MambaVision-L(S4)	228M	224x224	49	35.34	24.22	30.37	9.35	21.50	53.79

Appendix

Table 10: MambaVision feature-stage ablation on localization benchmarks. Rows marked (S4) use stage-4 features, while unmarked rows use stage-3 features. Stage 3 provides a more balanced trade-off for localization under a fair token-count comparison to ViT/16-style backbones.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	Vim-T	7M	224x224	196	55.83	46.93	39.39	43.82	11.67	72.91	45.69	53.40
IN1K	Vim-T-ft	7M	224x224	729	56.82	47.62	41.51	43.56	11.76	74.69	47.73	54.52
IN1K	Vim-S	26M	224x224	196	56.84	47.73	40.47	43.81	11.58	73.58	49.58	54.74
IN1K	Vim-S-ft	26M	224x224	729	57.68	48.55	45.68	43.78	11.89	74.31	45.96	55.02
IN1K	Vim-B	98M	224x224	196	62.93	52.11	45.52	44.81	12.50	78.97	55.07	60.46

Appendix

Table 11: Vim vs. VMamba on VQA benchmarks under the shared VLM training recipe. VMamba achieves stronger overall performance and is therefore used as the representative SSM-based backbone in the main paper.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	Vim-T	7M	224x224	196	21.35	12.42	14.99	3.51	11.21	47.73
IN1K	Vim-T-ft	7M	224x224	729	21.28	12.30	15.13	3.96	11.37	48.72
IN1K	Vim-S	26M	224x224	196	22.33	12.60	15.32	2.70	11.20	48.88
IN1K	Vim-S-ft	26M	224x224	729	22.99	13.09	15.01	3.57	11.80	49.21
IN1K	Vim-B	98M	224x224	196	44.62	34.10	38.54	13.17	28.56	56.17

Appendix

Table 12: Vim vs. VMamba on localization benchmarks under the shared VLM training recipe. VMamba yields stronger grounding performance overall and is therefore used as the representative SSM-based backbone in the main paper.

Figure 3: Correlation matrix of benchmark scores across all evaluated models.

Pretrained Dataset	Visual Encoder	Encoder Size	Resolution	Weighted Probing	Weighted VQA	Weighted Loc.	Weighted Overall
IN1K	ViT-S	22M	224	39.19	57.25	17.82	51.95
IN1K	ViT-B	87M	224	49.88	57.17	13.92	51.36
IN1K	MaxViT-T	31M	224	69.22	58.75	15.79	52.98
IN1K	MaxViT-S	69M	224	76.23	57.42	13.32	51.49
IN1K	MaxViT-B	119M	224	78.13	57.60	11.41	51.39
IN1K	MaxViT-L	212M	224	75.92	57.30	10.81	51.05

Appendix

Table 13: Weighted linear probing scores versus downstream weighted VQA, localization, and overall performance on the test split (%). The Weighted Probing score is the dataset-size-weighted average across probing benchmarks.

Visual	food101	cifar10	cifar100	dtd	oxford	stanford	fgvc	sun397	oxford	caltech101	birdsnap	voc2007	coco
Encoder	food101	cifar10	cifar100	dtd	pets	cars	aircraft	sun397	flowers102	caltech101	birdsnap	voc2007	multilabel
ViT-S	23.56	78.83	51.55	53.88	25.02	7.20	8.94	45.15	28.90	71.10	37.09	36.69	42.60
ViT-B	39.58	87.88	62.13	54.31	60.53	9.75	16.26	49.43	30.54	77.61	68.83	54.92	50.87
MaxViT-T	71.64	92.23	76.40	75.11	74.76	30.53	28.86	68.69	77.69	89.99	59.35	61.57	61.38
MaxViT-S	76.91	94.82	80.56	77.87	86.64	46.61	37.77	71.43	86.27	92.62	88.61	78.69	68.13
MaxViT-B	77.71	96.51	82.21	73.99	92.31	54.46	36.51	70.91	84.75	92.95	98.14	86.11	73.63

Appendix

Table 14: Per-dataset linear probing results on the test split (%). Results are obtained by training a linear classifier on frozen global image features.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN21k	ViT-T	5.7M	224x224	196	62.82	51.30	46.03	44.62	11.90	76.94	52.65	59.95
IN21k	ViT-S	22M	224x224	196	61.28	50.34	47.95	44.18	12.30	78.59	53.62	58.97
IN21k	ViT-B	87M	224x224	196	68.16	55.96	49.40	44.75	12.77	82.34	55.26	64.70
IN21k	ViT-L	304M	224x224	196	67.13	54.85	49.15	44.28	12.64	82.38	53.32	63.61
IN21k $\to$ IN1K	ViT-T	5.7M	224x224	196	61.34	50.00	45.30	44.02	11.84	77.90	53.82	58.96

Appendix

Table 15: Representative VQA comparison at matched token count. All models are evaluated in the standard

224\times 224

/ 196-token setting, or the stage-3 equivalent for hierarchical backbones, providing a compact subset of the main comparison.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN21k	ViT-T	5.7M	224x224	196	34.52	21.72	26.47	7.30	19.43	54.50
IN21k	ViT-S	22M	224x224	196	24.40	13.79	17.42	4.73	13.04	52.80
IN21k	ViT-B	87M	224x224	196	55.00	44.79	48.35	19.90	37.46	61.04
IN21k	ViT-L	304M	224x224	196	40.64	31.38	35.21	11.40	25.86	58.54
IN21k $\to$ IN1K	ViT-T	5.7M	224x224	196	31.56	19.84	23.79	8.35	18.40	53.51

Appendix

Table 16: Representative localization comparison at matched token count. All models are evaluated in the standard

224\times 224

/ 196-token setting, or the stage-3 equivalent for hierarchical backbones, providing a compact subset of the main comparison.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	MaxViT-T	31M	384x384	576	64.04	53.04	48.00	44.69	11.71	79.86	53.30	61.13
IN1K	MaxViT-S	69M	384x384	576	61.31	49.85	46.73	44.19	11.79	77.57	52.05	58.70
IN1K	MaxViT-B	119M	384x384	576	63.90	52.76	46.61	43.95	12.27	81.23	52.83	60.97
IN1K	MaxViT-L	212M	384x384	576	62.90	51.26	49.94	43.80	12.41	81.00	49.25	59.73
IN1K	MaxViT-T	31M	512x512	1024	64.25	52.84	48.14	44.40	11.84	79.24	51.93	61.08

Appendix

Table 17: Additional VQA results for selected MaxViT and MambaVision configurations beyond the matched 196-token setting. We report larger-resolution and larger-token variants for hierarchical backbones and compare them with the standard VMamba models.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	MaxViT-T	31M	384x384	576	55.01	43.06	47.75	22.30	37.97	58.02
IN1K	MaxViT-S	69M	384x384	576	25.68	14.97	18.22	4.08	13.46	52.62
IN1K	MaxViT-B	119M	384x384	576	28.18	18.45	20.61	6.08	15.98	54.92
IN1K	MaxViT-L	212M	384x384	576	23.92	13.65	17.12	5.36	13.12	53.46
IN1K	MaxViT-T	31M	512x512	1024	54.54	43.28	48.24	23.42	38.42	58.03

Appendix

Table 18: Additional localization results for selected MaxViT and MambaVision configurations beyond the matched 196-token setting. We report larger-resolution and larger-token variants for hierarchical backbones and compare them with the standard VMamba models.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
Multi-Modal $\to$ ADE20K	Perceiver	364M	512x512	256	65.62	54.63	46.25	44.26	12.25	83.79	56.62	62.92
IN22K $\to$ ADE20K	BeiT-L	451M	640x640	400	68.78	56.52	48.06	44.75	13.01	86.88	58.24	65.70
IN22K $\to$ ADE20K	BeiT-L	568M	640x640	400	69.57	57.12	51.95	44.03	13.14	86.57	58.64	66.41
IN22K+COCO $\to$ ADE20K	BeiT-L	571M	896x896	784	69.57	57.47	49.73	44.58	12.89	88.09	59.03	66.49
IN22K+COCO $\to$ ADE20K	BeiTv2-L	571M	896x896	784	70.84	58.58	49.84	44.28	13.39	88.24	61.32	67.80

Appendix

Table 19: VQA benchmarks for additional backbones with non-classification pretraining or task-adapted checkpoints, alongside VMamba transfer variants. Because these settings differ from the main controlled comparison in pretraining data and objective, they are reported separately.

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
Multi-Modal $\to$ ADE20K	Perceiver	364M	512x512	256	63.31	51.55	55.88	20.81	42.29	60.14
IN22K $\to$ ADE20K	BeiT-L	451M	640x640	400	52.99	42.75	47.81	16.94	35.22	61.61
IN22K $\to$ ADE20K	BeiT-L	568M	640x640	400	56.09	46.49	51.45	18.60	37.94	62.58
IN22K+COCO $\to$ ADE20K	BeiT-L	571M	896x896	784	55.53	46.07	49.43	18.51	37.45	62.59
IN22K+COCO $\to$ ADE20K	BeiTv2-L	571M	896x896	784	60.34	49.29	53.33	24.69	42.34	64.38

Appendix

Table 20: Localization benchmarks for additional backbones with non-classification pretraining or task-adapted checkpoints, alongside VMamba transfer variants. Because these settings differ from the main controlled comparison in pretraining data and objective, they are reported separately.

Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
CLIP-B	86M	224x224	196	72.25	58.41	52.92	51.01	31.98	85.04	60.65	69.14
CLIP-L	304M	224x224	256	74.69	59.15	53.73	52.87	37.29	86.20	60.49	71.13
DINOv2-L	304M	224x224	256	64.15	52.37	46.30	44.37	12.67	82.18	56.58	61.68
SigLIP-B16	93M	224x224	196	72.52	58.26	44.73	50.70	31.48	83.62	61.16	69.22
SigLIP-SO400M	428M	224x224	256	75.88	59.76	52.79	55.09	42.13	84.93	64.50	72.65

Appendix

Table 21: VQA benchmarks for backbones pretrained with contrastive or self-supervised objectives, along with fused DINOv2/CLIP and DINOv2/SigLIP variants. Rows marked (f) use a stronger connector for stabilization.

Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
CLIP-B	86M	224x224	196	64.85	55.57	60.05	28.40	47.19	66.19
CLIP-L	304M	224x224	256	68.05	58.26	62.79	32.70	50.66	68.38
DINOv2-L	304M	224x224	256	27.53	16.46	18.40	4.07	14.28	55.31
SigLIP-B16	93M	224x224	196	63.85	54.60	60.36	23.78	44.85	65.94
SigLIP-SO400M	428M	224x224	256	63.26	54.39	57.60	23.61	44.29	68.84

Appendix

Table 22: Localization benchmarks for backbones pretrained with contrastive or self-supervised objectives, along with fused DINOv2/CLIP and DINOv2/SigLIP variants. Rows marked (f) use a stronger connector for stabilization. Direct fusion can hurt localization, while stabilized fusion substantially improves grounding performance and yields the strongest results in this group.

Figure 4: Scaling of single-GPU VLM inference cost with image resolution at batch size 1. The x-axis shows the square input resolution, written as

R^{2}

. The six panels report (a) host-side vision latency, (b) host-side LLM latency, (c) GPU vision latency, (d) GPU LLM latency, (e) end-to-end VLM latency, and (f) peak allocated GPU memory. For each model and resolution, we run 100 warm-up iterations and report the mean over the next 100 iterations. Dotted segments ending with an

\times

indicate the first resolution that exceeds GPU memory capacity. The marker is placed at the top of each subplot to show the out-of-memory threshold rather than a measured metric value. In panel (f), it therefore marks the first failed resolution, not an observed peak-memory measurement.

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	VMamba-T	30M	224x224	196	64.99	54.02	44.96	44.62	12.22	81.70	54.58	62.07
IN1K	VMamba-S	50M	224x224	196	65.24	54.08	44.95	44.06	12.24	82.20	55.54	62.39
IN1K	VMamba-B	80M	224x224	196	64.20	53.25	47.97	43.81	12.08	80.75	54.05	61.38
IN1K	VMamba-T(f)	30M	224x224	196	65.22	53.87	44.88	44.84	12.19	81.19	56.29	62.45
IN1K	VMamba-S(f)	50M	224x224	196	65.49	54.75	47.40	43.77	12.40	81.70	55.93	62.68

Appendix

Table 23: VMamba VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	VMamba-T	30M	224x224	196	58.25	46.64	51.74	20.24	39.20	59.00
IN1K	VMamba-S	50M	224x224	196	56.48	44.27	49.88	23.09	39.17	59.27
IN1K	VMamba-B	80M	224x224	196	42.06	31.43	36.15	15.23	27.89	56.88
IN1K	VMamba-T(f)	30M	224x224	196	58.22	47.30	51.76	22.00	40.07	59.44
IN1K	VMamba-S(f)	50M	224x224	196	54.65	43.94	49.24	24.35	39.09	59.51

Appendix

Table 24: VMamba localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN21k	ViT-T	5.7M	224x224	196	62.82	51.30	46.03	44.62	11.90	76.94	52.65	59.95
IN21k	ViT-S	22M	224x224	196	61.28	50.34	47.95	44.18	12.30	78.59	53.62	58.97
IN21k	ViT-B	87M	224x224	196	68.16	55.96	49.40	44.75	12.77	82.34	55.26	64.70
IN21k	ViT-L	304M	224x224	196	67.13	54.85	49.15	44.28	12.64	82.38	53.32	63.61
IN21k $\to$ IN1K	ViT-T	5.7M	224x224	196	61.34	50.00	45.30	44.02	11.84	77.90	53.82	58.96

Appendix

Table 25: ViT VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN21k	ViT-T	5.7M	224x224	196	34.52	21.72	26.47	7.30	19.43	54.50
IN21k	ViT-S	22M	224x224	196	24.40	13.79	17.42	4.73	13.04	52.80
IN21k	ViT-B	87M	224x224	196	55.00	44.79	48.35	19.90	37.46	61.04
IN21k	ViT-L	304M	224x224	196	40.64	31.38	35.21	11.40	25.86	58.54
IN21k $\to$ IN1K	ViT-T	5.7M	224x224	196	31.56	19.84	23.79	8.35	18.40	53.51

Appendix

Table 26: ViT localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN21K	MaxViT-B	119M	224x224	196	62.67	49.98	45.92	44.08	12.38	76.46	53.58	59.88
IN21K	MaxViT-L	212M	224x224	196	65.50	52.03	47.31	44.30	12.49	80.59	53.62	62.23
IN21K	MaxViT-XL	475M	224x224	196	65.55	51.61	50.14	44.13	12.58	80.07	54.78	62.43
IN21K $\to$ IN1K	MaxViT-B	119M	384x384	576	63.44	50.92	50.41	44.38	12.71	76.72	54.53	60.71
IN21K $\to$ IN1K	MaxViT-L	212M	384x384	576	67.15	52.82	53.89	43.72	13.28	82.30	55.45	63.89

Appendix

Table 27: MaxViT VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN21K	MaxViT-B	119M	224x224	196	23.04	12.72	16.44	2.56	11.46	53.38
IN21K	MaxViT-L	212M	224x224	196	26.27	15.83	19.53	4.94	14.30	55.79
IN21K	MaxViT-XL	475M	224x224	196	24.43	13.70	18.34	4.46	13.02	55.79
IN21K $\to$ IN1K	MaxViT-B	119M	384x384	576	21.22	12.01	14.56	2.87	10.77	54.00
IN21K $\to$ IN1K	MaxViT-L	212M	384x384	576	26.46	15.92	19.36	5.79	14.70	57.28

Appendix

Table 28: MaxViT localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN21K	MambaVision-B	98M	224x224	196	49.42	42.07	41.98	42.71	10.36	58.90	42.05	47.50
IN21K	MambaVision-L	228M	224x224	196	52.06	43.27	39.10	43.12	10.64	65.22	43.49	49.87
IN21K	MambaVision-L2	242M	512x512	1024	50.80	41.95	41.57	42.79	10.50	62.97	41.93	48.63
IN21K	MambaVision-L3	740M	256x256	256	52.18	43.58	36.27	42.94	10.66	65.13	38.51	49.26
IN21K	MambaVision-L3	740M	512x512	1024	51.50	42.62	40.91	43.40	10.45	61.98	43.19	49.32

Appendix

Table 29: MambaVision VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN21K	MambaVision-B	98M	224x224	196	19.44	10.68	12.99	3.52	10.12	42.48
IN21K	MambaVision-L	228M	224x224	196	22.52	11.91	14.69	3.91	11.50	44.72
IN21K	MambaVision-L2	242M	512x512	1024	21.80	12.46	14.17	3.45	11.22	43.60
IN21K	MambaVision-L3	740M	256x256	256	21.39	11.89	13.93	3.71	11.06	44.13
IN21K	MambaVision-L3	740M	512x512	1024	21.27	11.61	13.46	3.66	10.89	44.15

Appendix

Table 30: MambaVision localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K	Vim-T	7M	224x224	196	55.83	46.93	39.39	43.82	11.67	72.91	45.69	53.40
IN1K	Vim-T-ft	7M	224x224	729	56.82	47.62	41.51	43.56	11.76	74.69	47.73	54.52
IN1K	Vim-S	26M	224x224	196	56.84	47.73	40.47	43.81	11.58	73.58	49.58	54.74
IN1K	Vim-S-ft	26M	224x224	729	57.68	48.55	45.68	43.78	11.89	74.31	45.96	55.02
IN1K	Vim-B	98M	224x224	196	62.93	52.11	45.52	44.81	12.50	78.97	55.07	60.46

Appendix

Table 31: Vim VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K	Vim-T	7M	224x224	196	21.35	12.42	14.99	3.51	11.21	47.73
IN1K	Vim-T-ft	7M	224x224	729	21.28	12.30	15.13	3.96	11.37	48.72
IN1K	Vim-S	26M	224x224	196	22.33	12.60	15.32	2.70	11.20	48.88
IN1K	Vim-S-ft	26M	224x224	729	22.99	13.09	15.01	3.57	11.80	49.21
IN1K	Vim-B	98M	224x224	196	44.62	34.10	38.54	13.17	28.56	56.17

Appendix

Table 32: Vim localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
IN1K $\to$ COCO	ViTDet-B	111M	1024x1024	4096	65.83	53.61	50.58	43.75	11.60	84.46	55.96	63.00
IN1K $\to$ COCO	ViTDet-L	331M	1024x1024	4096	60.00	48.59	44.78	43.88	11.78	78.14	51.94	57.64
IN1K $\to$ COCO	ViTDet-H	662M	1024x1024	4096	64.43	52.62	48.99	43.93	11.50	84.29	55.95	61.89
IN1K $\to$ COCO	ViTDet-B(f)	111M	1024x1024	4096	63.68	52.14	46.14	43.64	11.69	82.58	53.52	60.89
IN1K $\to$ COCO	ViTDet-L(f)	331M	1024x1024	4096	66.43	53.54	45.52	43.76	11.58	84.82	57.24	63.55

Appendix

Table 33: ViTDet VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
IN1K $\to$ COCO	ViTDet-B	111M	1024x1024	4096	66.03	51.17	58.17	22.37	43.74	60.42
IN1K $\to$ COCO	ViTDet-L	331M	1024x1024	4096	24.62	13.79	17.44	4.62	13.05	51.65
IN1K $\to$ COCO	ViTDet-H	662M	1024x1024	4096	56.41	40.48	50.25	16.01	35.38	58.33
IN1K $\to$ COCO	ViTDet-B(f)	111M	1024x1024	4096	46.03	33.49	39.40	11.72	28.26	56.51
IN1K $\to$ COCO	ViTDet-L(f)	331M	1024x1024	4096	65.73	52.09	58.86	23.87	44.58	61.00

Appendix

Table 34: ViTDet localization benchmarks

Pretrained	Visual	Encoder	Image	Vision	VQA-v2	GQA	VizWiz	TextVQA	TextVQA	POPE	TallyQA	Weighted
Dataset	Encoder	Size	Size	Token #	VQA-v2	GQA	VizWiz	+OCR	TextVQA	POPE	TallyQA	VQA
Multi-Modal $\to$ ADE20K	Perceiver	364M	512x512	256	65.62	54.63	46.25	44.26	12.25	83.79	56.62	62.92
IN22K $\to$ ADE20K	BeiT-L	451M	640x640	400	68.78	56.52	48.06	44.75	13.01	86.88	58.24	65.70
IN22K $\to$ ADE20K	BeiT-L	568M	640x640	400	69.57	57.12	51.95	44.03	13.14	86.57	58.64	66.41
IN22K+COCO $\to$ ADE20K	BeiT-L	571M	896x896	784	69.57	57.47	49.73	44.58	12.89	88.09	59.03	66.49
IN22K+COCO $\to$ ADE20K	BeiTv2-L	571M	896x896	784	70.84	58.58	49.84	44.28	13.39	88.24	61.32	67.80

Appendix

Table 35: ViT-Adapter VQA benchmarks

Pretrained	Visual	Encoder	Image	Vision	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Weighted	Weighted
Dataset	Encoder	Size	Size	Token #	RefCOCO	RefCOCO+	RefCOCOg	OCID-Ref	Loc.	Overall
Multi-Modal $\to$ ADE20K	Perceiver	364M	512x512	256	63.31	51.55	55.88	20.81	42.29	60.14
IN22K $\to$ ADE20K	BeiT-L	451M	640x640	400	52.99	42.75	47.81	16.94	35.22	61.61
IN22K $\to$ ADE20K	BeiT-L	568M	640x640	400	56.09	46.49	51.45	18.60	37.94	62.58
IN22K+COCO $\to$ ADE20K	BeiT-L	571M	896x896	784	55.53	46.07	49.43	18.51	37.45	62.59
IN22K+COCO $\to$ ADE20K	BeiTv2-L	571M	896x896	784	60.34	49.29	53.33	24.69	42.34	64.38

Appendix

Table 36: ViT-Adapter localization benchmarks

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Stateful Visual Encoders for Vision-Language Models
Jun 3, 2026Zirui Wang, Junwei Yu, Adam Yala +3Vision-Language Models Visual Representation Learning
1Voio, Inc. · 2UC Berkeley · 3UCSF
Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models
Apr 13, 2026Sameera Horawalavithana, Lauren Phillips, Ian Stewart +2Vision-Language Models VLM Reasoning
Pacific Northwest National Laboratory, Richland, WA 99352, USA
V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models
Oct 27, 2026Miso Choi, Daekeun Kim, Eunji Kim +1Memory-Augmented VLMs Efficient VLM Inference
Dept. of Computer Science and Engineering, Korea University · Amazon

Jun 3, 2026J/K move · Enter open · S save

Stateful Visual Encoders for Vision-Language Models

Zirui Wang, Junwei Yu, Adam Yala, David M. Chan+2

1Voio, Inc. · 2UC Berkeley · 3UCSF

Vision-Language Models Visual Representation Learning VLM Adaptation Temporal Representation Learning

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the language model, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the language model has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions, language model sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/

Abstract

Vision-Language Models

Explore similar work

Stateful Visual Encoders for Vision-Language Models

Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models