ConvCue: Complementary Visual Inductive Biases for Vision-Language Models

Zixuan Lan; Shichu Sun

cs.CVSep 28, 2026

ConvCue: Complementary Visual Inductive Biases for Vision-Language Models

Authors: Zixuan Lan, Shichu Sun

Organizations: The University of Chicago · University of Chinese Academy of Sciences

Abstract

Modern vision-language models (VLMs) achieve strong performance across a broad range of multimodal tasks, yet still struggle with visual questions that require fine-grained discrimination and spatial understanding. These limitations motivate investigating whether supplementary visual representations can improve existing VLMs without replacing their native visual encoders. Pretrained convolutional networks offer a candidate feature source, motivated by their local connectivity and spatial weight sharing. We introduce CONVCUE, which augments the native visual representations of a pretrained VLM with final-stage features from a parallel, frozen pretrained CNN. A learnable adapter maps convolutional features to the native visual feature dimension, while gated cross-attention allows the original visual tokens to retrieve information from the CNN features. The enhanced tokens are passed through the original visual-to-language projector, and the model is adapted through a two-stage training procedure. We evaluate CONVCUE on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B across 13 multimodal benchmarks covering visual question answering, document and chart understanding, and multimodal reasoning. CONVCUE improves average benchmark performance over both the original models and matched two-stage fine-tuning controls on all three backbones. On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks and raises the average score from 75.00 to 78.82 relative to the matched fine-tuning control. These results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

Figures & tables

Figure 1 : Pilot experiments on Spatial QA. (a) Illustration of the example question–answer pairs. (b) Overall and per-position accuracy for different fusion strategies and trainable component configurations. A denotes the adapter and fusion module, P the original projector, and E the original visual encoder. The outlined row highlights the full visual-side adaptation setting, under which cross-attention outperforms weighted fusion by 6.14 percentage points.

Figure 2 : Overview of ConvCue . The original visual encoder processes image patches, while a frozen CNN extracts convolutional features from the same image and a trainable adapter maps them to the visual token dimension. Gated cross-attention uses the original visual tokens as queries and the adapted CNN features as keys and values. A channel-wise gate controls the residual update, followed by LayerNorm. The enhanced tokens then enter the original visual-to-language mapping and language model. The enlarged panel details the fusion mechanism; feature grids and token colors are schematic.

Figure 3 : (a) Comparison across 13 visual understanding benchmarks. ConvCue uses Qwen3-VL-4B; red labels indicate its scores. (b) Average scores across the same 13 benchmarks for the original models, direct supervised fine-tuning, and ConvCue. Parentheses show score-point changes from each original model; dashed vertical lines mark its average score.

Model	Scale	Avg.	MMBench	SEED	RealWorldQA	SQA	AI2D	HallusionBench	OCRBench	ERQA	MMMU	ChartQA	DocVQA	GQA	MMStar
Closed-source MLLMs
GPT-4.1-nano-20250414	–	58.47	62.40	–	–	65.90	68.00	36.90	71.00	–	57.60	–	–	–	47.50
GPT-4.1-mini-20250414	–	67.68	80.90	–	–	–	76.00	49.30	84.00	–	55.00	–	–	–	60.90
GPT-4V (0409)	–	70.75	79.80	73.00	68.00	84.80	78.60	43.90	65.60	–	61.70	78.50	88.40	–	56.00
Claude-3.7-Sonnet	–	72.18	79.70	74.30	55.40	90.90	82.50	55.40	70.10	35.50	71.00	92.20	94.10	–	65.10
Gemini-1.5-Pro	–	72.71	73.90	76.00	64.10	85.70	79.10	45.60	75.40	–	60.60	87.20	93.10	–	59.10

Table 1 : Comparison on visual-language benchmarks.

Method	Avg.	MMBench	SEED I	RealWorldQA	SQA I	AI2D	OCRBench	MMMU	ChartQA	DocVQA
Cambrian-1-8B	69.38	75.90	74.70	64.20	80.40	73.00	62.40	42.70	73.30	77.80
LLaVA-UHD-v3-7B	79.69	81.30	77.20	70.30	97.00	82.90	82.70	50.20	82.80	92.80
Eagle-X4-8B-Plus	72.42	75.90	76.30	66.50	84.30	76.10	62.60	43.40	80.10	86.60
Florence-VL-8B	71.34	76.20	74.90	64.20	85.90	74.20	63.40	43.70	74.70	84.90
ConvCue -4B (Ours)	84.86	92.57	82.49	75.03	95.39	84.97	89.41	60.62	87.08	96.20

Table 2 : Comparison with other baselines. Avg. denotes the unweighted mean over these nine benchmarks. Bold indicates the highest score in each column.

Figure 4 : An example asking where the red car is located. The correct answer is A (centered), while B denotes the foreground left. (a) Original image with the CNN crop boundary, ViT and CNN feature sensitivities in ConvCue , and cross-attention reading weights over CNN features. (b) Original image and direct visual-position contributions from tokens for the baseline, direct SFT, and ConvCue . Warm/cool colors in signed maps indicate positive/negative values toward A relative to B; brighter reading weights indicate stronger attention. (c) Layer-wise readouts: correct-versus-best-incorrect logit margin, its late-layer detail, correct-token vocabulary rank, and correct-option softmax score.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen3-VL-2B			Qwen3-VL-4B			LLaVA-OneVision-7B
Benchmark	Baseline	Direct SFT	ConvCue	Baseline	Direct SFT	ConvCue	Baseline	Direct SFT	ConvCue
AI2D	76.02	76.26	78.59	81.76	81.90	84.97	81.41	80.76	90.28
SEEDBench	75.90	76.02	79.94	78.74	77.99	82.49	76.54	76.56	84.94
HallusionBench	63.07	62.88	69.93	69.39	69.62	76.08	61.71	63.83	72.24
RealWorldQA	64.71	63.01	67.19	71.37	72.42	75.03	66.93	59.87	79.08
ScienceQA	87.66	88.70	89.04	92.36	95.74	95.39	95.93	96.92	97.97

Appendix

Table 3 : Comparison under matched two-stage training settings across three backbones. Baseline denotes the original model without additional training. Direct SFT follows the same training procedure as ConvCue but omits the convolutional augmentation architecture. Avg. is the unweighted mean over the 13 benchmarks. Bold indicates the highest score within each backbone, including ties.

		Latency (ms)		Peak memory (GiB)
Model	Input tokens	Baseline	ConvCue	Baseline	ConvCue
Qwen3-VL-2B	1024	83.98	91.34	5.53	5.95
	2048	104.52	112.84	5.54	5.95
	4096	194.41	202.24	7.09	7.50
	8192	543.02	551.61	15.22	15.63
Qwen3-VL-4B	1024	101.78	109.46	9.89	10.31

Appendix

Table 4: Inference efficiency with synthetic inputs. Latency is the median time to first token with inputs already on the GPU. Memory is the maximum peak allocated GPU memory across repeated measurements. Input length includes both visual and text tokens.

Explore similar work

Sep 29, 2026cs.CV

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5

NIT Silchar · MBZUAI · BML Munjal University +1

Textvqa Recent Vision-Language Models Visual Evidence Scene Understanding

Ask PDF

Jul 27, 2026cs.CV

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228,ms to 129,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.

Shaofei Lei

Vision Encoders Visual Perception Dataset Distillation Perception

Ask PDF

Aug 11, 2026cs.CV

Where To Look? : Causal Tracing of Vision Encoders in VLM

Vision-language models can describe an image with remarkable accuracy, yet a more fundamental question remains unanswered: what visual information actually drives their answers? In this work, we investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region. Extending the analysis to larger vision-language models reveals a similar pattern across models and corruption settings, suggesting that strong multimodal performance does not necessarily imply spatially localized causal representations. We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures. Together, our experiments expose a gap between seeing, using, and reasoning over visual structure, and provide a causal framework for studying how visual information is transformed, preserved, and ultimately used by modern vision-language models.

Naren Kumar S, Tirth Bhatt, Mayank Singh

LINGO Research Group, Indian Institute of Technology Gandhinagar, India

Recent Vision-Language Models Vision Encoders

Ask PDF