Seeing Through Words: Controlling Visual Retrieval Quality with Language Models

Jianglin Lu; Simon Jenni; Kushal Kafle; Jing Shi; Handong Zhao; Yun Fu

cs.CVFeb 24, 2026

Seeing Through Words: Controlling Visual Retrieval Quality with Language Models

Authors: Jianglin Lu, Simon Jenni, Kushal Kafle, Jing Shi, Handong Zhao, Yun Fu

Organizations: Adobe Research · Northeastern University

Abstract

Text-to-image retrieval is a fundamental task in vision-language learning, yet in real-world scenarios it is often challenged by short and underspecified user queries. Such queries are typically only one or two words long, rendering them semantically ambiguous, prone to collisions across diverse visual interpretations, and lacking explicit control over the quality of retrieved images. To address these issues, we propose a new paradigm of quality-controllable retrieval, which enriches short queries with contextual details while incorporating explicit notions of image quality. Our key idea is to leverage a generative language model as a query completion function, extending underspecified queries into descriptive forms that capture fine-grained visual attributes such as pose, scene, and aesthetics. We introduce a general framework that conditions query completion on discretized quality levels, derived from relevance and aesthetic scoring models, so that query enrichment is not only semantically meaningful but also quality-aware. The resulting system provides three key advantages: 1) flexibility, it is compatible with any pretrained vision-language model (VLMs) without modification; 2) transparency, enriched queries are explicitly interpretable by users; and 3) controllability, enabling retrieval results to be steered toward user-preferred quality levels. Extensive experiments demonstrate that our proposed approach significantly improves retrieval results and provides effective quality control, bridging the gap between the expressive capacity of modern VLMs and the underspecified nature of short user queries. Our code is available at https://github.com/Jianglin954/QCQC.

Figures & tables

Panel of Figure 1 — Figure 1: Aesthetic and relevance score distributions of Flickr2.4M in (a) and (b), and of MS-COCO in (c) and (d). It is worth noting that the numbers of high-quality and low-quality images are limited, which leads to the average scores of any two random sets being very close.

Panel of Table 1 — Table 1: Query completions with their retrieved images and quality scores on MS-COCO

Panel of Table 2 — Table 2: Query completions with their retrieved images and quality scores on Flickr2.4M

Quality	VLM	Aes Cond	L	L	L	M	M	M	H	H	H	Ctrl ?
Quality	VLM	Rel Cond	L	M	H	L	M	H	L	M	H	Ctrl ?
Prefix	$--$	Ave Aes	4.735	4.735	4.735	4.735	4.735	4.735	4.735	4.735	4.735	$\times$
Prefix	$--$	Ave Rel	0.350	0.350	0.350	0.350	0.350	0.350	0.350	0.350	0.350	$\times$
LLaMA3	$--$	Ave Aes	4.730	4.822	4.831	4.823	4.837	4.784	4.798	4.722	4.842	$\times$
LLaMA3	$--$	Ave Rel	0.351	0.351	0.351	0.353	0.351	0.350	0.354	0.354	0.352	$\times$
GPT-4o	$--$	Ave Aes	4.359	4.651	4.728	4.712	4.668	4.791	4.791	4.816	5.056	$\times$

Table 3: Retrieval quality of various methods on Flickr2.4M . CoCa and Blip2 are used to generate textual descriptions; L (Low), M (Medium), and H (High) indicate the quality conditions; and Ctrl specifies whether the method enables controllable retrieval over quality. For both average relevance (Ave Rel) and average aesthetics (Ave Aes), higher values indicate better retrieval quality.

Quality	Aes Cond	L	L	L	M	M	M	H	H	H	Ctrl ?
Quality	Rel Cond	L	M	H	L	M	H	L	M	H	Ctrl ?
Prefix	Ave Aes	4.817	4.817	4.817	4.817	4.817	4.817	4.817	4.817	4.817	$\times$
Prefix	Ave Rel	0.349	0.349	0.349	0.349	0.349	0.349	0.349	0.349	0.349	$\times$
LLaMA3	Ave Aes	4.903	4.891	4.855	4.916	4.875	4.880	4.871	4.858	4.911	$\times$
LLaMA3	Ave Rel	0.348	0.349	0.347	0.348	0.349	0.347	0.348	0.350	0.344	$\times$
GPT-4o	Ave Aes	4.673	4.754	4.686	4.782	4.808	4.880	4.838	5.075	5.048	$\times$

Table 4: Retrieval quality of various methods on MS-COCO , where L (Low), M (Medium), and H (High) indicate the quality conditions for retrieval, and Ctrl specifies whether the method enables controllable retrieval over image quality. For both average relevance (Ave Rel) and average aesthetics (Ave Aes), higher values indicate better retrieval quality.

Table 6

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$\mathcal{M}$		Relevance ( Red $\rightarrow$ Red )
$\mathcal{M}$		VL	L	M	H	VH
Aesthetics ( Green $\leftarrow$ Green )	VL	4.581	4.551	4.559	4.579	4.507
	VL	0.355	0.364	0.372	0.376	0.382
	L	4.870	4.792	4.784	4.748	4.718
	L	0.357	0.363	0.370	0.376	0.383
	M	4.882	4.954	4.863	4.849	4.820

Appendix

Table 7: Retrieval quality with five quality levels on CoCa.

Quality	VLM	Aes Cond	L	L	L	M	M	M	H	H	H	Ctrl ?
Quality	VLM	Rel Cond	L	M	H	L	M	H	L	M	H	Ctrl ?
Prefix	$--$	Ave Aes	4.735	4.735	4.735	4.735	4.735	4.735	4.735	4.735	4.735	$\times$
Prefix	$--$	Ave Rel	0.350	0.350	0.350	0.350	0.350	0.350	0.350	0.350	0.350	$\times$
LLaMA3	$--$	Ave Aes	4.730	4.822	4.831	4.823	4.837	4.784	4.798	4.722	4.842	$\times$
LLaMA3	$--$	Ave Rel	0.351	0.351	0.351	0.353	0.351	0.350	0.354	0.354	0.352	$\times$
GPT-4o	$--$	Ave Aes	4.359	4.651	4.728	4.712	4.668	4.791	4.791	4.816	5.056	$\times$

Appendix

Table 8: Retrieval quality of various methods on Flickr2.4M . CoCa and Blip2 are used to generate textual descriptions; L (Low), M (Medium), and H (High) indicate the quality conditions; and Ctrl specifies whether the method enables controllable retrieval over quality. For both average relevance (Ave Rel) and average aesthetics (Ave Aes), higher values indicate better retrieval quality.

Quality	Aes Cond	L	L	L	M	M	M	H	H	H	Ctrl ?
Quality	Rel Cond	L	M	H	L	M	H	L	M	H	Ctrl ?
Prefix	Ave Aes	4.817	4.817	4.817	4.817	4.817	4.817	4.817	4.817	4.817	$\times$
Prefix	Ave Rel	0.349	0.349	0.349	0.349	0.349	0.349	0.349	0.349	0.349	$\times$
LLaMA3	Ave Aes	4.903	4.891	4.855	4.916	4.875	4.880	4.871	4.858	4.911	$\times$
LLaMA3	Ave Rel	0.348	0.349	0.347	0.348	0.349	0.347	0.348	0.350	0.344	$\times$
GPT-4o	Ave Aes	4.673	4.754	4.686	4.782	4.808	4.880	4.838	5.075	5.048	$\times$

Appendix

Table 9: Retrieval quality of various methods on MS-COCO , where L (Low), M (Medium), and H (High) indicate the quality conditions for retrieval, and Ctrl specifies whether the method enables controllable retrieval over image quality. For both average relevance (Ave Rel) and average aesthetics (Ave Aes), higher values indicate better retrieval quality.

Rel: Low , Aes: Low	Rel: Medium , Aes: Medium	Rel: High , Aes: High

a wine glass next to a plate with some fish and veggies on it	a wine glass next to a plate with some meat and vegetables on it	a wine glass and three clocks all set at different times
Aes ${\color[rgb]{0,0,0.5}4.790}$ , Rel ${\color[rgb]{0,0,0.5}0.350}$	Aes ${\color[rgb]{0,0.5,0}5.209}$ , Rel ${\color[rgb]{0,0.5,0}0.376}$	Aes ${\color[rgb]{0.5,0,0}5.555}$ , Rel ${\color[rgb]{0.5,0,0}0.417}$

a toothbrush on a table with a bunch of scissors	a toothbrush that is on down on the counter	a toothbrush with a smiley face sitting on a sink
Aes ${\color[rgb]{0,0,0.5}4.149}$ , Rel ${\color[rgb]{0,0,0.5}0.345}$	Aes ${\color[rgb]{0,0.5,0}4.837}$ , Rel ${\color[rgb]{0,0.5,0}0.370}$	Aes ${\color[rgb]{0.5,0,0}5.184}$ , Rel ${\color[rgb]{0.5,0,0}0.412}$

Appendix

Table 10: Query completions with their retrieved images and quality scores on MS-COCO

Panel of Table 11 — Table 11: Query completions with their retrieved images and quality scores on Flickr2.4M

Panel of Table 12 — Table 12: Query completions with their retrieved images and quality scores on MS-COCO

Panel of Table 13 — Table 13: Some bad retrieval cases on the two datasets

Explore similar work

Aug 5, 2026

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

Contrastive vision-language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the token limit through positional embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2 Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens. Increasing caption diversity improves short-caption retrieval with diminishing returns, while paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity, retrieval direction, and description length.

Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz +2

The George Washington University

Image-Text Retrieval Vision-Language Retrieval

Ask PDF

Jul 30, 2026

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Yao Xiao, Reuben Tan, Zhen Zhu +3

University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind

Vision-Language Models Vision-Language Retrieval Visual Token Pruning Long-Context VLMs

Ask PDF

Oct 1, 2026

AiSearch: Interactive Multi-Modal Search with VLMs

Modern retrieval systems must both be automated and interactive, allowing users to search and refine results in real time. We present AiSearch, a flexible multimodal retrieval framework that leverages the zero shot capabilities of Vision Language Models (VLMs) for natural language search over images and videos. AiSearch supports interactive search refinement through user feedback to tailor results to the user's intent, and allows visual benchmarking across multiple VLMs, enabling users to select the most suitable model for their task.

Ali Koksal, Mei Chee Leong, Vicky Sintunata +2

Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore

Multimodal IR Vision-Language Models Information Retrieval Vision-Language Retrieval

Ask PDF