Query-adaptive Recurrent Tokenization

Bavesh Balaji; Pranav Gopinath; Pouya Sadeghi; Sirisha Rambhatla

cs.CVOct 16, 2026

Query-adaptive Recurrent Tokenization

Authors: Bavesh Balaji, Pranav Gopinath, Pouya Sadeghi, Sirisha Rambhatla

Organizations: Critical ML Lab, University of Waterloo

Abstract

Visual information worth extracting from an image and the token budget it warrants depend on the text query. Yet modern vision and vision-language models largely encode images independently of the query and allocate a fixed visual token budget to every input. We introduce the Query-Adaptive Recurrent Tokenizer (QART), a backbone-agnostic module that makes visual representations both query-steerable and budget-adaptive. QART steers the visual representation toward query-relevant evidence while adapting its token budget to the amount of information required. Its key principle is simple. Additional tokens are allocated until the visual features relevant to the query are represented with sufficient fidelity. Starting from frozen encoder features, QART recurrently constructs a compact representation and uses reconstruction quality over the query-relevant visual support to control token allocation. On Visual7W and VQAv2, QART achieves 81.96% and 76.37% accuracy using only about 29 visual tokens, reducing token counts by 20-25x relative to dense VLM inputs. It further reaches 92.0% P@1 on CORE and 36.3% PR-AUC on PODS, outperforming SteerViT on PODS by 8.4%. Together, these results show that visual representations need not be fixed in either content or capacity and can instead adapt to what each query requires.

Figures & tables

Figure 1: QART combines query steerability with budget adaptivity. Fixed patch tokenizers use the same representation and token budget for every query. Adaptive tokenizers change how many tokens are allocated, while steerable methods change what visual information is emphasized. QART does both, steering the representation toward the query while adapting its capacity to the required visual evidence. The clear regions represent the tokenizers’ focus.

Figure 2: Overview of QART. Given an image and a text query, frozen encoders provide visual and linguistic features $\mathbf{V}$ and $\mathbf{T}$ respectively. A relevance module $H_{\phi}$ selects query-relevant patches $\mathbf{V}_{\mathcal{A}}$ ; QART then recurrently refines a small set of query-conditioned slots $\mathbf{S}$ by passing them along with $\mathbf{V}_{\mathcal{A}}$ and $\mathbf{T}$ to the shared transformer $F_{\theta}$ . After each round $r$ , a shared decoder $D_{\psi}$ reconstructs the visual features, and reconstruction quality determines whether more slots are added or recurrence halts. Only slot states propagate across rounds.

Panel of Figure 3 — Figure 3: QART steers visual representations while preserving transferable visual information. (a) On CORE, QART redirects visual representations toward queried content. (b) The same representation transfers zero-shot to natural images on GeneCIS without dataset-specific tuning. (c) Replacing the sample-specific query with fixed conditioning substantially degrades Winoground performance. (d) On PODS, coarse conditioning preserves fine-grained instance information beyond the prompt.

Table 1: Reasoning and grounding under aggressive visual compression. QART preserves strong downstream performance with approximately 29 visual tokens on both VQAv2 and Visual7W. Visual7W additionally reports the average token count processed across language-model layers for methods that prune within the LLM.

Panel of Figure 4 — Figure 4: Recurrent refinement and inference efficiency. (a) QART’s recurrence refines representations even upon disabling token growth. (b–c) The compact visual sequence increasingly improves throughput and language-model prefill as batch size grows.

Query-steered support	Query-aware budget	P@1 $\uparrow$	Mean $K$ $\downarrow$	Mean rounds $\downarrow$
✓	✓	92.01	17.25	4.25
✓	✗	92.08	32.00	6.00
✗	✗	79.7	32	6.00

Table 2: Ablation of the two roles of query conditioning in QART on CORE. Query conditioning helps steer the visual support used during recurrence and independently control the evidence used for slot growth and halting.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Sampling probability	Validation records
RefCOCO	0.10	10,834
RefCOCO+	0.10	10,758
RefCOCOg	0.10	9,536
gRefCOCO	0.20	14,229
Mapillary Vistas	0.15	3,053
VG, native images	0.33	3,003

Appendix

Table 3: Data mixture used for QART pretraining. VG is separated into native Visual Genome images and images overlapping with COCO.

Metric	SigLIP2 QART	DINOv2 QART
Overall accuracy	76.37%	72.93%
Yes/No	91.20%	89.87%
Number	60.80%	54.16%
Other	69.21%	65.01%
Mean active slots	29.45	30.96

Appendix

Table 4: VQAv2 results for QART with SigLIP2 and DINOv2 visual backbones. Both variants use the same LLaVA-1.5-7B integration and projector-training protocol.

Task	$\tau$	Primary metric $\uparrow$	Mean $K$
CORE	0.10	85.61	–
	0.20	90.96	–
	0.25	92.01	17.25
	0.35	91.92	–
	0.50	89.79	–
GeneCIS	0.05	12.81	30.04

Appendix

Table 5: Sensitivity to the relevance threshold. Lower thresholds generally increase the selected visual support, but performance does not vary monotonically with the resulting token budget. Mean

K

denotes the average number of active QART slots.

Figure 5: Representation refinement across recurrent rounds. Each row shows the spatial support of an active QART slot over successive rounds for a different query. The support progressively shifts toward the query-relevant region, including the foreground backpack, the person in plaid, and jersey number 13.

Figure 6: Query-dependent visual support and token allocation. For the same input image, changing the query from “left pizza” to “right pizza” redirects the visual support to the requested region and changes the active token budget from $K=8$ to $K=20$ .

Figure 7: QART adapts its token budget across image–query pairs. Representative examples span the allocation range from $K=2$ to $K=32$ . Localized queries can be represented with only a few active tokens, while broader or more spatially distributed visual evidence receives a larger representation.

Visual Tokenization

Visual tokenization turns images and video into compact, structured tokens for efficient perception, compression, and generation.

See all 70 papers

ReToken: Improving Long-Context VLMs with Visual Retrieval Token
Jul 30, 2026Yao Xiao, Reuben Tan, Zhen Zhu +3Vision-Language Models Vision-Language Retrieval
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind
AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens
Jun 5, 2026Xiaocheng Lu, Yuxi Chen, Jie Zhang +5Image Tokenization Visual Tokenization
The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University
RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
Aug 4, 2026Jian Zou, Xiaoyu Xu, Zhihua Wang +3Efficient VLM Inference Visual Token Pruning
City University of Hong Kong · Google Inc.

Jul 30, 2026J/K move · Enter open · S save

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu+2

University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind

Vision-Language Models Vision-Language Retrieval Visual Token Pruning Long-Context VLMs

Open paper AskPDF arXiv

Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Abstract

Visual Tokenization

Explore similar work

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens

RUTA: Principled Visual Token Allocation via Rate-Utility Optimization