AdaLoop: Thinking with the Vision Encoder

Xinran Nicole Han; Yiyang Lu; Katelyn Gan; Todd Zickler

cs.CVOct 15, 2026

AdaLoop: Thinking with the Vision Encoder

Authors: Xinran Nicole Han, Yiyang Lu, Katelyn Gan, Todd Zickler

Organizations: Harvard University · MIT

Abstract

Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.

Figures & tables

Figure 1: Looking once versus repeatedly. Left: A standard VLM encodes the image once, independently of the question. Right: AdaLoop turns a span of the vision encoder into a question-conditioned perception loop , updating its visual state over

T

iterations. The language model answers from the resulting sequence, using either the current state or retained history.

Figure 2: Connecting visual iteration to language. The perception loop produces visual tokens

I_{t}

across iterations, which are passed to the LLM by concatenation

[\ \cdot\ |\ \cdot\ ]

with question

q

. A generates intermediate and final answers

y_{t}

using each

I_{t}

in isolation, and B does so using the entire visual history

I_{1}\ldots I_{t}

. C interleaves each

I_{t}

with intermediate answers using separator

S

to distinguish rounds. In all cases, the stability of intermediate answers

y_{t}

can be used for adaptive halting.

Figure 3: VisRO examples. (a) Examples from six core tasks, with level of difficulty shown in blue. (b) Examples from the held-out out-of-distribution (OOD) split that exceed training difficulty (e.g., with larger grids) or use held-out generation rules or appearances. Each panel shows a $448\times 448$ image, abbreviated question, and programmatically computed answer. Appendix E has more details.

Closed-source models (zero-shot)
Method	Params	$\bar{T}$ / tok.	ID							OOD
Method	Params	$\bar{T}$ / tok.	Count	Flow	Glyph	Metro	Maze	Path	Overall	Overall
GPT-6 Astra (max)	–	2,892	88.3	95.0	55.0	91.7	78.3	85.0	82.2	71.4
Claude Opus 5 (max)	–	15,807	45.0	5.0	1.7	18.3	25.0	36.7	21.9	11.7
Qwen2.5-VL-3B
Zero-shot	0	1.0	7.3	0.0	23.5	0.0	37.3	14.8	13.8	8.9

Table 1: Results on VisRO-Core6. Single-pass question-conditioned AdaLN improves over supervised fine-tuning (SFT), and recurrence (AdaLoop) further improves both in-distribution (ID) and out-of-distribution (OOD) accuracy across all backbones. AdaLoop uses Variant A with adaptive answer-stability stopping (

p=1

T_{\mathrm{test}}=3

). Scores are exact-match accuracy (%); Overall is the macro average over six tasks. Params : trainable parameters (M);

\bar{T}

: mean selected stopping depth on the ID set. Closed-source models use maximum reasoning effort on a stratified 10% test subset; tok. : average generated tokens per completed response, including reasoning tokens. Best overall scores for each backbone are in bold.

Method	Params	ID	OOD
SFT	66	49.9	23.9
+ GRPO	66	49.5	24.2
LIVR	66	47.8	22.0
AdaLoop	82	64.6	32.4

Table 2: Comparison with existing methods on Qwen2.5-VL. ID/OOD report accuracy (%) over VisRO-Core6.

Table 3: Qwen2.5-VL ablations. Notation as in Table 1 with shading indicating AdaLoop-A. (a) All interfaces improve over single-pass AdaLN, and Variant A performs best. (b) With no modulation, looping gains little over a single pass; modulation with question conditioning performs better at the same parameter count.

Panel of Figure 4 — Figure 4: Qwen2.5-VL loop placement. (a) Three candidate loop spans that are contained within, or that cross, encoder spans having high CKA similarity. (b) Recurrence improves overall accuracy for all spans. Gains vary by task for each loop span.

Panel of Figure 5 — Figure 5: Inference-time visual computation. All models are trained with $T_{\mathrm{train}}=3$ . (a) Mean stopping depth increases with difficulty on every backbone. (b) Beyond the training depth, Variants B and C remain largely stable, whereas A degrades.

Figure 6: Selected answer trajectories beyond the training depth (Qwen2.5-VL, Variant C). At each depth $t$ , the model answers from the visual history and its earlier drafts. In these selected examples, the shaded iterations ( $t>T_{\mathrm{train}}$ ) correct the answer and retain it.

Panel of Figure 7 — Figure 7: Qwen2.5-VL training compute allocation. (a) Radar visualization of numbers in Table (b). At approximately matched training FLOPs, depth-sampled AdaLoop matches the 50k-update single-pass control on VisRO and improves three of four VQA benchmarks. Deeply supervised AdaLoop is slightly below the 120k-update control on VisRO but outperforms it on all four general VQA benchmarks, using one third as many optimizer updates.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Qwen2.5-VL-3B	PaliGemma 2 3B	LLaVA-1.5-7B §
Vision encoder	Native ViT	SigLIP So400m/14	CLIP ViT-L/14
Vision hidden width $d_{v}$	1280	1152	1024
LM embedding width	2048	2304	4096
Image preprocessing	Dynamic	$448\times 448$	$448\times 448$
Visual tokens for the LM	$N/4$	1024	256
LoRA rank	64	64	128

Appendix

Table 4: Configurations and training hyperparameters. Defaults for the main experiments; exceptions are specified with the corresponding results. Batch sizes are global, and training length and warmup count optimizer updates.

N

denotes the number of recurrent visual tokens in Qwen. Encoder references: SigLIP So400m/14 [ Zhai et al., 2023 , Alabdulmohsin et al., 2023 ] , CLIP ViT-L/14 [ Radford et al., 2021 ] . § We use our re-trained LLaVA-1.5-7B checkpoint at

448\times 448

, with

2\times 2

average pooling of the 1,024 CLIP patch tokens, rather than the released 336-px checkpoint.

Figure 8: Pretrained visual representation similarity. Linear CKA across depths in Qwen2.5-VL, PaliGemma 2, and LLaVA, using the same $0$ – $1$ color scale. Boxes mark the default loop spans in Table 4 ; their corners correspond to each span’s input and output depths. Depth $k$ denotes the representation after $k$ blocks. LLaVA is shown through the penultimate block used by the model. The Qwen matrix is the same as in Figure 4(a) .

Figure 9: Question-conditioned visual updates. One residual branch of a frozen vision block (gray). The question is pooled into

c_{q}

and mapped to branch-specific shifts, scales, and gains. Coefficients are shared across visual tokens and iterations; the multiplicative factors are

\mathbf{1}+\gamma

and

\mathbf{1}+\alpha

(Equation 2 ). Blue modules are trainable. Only each generator’s output layer is zero-initialized, preserving the pretrained computation at initialization.

Figure 10: Examples from the fixed LLaVA-NeXT training subset. The examples illustrate general visual question answering, counting, optical character recognition, and captioning. Answers use the original dataset annotations [ Liu et al., 2024b ] ; prompts are shortened for readability.

Method	Params	VisRO overall			General VQA ∗
Method	Params	$T$	ID	OOD	MMStar	MMBench
Qwen2.5-VL-3B
SFT	66	1	49.9	23.9	51.7	77.1
Single-pass AdaLN	82	1	60.0	30.2	51.1	75.4
AdaLoop-A	82	3	63.7	32.4	51.1	75.5
PaliGemma2-3B

Appendix

Table 5: Fixed-depth evaluation on VisRO and general VQA. Scores are accuracies (%); VisRO scores are macro averages across six tasks. Params denotes trainable parameters (M). ∗ General VQA evaluation uses fixed

T=1

Variant	Fixed depth $t$		Adaptive depth $\tau$
Variant	Readouts	Generations	Readouts	Generations
A	1	1	$\tau$	$\tau$
B	$t$	1	$\tau$	$\tau$
C	$t$	$t$	$\tau$	$\tau$

Appendix

Table 6: Operation counts for online inference. All variants execute the visual loop

t

times at fixed depth and

\tau

times under answer-stability stopping. A readout applies the coda and connector to a visual state. A generation includes language-model prefill and answer decoding. Adaptive stopping checks the answer at every executed depth.

Table 7: Estimated computational cost on Qwen2.5-VL. AdaLoop uses variant A with loop span

[2,14)

. Costs exclude padding and activation-checkpoint recomputation; TFLOPs and EFLOPs denote

10^{12}

and

10^{18}

operations.

Table 8: Looped architectures and language-guided visual processing. The two panels compare complementary architectural choices across different tasks and training settings. LM denotes language model.

Task	Representative question	Answer form
Discrimination and grouping
Count clusters	How many objects overlap another object?	Integer
Points on lines	How many points lie on the highlighted lines?	Integer
Find the same	Which items are identical?	Correspondences
Find the different	Which item differs from the others?	Position
Find the shadow	Which silhouettes match the objects?	Correspondences

Appendix

Table 9: VisRO task coverage. The 24 tasks are grouped by the visual operations they require. Questions and answer formats correspond to representative examples in Figures 3 , 11 , and 12 ; difficulty extensions can use different formats.

Task	ID (xhard)	OOD
Find the same	20 icons; 6 matching pairs; 6 near matches	25 icons; 8 matching pairs; 8 near matches
Find the different	9×9 grid; local alteration	10×10 grid; smaller alteration cap
Cube unfolding	10 views; orientation distractors	12 views; glyph-orientation cues
3D block counting	4×5 base; max height 5; min count 30+	5×5 base; max height 6; min count 45+
Block changes	8–10 changes; 5+ columns	12–16 changes; 8+ columns
Viewpoint rotation	4×5 base; 20–28 cubes	5×5 base; 30–42 cubes

Appendix

Table 10: Example ID and OOD settings in VisRO. We compare xhard, the highest ID difficulty, with the held-out OOD settings. A trailing + denotes a lower bound. In cube unfolding, OOD items distinguish same-color faces by glyph identity and orientation.

Panel of Figure 11 — Figure 11: VisRO task examples (1/2). The examples illustrate matching and discrimination (top), composition and transformations (middle), and spatial structure (bottom). All examples use the task-specific hard setting from a fixed generator revision; the text below each image summarizes its question.

Panel of Figure 12 — Figure 12: VisRO task examples (2/2). The examples illustrate point counting, overlap matching, and jigsaw completion (top), planar composition and transformations (middle), and folding and three-dimensional views (bottom). As in Figure 11 , all examples use the task-specific hard setting, and the text summarizes each question.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models
Oct 22, 2026Wencheng Ye, Anning Hu, Hongye Fang +2Visual Evidence Sensitivity in VLMs
Tongji University · Shanghai Jiao Tong University
LoopVL: Recurrent Visual Intelligence
Sep 29, 2026Zhe Qian, Ziyang Gong, Zhongxing Xu +13Vision-Language Models Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Baidu · Shanghai Jiao Tong University +5
RECAP: Looking Once Is Not Enough for Vision-Language Reasoning
Oct 22, 2026Zhaolu Kang, Tailong Luo, Chenxin Li +13Reasoning for VLMs Visual Evidence Sensitivity in VLMs
Peking University · Chengdu Minto Tech

Oct 22, 2026J/K move · Enter open · S save

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models

Wencheng Ye, Anning Hu, Hongye Fang, Tianyi Wang+1

Tongji University · Shanghai Jiao Tong University

Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.

Abstract

VLM Reasoning

Explore similar work

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models

LoopVL: Recurrent Visual Intelligence

RECAP: Looking Once Is Not Enough for Vision-Language Reasoning