Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.
Figures & tables
Figure 1: Comparison of visual encoding paradigms. (a) The base encoder processes inputs in a single pass. (b) Query-conditioned encoding incorporates the question into encoding but remains single-pass. (c) LoopLook enables task-guided refinement with adaptive computation depth.
Figure 2: Overview of LoopLook framework.
Figure 3: Block structure in pretrained visual towers. Layer-wise linear CKA forms coherent diagonal blocks, indicating contiguous encoder phases with aligned visual transformations. LoopLook places recurrence inside a middle-to-late block, reusing a consistent visual operation across loops.
Backbone
Method
General Understanding
Perception and Reasoning
Hallucination Robustness
Avg.
MMStar
VMCBench
AI2D
V ∗
BabyVision
VisualPuzzles
POPE
HalluBench
Qwen2.5 VL-3B
Base
55.9
74.9
81.5
72.8
14.9
31.2
88.8
54.3
59.3
CoT
56.2 ↑0.3
73.5 ↓1.4
82.8 ↑1.3
65.4 ↓7.4
13.9 ↓1.0
30.1 ↓1.1
89.0 ↑0.2
54.5 ↑0.2
58.2 ↓1.1
SC@5
54.6 ↓1.3
76.3 ↑1.4
83.9 ↑2.4
67.0 ↓5.8
16.5 ↑1.6
31.8 ↑0.6
88.6 ↓0.2
58.7 ↑4.4
59.7 ↑0.4
LoopLook
64.5 ↑8.6
78.8 ↑3.9
84.3 ↑2.8
78.5 ↑5.7
18.6 ↑3.7
34.4 ↑3.2
89.5 ↑0.7
62.4 ↑8.1
63.9 ↑4.6
Qwen3 VL-4B
Base
69.2
76.7
82.6
74.9
17.8
34.1
90.1
60.8
63.3
Table 1: Performance across backbones and traditional prompting/decoding strategies on eight benchmarks.
Method
Adaptive computation
Task- guided
Depth recurrence
MMStar
VMC Bench
AI2D
V ∗
Baby Vision
Visual Puzzles
POPE
Hallusion Bench
Avg.
Base
✗
✗
✗
32.3
53.8
53.7
49.7
10.8
23.9
89.7
47.3
45.2
Dense Connector
✗
✗
✗
27.9 ↓4.4
46.6 ↓7.2
50.6 ↓3.1
44.5 ↓5.2
13.7 ↑2.9
23.2 ↓0.7
77.2 ↓12.5
49.3 ↑2.0
41.6 ↓3.6
TGIF
✗
✓
✗
33.9 ↑1.6
52.8 ↓1.0
56.7 ↑3.0
55.0 ↑5.3
10.3 ↓0.5
25.0 ↑1.1
89.6 ↓0.1
46.2 ↓1.1
46.2 ↑1.0
DeepStack
✗
✗
✗
31.7 ↓0.6
49.4 ↓4.4
60.6 ↑6.9
49.2 ↓0.5
9.8 ↓1.0
25.3 ↑1.4
86.7 ↓3.0
45.3 ↓2.0
44.8 ↓0.4
LoopLook
✓
✓
✓
39.2 ↑6.9
55.1 ↑1.3
55.8 ↑2.1
54.5 ↑4.8
17.0 ↑6.2
24.8 ↑0.9
90.2 ↑0.5
51.4 ↑4.1
48.5 ↑3.3
Table 2: Performance comparison against existing VLM enhancement methods. The first three columns summarize whether each method supports adaptive computation, task guidance, and depth recurrence. Since all existing baselines provide LLaVA-based checkpoints, we use LLaVA-1.5-7B as the base model in this experiment.
Figure 4: Attention heatmap visualization of progressive visual refinement. We visualize the penultimate encoder layer of Qwen2.5-VL-3B at the end of each loop, providing a comparable view of the evolving visual state, and compare the base VLM with LoopLook on V ∗ cases. The base encoder produces diffuse attention heatmaps and incorrect answers, while LoopLook progressively concentrates attention on task-relevant evidence across recurrent loops.
Resolution
Method
CP
FP
IR
LR
Math
S&T
Overall
Loops
Base
61.2
37.2
61.2
48.4
50.0
33.6
48.6
1.0
50K
LoopLook
68.4
49.6
63.2
55.2
54.0
39.2
54.9
2.3
Base
64.4
46.0
61.2
48.4
53.2
36.0
51.5
1.0
200K
LoopLook
68.4
50.4
64.4
56.4
53.2
38.4
55.2
1.8
Base
68.8
45.6
64.4
54.8
54.4
39.6
54.6
1.0
450K
LoopLook
69.6
59.6
68.0
59.2
60.8
45.2
60.4
1.6
Table 3: Robustness to reduced resolution on MMStar. We report accuracy for six categories and overall, along with the average number of visual loops. Base denotes a single pass through the visual block.
Figure 5: Adaptive loop-depth allocation across benchmarks. LoopLook assigns more recurrent visual processing to fine-grained and hallucination-prone benchmarks, while easier or more direct recognition tasks exit earlier.
Figure 6: Token-wise routing patterns in Mixture-of-Looks. For each image token, we visualize the loop state with the largest routing weight. Earlier loops cover broad background regions, while later-loop assignments appear around local evidence required by the question, such as the bag region and the helmet area.
Resolution
Method
TTFT
Overhead
Decode
E2E
Base
67.2 ms
–
21.6 ms/tok
1.425 s
300K
LoopLook
71.6 ms
+6.5%
21.7 ms/tok
1.435 s
Base
84.9 ms
–
21.8 ms/tok
1.459 s
550K
LoopLook
89.2 ms
+5.1%
21.5 ms/tok
1.447 s
Table 4: Runtime overhead evaluated on Qwen2.5-VL-3B. We report time to first token (TTFT), per-token decoding latency, and end-to-end (E2E) runtime for generating 64 tokens.
Figure 7: Accuracy–compute tradeoff under exit thresholds. Each point corresponds to one exit threshold on MMStar.
Variant
MMStar
V ∗
HallusionBench
Avg.
Loops
LoopLook (Full)
64.5
78.5
62.4
68.5
1.89
Param.-matched
57.2 ↓7.3
73.8 ↓4.7
53.6 ↓8.8
61.5 ↓7.0
1.00
FLOPs-matched
58.6 ↓5.9
73.8 ↓4.7
58.7 ↓3.7
63.7 ↓4.8
1.00
w/o recurrence
59.7 ↓4.8
74.9 ↓3.6
57.8 ↓4.6
64.1 ↓4.4
1.00
w/o task condition
58.8 ↓5.7
73.8 ↓4.7
58.7 ↓3.7
63.8 ↓4.7
2.12
w/o token fusion
62.2 ↓2.3
76.4 ↓2.1
60.0 ↓2.4
66.2 ↓2.3
1.86
Table 5: Component ablations and fixed-depth comparisons. Avg. Loops denotes the sample-weighted average number of loops.
Window
Layers
MMStar
V ∗
HalluBench
Avg.
Mid-Late (Default)
17–30
64.5
78.5
62.4
68.5
Cross-block
14–27
58.4 ↓6.1
74.3 ↓4.2
56.9 ↓5.5
63.2 ↓5.3
Early
0–13
62.0 ↓2.5
77.0 ↓1.5
62.5 ↑0.1
67.2 ↓1.3
Early-Mid
3–16
63.1 ↓1.4
77.5 ↓1.0
62.7 ↑0.3
67.8 ↓0.7
Table 6: Recurrent window ablations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Layer-wise linear CKA maps for six visual towers. Across backbones, the encoder depth is organized into a small number of coherent visual blocks.
Backbone
Layers
CKA blocks
Qwen2.5-VL-3B
32
0–14, 15–16, 17–30 , 31
Qwen3-VL-4B
24
0–9, 10–22 , 23
InternVL3.5-4B
24
0–11, 12–22 , 23
InternVL3.5-8B
24
0–11, 12–22, 23
LoopVL-1B
28
0–18, 19–26, 27
LLaVA-1.5-7B
24
0–10, 11–19 , 20–23
Appendix
Table 7: CKA blocks for the visual towers analyzed in this work. Layer ranges are zero-indexed and interpreted from the CKA maps; singleton ranges denote isolated readout-style layers. Bold layer ranges indicate the blocks selected in our experiments for recurrent processing in each backbone.
Figure 9: A correct LoopLook open-ended counting case. The model decomposes the stacked-cube structure into a base layer and upper layers, verifies that every upper cube is supported, and reaches the minimum count of 13.
Figure 10: A correct LoopLook open-ended subtraction case. The model counts the two groups of cartoon planets separately and forms the correct subtraction equation.
Figure 11: Failure-to-success comparisons between the base VLM and LoopLook. Yellow boxes mark the queried evidence region.
Figure 12: Per-token dominant-loop tier maps on three additional cases. Blue/yellow/red denote tokens whose evidence is dominated by Loop 1/2/3 respectively.
Component
Qwen2.5-VL-3B
Qwen3-VL-4B
InternVL3.5-4B
LLaVA-1.5-7B
LoRA (vision tower, r=8 )
4.37
3.15
3.15
3.15
Task projection ( dl→d )
2.62
2.62
2.62
4.20
Mz
0.44
0.28
0.28
0.28
Mℓ (all recurrent blocks)
5.77
3.43
2.91
2.38
Loop-step embeddings er
0.001
0.001
0.001
0.001
Exit head
0.87
0.56
0.56
0.56
Appendix
Table 8: Trainable-parameter composition of LoopLook (millions).
Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.
Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).
We introduce LoopVL to study whether Loop Transformers can be effectively extended to vision- language models. LoopVL combines Module-Loop and Model-Loop computation to iteratively update a unified vision-language state through shared modules. We train LoopVL from scratch through language pre-training, multimodal training, and post-training. LoopVL outperforms a range of similarly sized and larger non-recurrent models on multimodal understanding and visual reasoning benchmarks. We also observe Visual Aha Moments in LoopVL, characterized by pronounced shifts in visual attention across loops. LoopVL provides practical evidence for recurrent vision-language modeling and offers an intuitive perspective on how shared parameters can support deeper multimodal computation over continuously evolving visual-language states.
Zhe Qian, Ziyang Gong, Zhongxing Xu +13
Gaoling School of Artificial Intelligence, Renmin University of China · Baidu · Shanghai Jiao Tong University +5