Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.
Figures & tables
Figure 1: Comparison of visual encoding paradigms. (a) The base encoder processes inputs in a single pass. (b) Query-conditioned encoding incorporates the question into encoding but remains single-pass. (c) LoopLook enables task-guided refinement with adaptive computation depth.
Figure 2: Overview of LoopLook framework.
Figure 3: Block structure in pretrained visual towers. Layer-wise linear CKA forms coherent diagonal blocks, indicating contiguous encoder phases with aligned visual transformations. LoopLook places recurrence inside a middle-to-late block, reusing a consistent visual operation across loops.
Backbone
Method
General Understanding
Perception and Reasoning
Hallucination Robustness
Avg.
MMStar
VMCBench
AI2D
V ∗
BabyVision
VisualPuzzles
POPE
HalluBench
Qwen2.5 VL-3B
Base
55.9
74.9
81.5
72.8
14.9
31.2
88.8
54.3
59.3
CoT
56.2 ↑0.3
73.5 ↓1.4
82.8 ↑1.3
65.4 ↓7.4
13.9 ↓1.0
30.1 ↓1.1
89.0 ↑0.2
54.5 ↑0.2
58.2 ↓1.1
SC@5
54.6 ↓1.3
76.3 ↑1.4
83.9 ↑2.4
67.0 ↓5.8
16.5 ↑1.6
31.8 ↑0.6
88.6 ↓0.2
58.7 ↑4.4
59.7 ↑0.4
LoopLook
64.5 ↑8.6
78.8 ↑3.9
84.3 ↑2.8
78.5 ↑5.7
18.6 ↑3.7
34.4 ↑3.2
89.5 ↑0.7
62.4 ↑8.1
63.9 ↑4.6
Qwen3 VL-4B
Base
69.2
76.7
82.6
74.9
17.8
34.1
90.1
60.8
63.3
Table 1: Performance across backbones and traditional prompting/decoding strategies on eight benchmarks.
Method
Adaptive computation
Task- guided
Depth recurrence
MMStar
VMC Bench
AI2D
V ∗
Baby Vision
Visual Puzzles
POPE
Hallusion Bench
Avg.
Base
✗
✗
✗
32.3
53.8
53.7
49.7
10.8
23.9
89.7
47.3
45.2
Dense Connector
✗
✗
✗
27.9 ↓4.4
46.6 ↓7.2
50.6 ↓3.1
44.5 ↓5.2
13.7 ↑2.9
23.2 ↓0.7
77.2 ↓12.5
49.3 ↑2.0
41.6 ↓3.6
TGIF
✗
✓
✗
33.9 ↑1.6
52.8 ↓1.0
56.7 ↑3.0
55.0 ↑5.3
10.3 ↓0.5
25.0 ↑1.1
89.6 ↓0.1
46.2 ↓1.1
46.2 ↑1.0
DeepStack
✗
✗
✗
31.7 ↓0.6
49.4 ↓4.4
60.6 ↑6.9
49.2 ↓0.5
9.8 ↓1.0
25.3 ↑1.4
86.7 ↓3.0
45.3 ↓2.0
44.8 ↓0.4
LoopLook
✓
✓
✓
39.2 ↑6.9
55.1 ↑1.3
55.8 ↑2.1
54.5 ↑4.8
17.0 ↑6.2
24.8 ↑0.9
90.2 ↑0.5
51.4 ↑4.1
48.5 ↑3.3
Table 2: Performance comparison against existing VLM enhancement methods. The first three columns summarize whether each method supports adaptive computation, task guidance, and depth recurrence. Since all existing baselines provide LLaVA-based checkpoints, we use LLaVA-1.5-7B as the base model in this experiment.
Figure 4: Attention heatmap visualization of progressive visual refinement. We visualize the penultimate encoder layer of Qwen2.5-VL-3B at the end of each loop, providing a comparable view of the evolving visual state, and compare the base VLM with LoopLook on V ∗ cases. The base encoder produces diffuse attention heatmaps and incorrect answers, while LoopLook progressively concentrates attention on task-relevant evidence across recurrent loops.
Resolution
Method
CP
FP
IR
LR
Math
S&T
Overall
Loops
Base
61.2
37.2
61.2
48.4
50.0
33.6
48.6
1.0
50K
LoopLook
68.4
49.6
63.2
55.2
54.0
39.2
54.9
2.3
Base
64.4
46.0
61.2
48.4
53.2
36.0
51.5
1.0
200K
LoopLook
68.4
50.4
64.4
56.4
53.2
38.4
55.2
1.8
Base
68.8
45.6
64.4
54.8
54.4
39.6
54.6
1.0
450K
LoopLook
69.6
59.6
68.0
59.2
60.8
45.2
60.4
1.6
Table 3: Robustness to reduced resolution on MMStar. We report accuracy for six categories and overall, along with the average number of visual loops. Base denotes a single pass through the visual block.
Figure 5: Adaptive loop-depth allocation across benchmarks. LoopLook assigns more recurrent visual processing to fine-grained and hallucination-prone benchmarks, while easier or more direct recognition tasks exit earlier.
Figure 6: Token-wise routing patterns in Mixture-of-Looks. For each image token, we visualize the loop state with the largest routing weight. Earlier loops cover broad background regions, while later-loop assignments appear around local evidence required by the question, such as the bag region and the helmet area.
Resolution
Method
TTFT
Overhead
Decode
E2E
Base
67.2 ms
–
21.6 ms/tok
1.425 s
300K
LoopLook
71.6 ms
+6.5%
21.7 ms/tok
1.435 s
Base
84.9 ms
–
21.8 ms/tok
1.459 s
550K
LoopLook
89.2 ms
+5.1%
21.5 ms/tok
1.447 s
Table 4: Runtime overhead evaluated on Qwen2.5-VL-3B. We report time to first token (TTFT), per-token decoding latency, and end-to-end (E2E) runtime for generating 64 tokens.
Figure 7: Accuracy–compute tradeoff under exit thresholds. Each point corresponds to one exit threshold on MMStar.
Variant
MMStar
V ∗
HallusionBench
Avg.
Loops
LoopLook (Full)
64.5
78.5
62.4
68.5
1.89
Param.-matched
57.2 ↓7.3
73.8 ↓4.7
53.6 ↓8.8
61.5 ↓7.0
1.00
FLOPs-matched
58.6 ↓5.9
73.8 ↓4.7
58.7 ↓3.7
63.7 ↓4.8
1.00
w/o recurrence
59.7 ↓4.8
74.9 ↓3.6
57.8 ↓4.6
64.1 ↓4.4
1.00
w/o task condition
58.8 ↓5.7
73.8 ↓4.7
58.7 ↓3.7
63.8 ↓4.7
2.12
w/o token fusion
62.2 ↓2.3
76.4 ↓2.1
60.0 ↓2.4
66.2 ↓2.3
1.86
Table 5: Component ablations and fixed-depth comparisons. Avg. Loops denotes the sample-weighted average number of loops.
Window
Layers
MMStar
V ∗
HalluBench
Avg.
Mid-Late (Default)
17–30
64.5
78.5
62.4
68.5
Cross-block
14–27
58.4 ↓6.1
74.3 ↓4.2
56.9 ↓5.5
63.2 ↓5.3
Early
0–13
62.0 ↓2.5
77.0 ↓1.5
62.5 ↑0.1
67.2 ↓1.3
Early-Mid
3–16
63.1 ↓1.4
77.5 ↓1.0
62.7 ↑0.3
67.8 ↓0.7
Table 6: Recurrent window ablations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Layer-wise linear CKA maps for six visual towers. Across backbones, the encoder depth is organized into a small number of coherent visual blocks.
Backbone
Layers
CKA blocks
Qwen2.5-VL-3B
32
0–14, 15–16, 17–30 , 31
Qwen3-VL-4B
24
0–9, 10–22 , 23
InternVL3.5-4B
24
0–11, 12–22 , 23
InternVL3.5-8B
24
0–11, 12–22, 23
LoopVL-1B
28
0–18, 19–26, 27
LLaVA-1.5-7B
24
0–10, 11–19 , 20–23
Appendix
Table 7: CKA blocks for the visual towers analyzed in this work. Layer ranges are zero-indexed and interpreted from the CKA maps; singleton ranges denote isolated readout-style layers. Bold layer ranges indicate the blocks selected in our experiments for recurrent processing in each backbone.
Figure 9: A correct LoopLook open-ended counting case. The model decomposes the stacked-cube structure into a base layer and upper layers, verifies that every upper cube is supported, and reaches the minimum count of 13.
Figure 10: A correct LoopLook open-ended subtraction case. The model counts the two groups of cartoon planets separately and forms the correct subtraction equation.
Figure 11: Failure-to-success comparisons between the base VLM and LoopLook. Yellow boxes mark the queried evidence region.
Figure 12: Per-token dominant-loop tier maps on three additional cases. Blue/yellow/red denote tokens whose evidence is dominated by Loop 1/2/3 respectively.
Component
Qwen2.5-VL-3B
Qwen3-VL-4B
InternVL3.5-4B
LLaVA-1.5-7B
LoRA (vision tower, r=8 )
4.37
3.15
3.15
3.15
Task projection ( dl→d )
2.62
2.62
2.62
4.20
Mz
0.44
0.28
0.28
0.28
Mℓ (all recurrent blocks)
5.77
3.43
2.91
2.38
Loop-step embeddings er
0.001
0.001
0.001
0.001
Exit head
0.87
0.56
0.56
0.56
Appendix
Table 8: Trainable-parameter composition of LoopLook (millions).