Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.
Figures & tables
Figure 1: Prune-then-loop retrofit. Left: a pretrained VLM embeds the image as vision tokens, the prompt as text tokens and passes them through its language backbone. The layers marked in grey are pruned. Right: Our retrofitting mechanism reuses the remaining layers. The prelude is executed once, and an adapter merges the prelude’s output with the hidden state of the recurrent block initialized with noise. The adapter and recurrent block are applied repeatedly, with the coda used to decode to logits.
Model
LM (B)
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
Baseline
TinyLLaVA-1.4B
1.10
57.6
85.1
46.7
59.6
49.7
45.4
54.1
39.4
38.6
52.91
Baselines, no recurrence
Baseline+ (S1,S2)
1.10
59.9
86.9
45.5
57.8
55.6
47.3
59.8
41.3
43.4
55.27
Baseline+ (S0,S1,S2)
1.10
60.4
86.7
47.3
60.2
55.5
46.4
60.4
43.2
42.2
55.80
Baseline(pruned)+ (S0,S1,S2) ( F1 )
0.84
58.0
86.1
37.7
50.3
48.6
47.6
58.3
38.1
47.1
52.41
Table 1: Comparison to baselines. The retrofit outperforms the baselines on average over nine benchmarks with 23.3% fewer stored language-model parameters. Sec. 5.1 defines the models. RR-VLM denotes our retrofitted recurrent VLM F4/8/46 of Sec. 3 and F1 the pruned single-pass model.
Model and training stages
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
Baseline+ (S0,S1,S2)
60.4
86.7
47.3
60.2
55.5
46.4
60.4
43.2
42.2
55.80
RR-VLM, S1 , S2
59.6
86.5
45.5
54.6
52.1
44.1
61.0
40.0
46.6
54.45
RR-VLM, S0 , S2
59.3
86.6
47.0
57.1
56.7
48.8
62.4
42.7
51.2
56.85
RR-VLM, S0 , S1 , S2 (ours)
59.2
86.7
47.7
59.1
58.9
50.5
62.6
44.6
53.9
58.13
Table 2: Training stages. We remove single stages of the training strategy of Sec. 3.2 from our best performing setting ( 4/8/4 , r=6 , 60B S0 budget), each stage with its distillation term. The first row repeats the unpruned baseline of Table 1 , which is trained with the full strategy. Avg is the mean of the nine columns to its left.
(a) Layer allocation, 56 layers
6/4/6 , r=11
52.56
4/6/4 , r=8
54.15
4/8/4 , r=6
55.75
Table 3: Ablations. Each table changes one setting of the model named in its header and the best value is in bold. All tables (a), (b) and (c) use the 7.5B S0 budget without text distillation.
Knowledge distillation
Avg
Baseline+ (S0,S1,S2) (Table 1 )
55.80
without S2 distillation
55.55
RR-VLM, no S2 distillation
56.65
RR-VLM, VL-KD only, λ=0.25
56.88
+ text-KD, τ=2
57.24
+ text-KD, τ=1 (ours)
58.13
Table 4: Knowledge distillation. We evaluate the impact of the knowledge distillation in S2 . Each row states on which batches the distillation term is applied and with which weight λ and temperature τ . All rows use the 60B S0 budget and the 4/8/4 layer allocation and share the same S0 , which applies the distillation term with λ=0.5 at τ=2 . The first two rows give the unpruned baseline of Table 1 with and without the S2 distillation.
GQA at test-time depth r
Training depth
r=2
r=4
r=5
r=6
r=8
r=12
r=16
sampled, rˉ=6 ( Fr~ )
57.8
59.3
59.5
59.5
59.4
59.1
59.3
fixed, r=6 ( Fr , our setting)
0.0 †
0.0 †
43.1
59.7
50.9
4.2 †
0.0 †
Table 5: Fixed against sampled training depth. Both models use the 4/8/4 layer allocation and the 7.5B S0 budget and differ only in the number of recurrences during training, fixed at r=6 or sampled with mean 6 . The columns give GQA accuracy when the model is evaluated with r passes. Values marked † come from runs that were stopped early because the output had degenerated.
Setting
Cache
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
n=1 (last pass)
0.727×
49.1
86.7
18.6
59.3
57.0
47.7
61.2
42.2
52.5
52.70
n=2
1.09×
56.5
86.7
31.5
59.2
58.6
49.7
62.6
44.1
53.8
55.85
n=3
1.45×
58.6
86.7
41.8
59.4
58.6
50.5
62.6
44.4
53.9
57.38
n=6 (ours)
2.55×
59.2
86.7
47.7
59.1
58.9
50.5
62.6
44.6
53.9
58.13
Table 6: KV-cache sharing across passes at inference. The table analyses the KV cache per token as a multiple of the base model’s cache. The model keeps only n copies of the cache instead of one per pass. The six passes use these copies in turn, so with n=1 every pass shares one cache, and n=6 is our model without sharing.
Figure 2: Qualitative examples. One item from each of the four reasoning-oriented benchmarks with the answers of the pretrained model, the converted recurrent model and our model below the image, a check mark denotes the correct option. The three trained non-recurrent baselines of Table 1 answer all four items wrongly.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Distillation
Avg
S0,S1,S2 (every stage)
53.53
S0 only
54.57
S0,S2 , λ=0.50
54.45
S0,S2 , λ=0.25
55.62
Appendix
Table 7: Placement of the distillation term at the 7.5B S0 budget. Each row shows in which training stages we apply the distillation term and with which weight λ , on the 4/6/4 layer allocation with r=6 . Seed 42. These are separate runs from the 4/6/4 row of Table 3 (b).
Setting
Cache
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
KV-sharing
0.727×
57.6
86.1
44.5
51.1
50.0
43.8
57.8
37.3
47.1
52.81
K-sharing
1.64×
58.5
86.6
44.7
53.5
56.4
48.5
61.8
39.1
46.1
55.02
no sharing
2.55×
59.7
86.5
45.1
57.3
55.1
46.7
62.8
41.1
47.5
55.75
Appendix
Table 8: KV-cache sharing across passes during training. The model is trained with the sharing active, at the 7.5B S0 budget and the 4/8/4 layer allocation of our ablations. The Cache column gives the KV cache per token as a multiple of the base model’s cache. K-sharing shares the keys across passes, KV-sharing the keys and the values. Seed 42.
Interleaving
Avg
no text batches
56.19
K=2
56.68
K=6
56.81
K=4 (our setting)
56.88
Appendix
Table 9: Text interleaving in S2 . Number of multimodal batches K per text batch during S2 , on the 60B S0 budget and the 4/8/4 layer allocation without text distillation. The best value is in bold. Seed 42.
Model
Variant
ϵ
Cap
GQA
Δ
Pass read
Passes run
Settled @6
sampled, mean 6
no exit
–
6
58.92
–
6.00
6.00
98.4%
read-out only
5⋅10−3
6
58.53
−0.39
2.81
6.00
compute-skipped
5⋅10−3
6
58.31
−0.61
2.79
3.75
compute-skipped
10−3
6
58.76
−0.16
3.40
3.89
fixed, 6
no exit
–
6
59.41
–
6.00
6.00
3.7%
read-out only
5⋅10−3
6
59.41
+0.00
6.00
6.00
Appendix
Table 10: Early exit (Eq. 5 ) on GQA. The retrofits use the 7.5B-token S0 and are trained at a fixed depth of 6 or with depth sampled at mean 6 or 16. The converted model at mean depth 32 is built like the converted recurrent VLM of Table 1 from the McLeish et al. (2025) checkpoint pretrained at that depth. Pass read is the mean pass a generated token is read from. Passes run is the mean number of block passes computed per generated token, with the first token counted at the cap ( † : tokens after the first only). Settled@6 is the share of generated tokens whose successive readouts have a KL divergence below 5⋅10−3 at pass 6. Rows without exit have no ϵ and no Δ . The readouts of the fixed-depth model do not settle, so the rule almost never fires on it (pass read 6.00 and 5.92). The values without exit differ from Tables 1 and 5 because this evaluation ran without image padding.
Figure 3: Qualitative examples (RealWorldQA, SEED-Bench). Two items per benchmark with the answer of every model in Table 1 . A check mark denotes the correct option.
Figure 4: Qualitative examples (AI2D, CV-Bench-2D). Two items per benchmark with the answer of every model in Table 1 . A check mark denotes the correct option.
A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing. Qwen2.5-0.5B-Instruct is split into a Prelude, a weight-tied Recurrent Block, and a Coda, with an identity-preserving one-loop path and a re-entry bridge on later loops. At loop 1 the retrofit remains non-inferior to its base on a preregistered ARC battery. Three findings. First, the mechanism is a reusable procedure rather than terminal-answer lookup, and installs at two budgets: 6M trained parameters over frozen base weights and 180M full-block. With intermediate-step supervision, the model computes one task step per loop and persists when only final answers are graded. The adapter matched the full block overall (83.8% versus 84.0%), led through depth 11, and trailed beyond. Verbal fine-tuning reached 79-86% on controlled verbal renderings (zero-shot transfer was minimal), and adapter verbal training begun from the installed mechanism outpaced matched fresh training by 18.6 points, including on a held-out test set. Second, the operation extrapolates to roughly 1.5 times its supervised depth, holding 70% accuracy through depth 18. Third, a same-size scratchpad-trained model matched the recurrent model within its learned horizon but collapsed beyond it. The recurrent model won overall, 84% versus 72%, retained 53% versus 2.5% beyond depth 10, and answered 7.6 times faster. An iterative transformer can therefore perform deeper reasoning in latent space faster than comparable or larger models fine-tuned on the same task, in a system-level comparison. A second task, running the rule in reverse, exposed the limits: the inverse was learnable in isolation, but no continuation acquired it while preserving the installed mechanism and general capability, a catastrophic-interference boundary. Learned depth selection remains open.
Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.
Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing visual-token reduction methods often irreversibly discard tokens despite depth-varying query relevance. Although recoverable routing allows excluded tokens to return, re-admitted tokens retain stale hidden states after bypassing intermediate decoder transformations. Thus, token recoverability does not guarantee representation readiness at the current decoder depth. Under a fixed routing trajectory at a 64-token budget, replacing stale re-admitted states with current-depth oracle representations improves average RefCOCO [email protected] from 18.96 to 23.18, revealing the impact of representation staleness. We propose CatchUp, a training-free framework combining reversible visual-token routing with Local Transition Correction (LTC). CatchUp repeatedly re-evaluates query-conditioned relevance over the complete visual-token set, preserving deferred tokens for later re-admission. LTC approximates their skipped representation evolution by transferring reference-to-current hidden-state transitions from semantically and spatially related active anchors. This correction improves alignment with current-depth oracle representations without recomputing the skipped decoder layers. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets. At a 64-token budget, CatchUp processes 88.9% fewer active visual tokens per post-routing decoder interval, retains 92.17% of dense-model multimodal performance, and reduces end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp
Eunju Park, Uiwon Hwang
Division of Artificial Intelligence & Software, Ewha Womans University