Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.
Figures & tables
Figure 1: Prune-then-loop retrofit. Left: a pretrained VLM embeds the image as vision tokens, the prompt as text tokens and passes them through its language backbone. The layers marked in grey are pruned. Right: Our retrofitting mechanism reuses the remaining layers. The prelude is executed once, and an adapter merges the prelude’s output with the hidden state of the recurrent block initialized with noise. The adapter and recurrent block are applied repeatedly, with the coda used to decode to logits.
Model
LM (B)
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
Baseline
TinyLLaVA-1.4B
1.10
57.6
85.1
46.7
59.6
49.7
45.4
54.1
39.4
38.6
52.91
Baselines, no recurrence
Baseline+ (S1,S2)
1.10
59.9
86.9
45.5
57.8
55.6
47.3
59.8
41.3
43.4
55.27
Baseline+ (S0,S1,S2)
1.10
60.4
86.7
47.3
60.2
55.5
46.4
60.4
43.2
42.2
55.80
Baseline(pruned)+ (S0,S1,S2) ( F1 )
0.84
58.0
86.1
37.7
50.3
48.6
47.6
58.3
38.1
47.1
52.41
Table 1: Comparison to baselines. The retrofit outperforms the baselines on average over nine benchmarks with 23.3% fewer stored language-model parameters. Sec. 5.1 defines the models. RR-VLM denotes our retrofitted recurrent VLM F4/8/46 of Sec. 3 and F1 the pruned single-pass model.
Model and training stages
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
Baseline+ (S0,S1,S2)
60.4
86.7
47.3
60.2
55.5
46.4
60.4
43.2
42.2
55.80
RR-VLM, S1 , S2
59.6
86.5
45.5
54.6
52.1
44.1
61.0
40.0
46.6
54.45
RR-VLM, S0 , S2
59.3
86.6
47.0
57.1
56.7
48.8
62.4
42.7
51.2
56.85
RR-VLM, S0 , S1 , S2 (ours)
59.2
86.7
47.7
59.1
58.9
50.5
62.6
44.6
53.9
58.13
Table 2: Training stages. We remove single stages of the training strategy of Sec. 3.2 from our best performing setting ( 4/8/4 , r=6 , 60B S0 budget), each stage with its distillation term. The first row repeats the unpruned baseline of Table 1 , which is trained with the full strategy. Avg is the mean of the nine columns to its left.
(a) Layer allocation, 56 layers
6/4/6 , r=11
52.56
4/6/4 , r=8
54.15
4/8/4 , r=6
55.75
Table 3: Ablations. Each table changes one setting of the model named in its header and the best value is in bold. All tables (a), (b) and (c) use the 7.5B S0 budget without text distillation.
Knowledge distillation
Avg
Baseline+ (S0,S1,S2) (Table 1 )
55.80
without S2 distillation
55.55
RR-VLM, no S2 distillation
56.65
RR-VLM, VL-KD only, λ=0.25
56.88
+ text-KD, τ=2
57.24
+ text-KD, τ=1 (ours)
58.13
Table 4: Knowledge distillation. We evaluate the impact of the knowledge distillation in S2 . Each row states on which batches the distillation term is applied and with which weight λ and temperature τ . All rows use the 60B S0 budget and the 4/8/4 layer allocation and share the same S0 , which applies the distillation term with λ=0.5 at τ=2 . The first two rows give the unpruned baseline of Table 1 with and without the S2 distillation.
GQA at test-time depth r
Training depth
r=2
r=4
r=5
r=6
r=8
r=12
r=16
sampled, rˉ=6 ( Fr~ )
57.8
59.3
59.5
59.5
59.4
59.1
59.3
fixed, r=6 ( Fr , our setting)
0.0 †
0.0 †
43.1
59.7
50.9
4.2 †
0.0 †
Table 5: Fixed against sampled training depth. Both models use the 4/8/4 layer allocation and the 7.5B S0 budget and differ only in the number of recurrences during training, fixed at r=6 or sampled with mean 6 . The columns give GQA accuracy when the model is evaluated with r passes. Values marked † come from runs that were stopped early because the output had degenerated.
Setting
Cache
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
n=1 (last pass)
0.727×
49.1
86.7
18.6
59.3
57.0
47.7
61.2
42.2
52.5
52.70
n=2
1.09×
56.5
86.7
31.5
59.2
58.6
49.7
62.6
44.1
53.8
55.85
n=3
1.45×
58.6
86.7
41.8
59.4
58.6
50.5
62.6
44.4
53.9
57.38
n=6 (ours)
2.55×
59.2
86.7
47.7
59.1
58.9
50.5
62.6
44.6
53.9
58.13
Table 6: KV-cache sharing across passes at inference. The table analyses the KV cache per token as a multiple of the base model’s cache. The model keeps only n copies of the cache instead of one per pass. The six passes use these copies in turn, so with n=1 every pass shares one cache, and n=6 is our model without sharing.
Figure 2: Qualitative examples. One item from each of the four reasoning-oriented benchmarks with the answers of the pretrained model, the converted recurrent model and our model below the image, a check mark denotes the correct option. The three trained non-recurrent baselines of Table 1 answer all four items wrongly.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Distillation
Avg
S0,S1,S2 (every stage)
53.53
S0 only
54.57
S0,S2 , λ=0.50
54.45
S0,S2 , λ=0.25
55.62
Appendix
Table 7: Placement of the distillation term at the 7.5B S0 budget. Each row shows in which training stages we apply the distillation term and with which weight λ , on the 4/6/4 layer allocation with r=6 . Seed 42. These are separate runs from the 4/6/4 row of Table 3 (b).
Setting
Cache
GQA
POPE
TVQA
SQA
MMB
RWQA
SEED
AI2D
CV2D
Avg
KV-sharing
0.727×
57.6
86.1
44.5
51.1
50.0
43.8
57.8
37.3
47.1
52.81
K-sharing
1.64×
58.5
86.6
44.7
53.5
56.4
48.5
61.8
39.1
46.1
55.02
no sharing
2.55×
59.7
86.5
45.1
57.3
55.1
46.7
62.8
41.1
47.5
55.75
Appendix
Table 8: KV-cache sharing across passes during training. The model is trained with the sharing active, at the 7.5B S0 budget and the 4/8/4 layer allocation of our ablations. The Cache column gives the KV cache per token as a multiple of the base model’s cache. K-sharing shares the keys across passes, KV-sharing the keys and the values. Seed 42.
Interleaving
Avg
no text batches
56.19
K=2
56.68
K=6
56.81
K=4 (our setting)
56.88
Appendix
Table 9: Text interleaving in S2 . Number of multimodal batches K per text batch during S2 , on the 60B S0 budget and the 4/8/4 layer allocation without text distillation. The best value is in bold. Seed 42.
Model
Variant
ϵ
Cap
GQA
Δ
Pass read
Passes run
Settled @6
sampled, mean 6
no exit
–
6
58.92
–
6.00
6.00
98.4%
read-out only
5⋅10−3
6
58.53
−0.39
2.81
6.00
compute-skipped
5⋅10−3
6
58.31
−0.61
2.79
3.75
compute-skipped
10−3
6
58.76
−0.16
3.40
3.89
fixed, 6
no exit
–
6
59.41
–
6.00
6.00
3.7%
read-out only
5⋅10−3
6
59.41
+0.00
6.00
6.00
Appendix
Table 10: Early exit (Eq. 5 ) on GQA. The retrofits use the 7.5B-token S0 and are trained at a fixed depth of 6 or with depth sampled at mean 6 or 16. The converted model at mean depth 32 is built like the converted recurrent VLM of Table 1 from the McLeish et al. (2025) checkpoint pretrained at that depth. Pass read is the mean pass a generated token is read from. Passes run is the mean number of block passes computed per generated token, with the first token counted at the cap ( † : tokens after the first only). Settled@6 is the share of generated tokens whose successive readouts have a KL divergence below 5⋅10−3 at pass 6. Rows without exit have no ϵ and no Δ . The readouts of the fixed-depth model do not settle, so the rule almost never fires on it (pass read 6.00 and 5.92). The values without exit differ from Tables 1 and 5 because this evaluation ran without image padding.
Figure 3: Qualitative examples (RealWorldQA, SEED-Bench). Two items per benchmark with the answer of every model in Table 1 . A check mark denotes the correct option.
Figure 4: Qualitative examples (AI2D, CV-Bench-2D). Two items per benchmark with the answer of every model in Table 1 . A check mark denotes the correct option.