Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks

Szymon Mazurek; Alberto Presta; Michał Byra

cs.CVOct 25, 2026

Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks

Authors: Szymon Mazurek, Alberto Presta, Michał Byra

Organizations: Samsung AI Center Warsaw, Poland · AGH University of Krakow, Poland · ACC Cyfronet AGH, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Poland

Abstract

Contrastive vision-language models such as CLIP typically rely on independently parameterized transformer blocks across depth. We investigate the extent to which these blocks can instead be reused recurrently while preserving the transferable capabilities of CLIP. We study recursive transformer backbones in which a small set of $N$ transformer blocks is applied $M$ times per forward pass. By holding $N \cdot M$ fixed to the depth of a standard ViT-B, our models preserve computational depth while progressively increasing parameter sharing. We consider two recursive CLIP model families: rCLIP, where only the vision encoder is recursive, and r $^2$ CLIP, where recurrence is also introduced in the text encoder. We study these models under both direct contrastive training and through knowledge distillation from a strong ViT-B/16 teacher pretrained on DFN-2B, using the substantially smaller CC3M and CC12M datasets. We evaluate the resulting models on zero-shot classification and cross-modal retrieval, and further study their adaptability through few-shot learning, visual grounding, and concept unlearning. Our results show that CLIP can tolerate substantial recurrent block reuse, although its effect depends on the encoder and downstream task. Recurrence also provides parameter efficiency by construction, since fewer distinct transformer blocks need to be stored and adapted. Our results establish recurrent block reuse as a practical design space for vision-language models and highlight its implications for knowledge transfer, downstream adaptation, and parameter efficiency.

Figures & tables

Figure 1: Accuracy–size trade-off of recursive CLIP models. Zero-shot ImageNet top-1 accuracy versus total parameter count for rCLIP and r²CLIP, compared with the DFN-2B ViT-B/16 teacher and distilled Reference CLIP. The parameter axis is reversed, with smaller models shown to the right.

Figure 2: Overview of the experimental approach. (1) A recursive encoder replaces L distinct transformer blocks ( A ) with N blocks applied M times, N·M = L ( B ), keeping computational depth fixed while reducing stored parameters. (2) Recursive (rCLIP, r²CLIP) and non-recursive students are distilled from a frozen DFN-2B ViT-B/16 teacher on CC3M + CC12M. (3) The resulting models are evaluated on classification (zero- and few-shot adapted), cross-modal retrieval, visual grounding and concept unlearning.

CLIP Model	Visual [M]	Text [M]	Total [M]
Ref. CLIP	86.19	63.43	149.62
rCLIP 1b12i	8.22	63.43	71.65
rCLIP 2b6i	15.31	63.43	78.74
rCLIP 3b4i	22.40	63.43	85.83
rCLIP 4b3i	29.48	63.43	92.91
rCLIP 6b2i	43.65	63.43	107.08

Table 1: Parameter counts across rCLIP and r²CLIP configurations. Each model uses 17.57 visual and 2.98 text GMACs at

224\times 224

resolution and 77-token context.

Figure 3: Baseline results for different rCLIP and r²CLIP configurations trained directly with the CLIP objective.

	Ref. CLIP		rCLIP 1b12i
Objective	IN	IN-R	IN	IN-R	Ret.
CRD + ICL + FD	37.3	46.9	30.3	33.2	81%
FD	37.0	46.3	28.2	29.2	76%
CRD	26.2	28.8	22.4	22.6	86%
ICL	19.3	22.3	18.4	19.6	95%
AM	31.0	47.5	15.5	17.0	50%

Table 2: Distillation loss components evaluated on the non-recursive Reference CLIP and the recursive rCLIP 1b12i student. Each objective is added to the primary CLIP contrastive loss with weight 1. CRD, ICL and FD use the weights of [ 35 ] (

1

1

and

2000

), while AM uses

\alpha=2000

. The top row reports the full CLIP-KD composition. Ret. denotes recursive ImageNet accuracy as a fraction of the corresponding non-recursive accuracy.

Figure 4: Results for the different rCLIP and r²CLIP configurations distilled from the DFN-2B ViT-B/16 teacher.

Setting	CLIP Model	ImageNet	SUN	Aircraft	EuroSAT	Food	Pets	Flowers	Caltech	DTD	UCF
Zero-shot	ViT-B/16 (OAI chkpt)	66.73	62.57	23.97	42.09	85.27	88.28	67.28	93.23	44.39	65.16
	Teacher	75.91	70.68	21.60	48.09	88.27	91.47	79.09	96.96	56.86	61.83
	Ref. CLIP	71.86	70.45	16.59	44.43	81.05	91.66	66.10	95.46	55.67	60.19
	rCLIP 1b12i	56.34	61.22	4.35	33.53	58.80	84.57	48.76	89.45	42.49	49.38
	rCLIP 3b4i	64.65	66.30	8.73	41.88	70.63	87.24	59.11	92.54	50.77	53.61
	rCLIP 6b2i	68.54	68.22	12.87	39.83	75.39	89.81	61.92	93.55	54.43	56.54

Table 3: Zero-shot and 16-shot accuracy [%] across ten classification datasets. Few-shot results use CLIP-LoRA [ 39 ] adaptation method. Teacher denotes the DFN-2B ViT-B/16, Ref. CLIP the non-recursive ViT-B/16 distilled from it under the same recipe as the recursive students. Within each setting, bold marks the best recursive model per column and underline the best model overall. The remaining block/iteration configurations are reported in the Supplementary.

CLIP Model	RefCOCO			RefCOCO+			RefCOCOg		ReferIt	Flickr
	val	testA	testB	val	testA	testB	val	test	test	test
ViT-B/16 (OAI chkpt)	83.99	86.86	78.45	73.96	81.98	63.06	74.16	74.39	68.62	80.85
Teacher	83.03	85.97	78.26	72.89	79.19	63.10	73.43	73.52	68.26	79.65
Ref. CLIP	84.31	86.68	78.85	73.21	78.46	64.74	75.25	74.00	68.98	80.18
rCLIP 1b12i	80.08	82.81	74.55	67.61	74.13	57.95	69.26	68.73	67.12	76.44
rCLIP 3b4i	81.75	84.56	76.96	70.99	76.78	61.30	72.24	71.93	68.19	78.28

Table 4: Visual grounding accuracy at IoU

\geq

0.5 following the HiVG [ 34 ] adaptation protocol. Bold marks the best recursive model per column, underline the best overall. The remaining block/iteration configurations are reported in the Supplementary.

CLIP Model	Flowers102		Pets		Food101		Average
CLIP Model	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)
Zero-shot
ViT-B/16 (OAI chkpt)	$92.61{\pm}8.30$	$70.56{\pm}0.12$	$83.25{\pm}21.55$	$89.12{\pm}0.37$	$87.68{\pm}4.34$	$85.85{\pm}0.05$	$87.85$	$81.84$
Teacher	$83.50{\pm}27.84$	$80.35{\pm}0.16$	$92.00{\pm}9.30$	$93.34{\pm}0.23$	$90.24{\pm}3.13$	$87.77{\pm}0.05$	$88.58$	$87.15$
Ref. CLIP	$88.39{\pm}19.50$	$66.72{\pm}0.13$	$89.25{\pm}13.91$	$92.18{\pm}0.29$	$88.44{\pm}2.75$	$80.23{\pm}0.04$	$88.69$	$79.71$
rCLIP 1b12i	$66.96{\pm}14.31$	$49.22{\pm}0.25$	$85.49{\pm}11.87$	$87.20{\pm}0.22$	$66.56{\pm}11.19$	$58.37{\pm}0.10$	$73.00$	$64.93$

Table 5: Zero-shot and post-DIET results.

D_{f}

is forget-set and

D_{t}

test-set accuracy, averaged over five seeds with one randomly selected class removed per seed; lower

D_{f}

indicates more effective forgetting and higher

D_{t}

better retention. In the post-DIET block, bold marks the best recursive model and underline the best model overall on average

D_{t}

. The remaining configurations are reported in the Supplementary.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Components	IN	IN-V2	IN-R	IN-S
Ref. CLIP	CRD + ICL + FD	37.3	32.5	46.9	27.0
Ref. CLIP	FD	37.0	31.7	46.3	27.9
Ref. CLIP	CRD	26.2	22.4	28.8	14.2
Ref. CLIP	ICL	19.3	16.4	22.3	9.8
Ref. CLIP	AM	31.0	26.5	47.5	23.9
rCLIP 1b12i	CRD + ICL + FD	30.3	26.0	33.2	16.3

Appendix

Table 6: Distillation loss components evaluated in isolation, with the reference composition of [ 35 ] as the top row of each block. Extended version of Table 2 . All runs additionally include the primary CLIP contrastive loss at weight 1; CRD, ICL and FD use the weights of [ 35 ] (

1

1

and

2000

), and AM uses

\alpha=2000

. Values are top-1 accuracy (%), single seed, CC3M, distilled from a DFN-2B ViT-B/16 teacher.

Model	$\alpha$	IN	IN-V2	IN-R	IN-S
Ref. CLIP	1	20.4	17.6	23.0	10.3
Ref. CLIP	2000	31.0	26.5	47.5	23.9
rCLIP 1b12i	1	15.7	13.0	15.9	5.6
rCLIP 1b12i	2000	15.5	13.1	17.0	5.9

Appendix

Table 7: Activation Matching weight sensitivity. All runs use the primary CLIP contrastive loss at weight 1 and AM as the only distillation term. Raising

\alpha

substantially improves the non-recursive baseline, with no visible effect in recursive one. Values are top-1 accuracy (%), single seed, CC3M, distilled from a DFN-2B ViT-B/16 teacher.

Model	CRD	ICL	FD	IN	IN-V2	IN-R	IN-S
rCLIP 1b12i	1	1	500	26.7	23.0	28.4	13.4
rCLIP 1b12i	1	1	1000	27.9	24.2	30.6	14.5
rCLIP 1b12i	1	1	2000	30.3	26.0	33.2	16.3
rCLIP 1b12i	1	1	3500	31.7	27.1	35.6	18.0
rCLIP 1b12i	1	1	5000	32.0	28.1	34.8	17.8
rCLIP 1b12i	0.5	1	5000	31.9	27.9	34.9	18.2

Appendix

Table 8: Distillation loss component weights, evaluated on the recursive rCLIP 1b12i student. The first block varies the FD weight at fixed CRD and ICL; the second varies CRD and ICL at an FD weight of

5000

; the third reports the highest FD weight examined. Last block applies the starting loss configuration and final one to the non-recursive model. The selected configuration is shown in bold. Values are top-1 accuracy (%), single seed, CC3M, distilled from a DFN-2B ViT-B/16 teacher.

CLIP Model	ImageNet	ImageNet-V2	ImageNet-R	ImageNet-S	MSCOCO		Flickr
CLIP Model	ImageNet	ImageNet-V2	ImageNet-R	ImageNet-S	I2T	T2I	I2T	T2I
Ref. CLIP	38.6	33.2	49.3	26.4	36.4	26.5	63.0	48.9
rCLIP 1b12i	32.8	28.6	36.8	17.8	30.6	22.4	57.6	42.9
rCLIP 2b6i	35.5	30.5	42.4	21.0	33.9	24.4	58.3	45.9
rCLIP 3b4i	36.2	30.6	45.0	23.3	35.5	25.0	60.0	47.6
rCLIP 4b3i	36.5	31.1	45.6	23.4	34.4	25.0	59.6	47.8

Appendix

Table 9: Zero-shot classification and cross-modal retrieval across the full block/iteration grid, trained directly with the CLIP objective and no knowledge distillation. Values are top-1 accuracy for classification [%] and Recall@1 [%] for retrieval. Ref. CLIP is the non-recursive ViT-B/16 trained under the same recipe. Bold marks the best recursive model per column and underline the best model overall. Exact values underlying Fig. 3 in the main paper.

CLIP Model	ImageNet	ImageNet-V2	ImageNet-R	ImageNet-S	MSCOCO		Flickr
CLIP Model	ImageNet	ImageNet-V2	ImageNet-R	ImageNet-S	I2T	T2I	I2T	T2I
Teacher	76.29	68.15	83.10	62.05	60.42	43.39	85.50	69.10
Ref. CLIP	72.50	64.30	81.60	58.50	61.04	44.00	86.20	70.00
rCLIP 1b12i	56.70	48.80	62.60	40.80	51.48	34.68	77.10	61.10
rCLIP 2b6i	62.80	54.60	70.70	48.10	55.68	38.19	81.80	65.68
rCLIP 3b4i	65.20	57.00	73.30	50.50	57.36	39.68	83.50	66.94

Appendix

Table 10: Zero-shot classification and cross-modal retrieval across the full block/iteration grid, distilled from the DFN-2B ViT-B/16 teacher on CC3M + CC12M under the objective selected in Sect. 4.2 and Sect. B.3 (

\lambda_{\mathrm{CRD}}=1

\lambda_{\mathrm{ICL}}=0

\lambda_{\mathrm{FD}}=10^{4}

). Values are top-1 accuracy for classification [%] and Recall@1 [%] for retrieval. Bold marks the best recursive model per column and underline the best model overall. Exact values underlying Fig. 4 in the main paper.

Setting	CLIP Model	ImageNet	SUN	Aircraft	EuroSAT	Food	Pets	Flowers	Caltech	DTD	UCF
Zero-shot	ViT-B/16 (OAI chkpt)	66.73	62.57	23.97	42.09	85.27	88.28	67.28	93.23	44.39	65.16
	Teacher	75.91	70.68	21.60	48.09	88.27	91.47	79.09	96.96	56.86	61.83
	Ref. CLIP	71.86	70.45	16.59	44.43	81.05	91.66	66.10	95.46	55.67	60.19
	rCLIP 1b12i	56.34	61.22	4.35	33.53	58.80	84.57	48.76	89.45	42.49	49.38
	rCLIP 2b6i	62.13	64.90	7.98	37.22	67.62	86.10	57.53	91.28	48.82	52.50
	rCLIP 3b4i	64.65	66.30	8.73	41.88	70.63	87.24	59.11	92.54	50.77	53.61

Appendix

Table 11: Zero-shot and 16-shot accuracy [%] across ten classification datasets, for the full block/iteration grid. Few-shot results use the CLIP-LoRA [ 39 ] adaptation method with

16

shots per class; the configuration is given in Supplementary D.3 . Within each setting, bold marks the best recursive model per column and underline the best model overall. Extended version of Table 3 in the main paper.

CLIP Model	RefCOCO			RefCOCO+			RefCOCOg		ReferIt	Flickr
CLIP Model	val	testA	testB	val	testA	testB	val	test	test	test
ViT-B/16 (OAI chkpt)	83.99	86.86	78.45	73.96	81.98	63.06	74.16	74.39	68.62	80.85
Teacher	83.03	85.97	78.26	72.89	79.19	63.10	73.43	73.52	68.26	79.65
Ref. CLIP	84.31	86.68	78.85	73.21	78.46	64.74	75.25	74.00	68.98	80.18
rCLIP 1b12i	80.08	82.81	74.55	67.61	74.13	57.95	69.26	68.73	67.12	76.44
rCLIP 2b6i	81.66	83.34	76.57	70.07	75.44	60.85	71.22	70.82	67.74	78.16

Appendix

Table 12: Visual grounding accuracy [%] at IoU

\geq 0.5

across the full block/iteration grid, following the HiVG [ 34 ] adaptation protocol; the configuration is given in Supplementary D.2 . Teacher denotes the DFN-2B ViT-B/16 and Ref. CLIP the non-recursive ViT-B/16 distilled from it under the same recipe. Unlike HiVG we omit LoRA pre-adaptation on MSCOCO and adapt each checkpoint directly to the target dataset, so absolute values are not directly comparable to the published ones. Bold marks the best recursive model per column and underline the best model overall. Extended version of Table 4 in the main paper.

CLIP Model	Flowers102		Pets		Food101		Average
CLIP Model	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)	$D_{f}$ (%)	$D_{t}$ (%)
Zero-shot
ViT-B/16 (OAI chkpt)	$92.61{\pm}8.30$	$70.56{\pm}0.12$	$83.25{\pm}21.55$	$89.12{\pm}0.37$	$87.68{\pm}4.34$	$85.85{\pm}0.05$	$87.85$	$81.84$
Teacher	$83.50{\pm}27.84$	$80.35{\pm}0.16$	$92.00{\pm}9.30$	$93.34{\pm}0.23$	$90.24{\pm}3.13$	$87.77{\pm}0.05$	$88.58$	$87.15$
Ref. CLIP	$88.39{\pm}19.50$	$66.72{\pm}0.13$	$89.25{\pm}13.91$	$92.18{\pm}0.29$	$88.44{\pm}2.75$	$80.23{\pm}0.04$	$88.69$	$79.71$
rCLIP 1b12i	$66.96{\pm}14.31$	$49.22{\pm}0.25$	$85.49{\pm}11.87$	$87.20{\pm}0.22$	$66.56{\pm}11.19$	$58.37{\pm}0.10$	$73.00$	$64.93$

Appendix

Table 13: Zero-shot and post-DIET results across the full block/iteration grid.

D_{f}

is forget-set and

D_{t}

test-set accuracy [%], averaged over five seeds with one randomly selected class removed per seed; lower

D_{f}

indicates more effective forgetting and higher

D_{t}

better retention. The Average columns report the unweighted mean over the three datasets. Unlearning uses DIET [ 16 ] with a learning rate selected per backbone–dataset pair; the configuration is given in Supplementary D.4 . In the post-DIET block, bold marks the best recursive model and underline the best model overall on average

D_{t}

. Extended version of Table 5 in the main paper.

Phase	Blocks †	Epochs ∗	LR	Batch
Warmup	–	60 / 30	$2.5\!\times\!10^{-4}$	640
Stage 1	1st group	20 / 5	$1.0\!\times\!10^{-4}$	480
Stage 2	2nd group	20 / 5	$5.0\!\times\!10^{-5}$	480
Stage 3	all $N$	20 / 5	$2.5\!\times\!10^{-5}$	320

Appendix

Table 14: Hi-LoRA curriculum. Each phase resumes from the previous phase’s best-validation checkpoint and restarts the cosine schedule. The number of phases is

1+\min(N,3)

, where

N

is the number of distinct transformer blocks: adapters are tied to unique blocks, so a backbone with

N<3

cannot unlock a new one at every stage. Vision adapters are unlocked cumulatively over three groups of distinct blocks, split as evenly as possible with any remainder assigned to the earlier groups; for

N=12

this gives

5/8/12

, reproducing HiVG’s original unlock pattern. The Blocks column refers to the vision tower only: text-tower adapters are unlocked across all blocks at once from stage 1 onward.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Oct 15, 2026Madhav Iyengar, Andreas Geiger, Hilde KühneVision-Language Models VLM Adaptation
University of Tübingen, Tübingen AI Center · KE:SAI
Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
May 11, 2026Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2Vision Transformer Recurrent Transformers
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
Oct 8, 2026Zidan Wang, Yaqian Li, Xiaokai Zhang +3VLM Post-Training VLM Adaptation
Huazhong University of Science and Technology · Li Auto Inc.

Oct 15, 2026J/K move · Enter open · S save

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Madhav Iyengar, Andreas Geiger, Hilde Kühne

University of Tübingen, Tübingen AI Center · KE:SAI

Vision-Language Models VLM Adaptation

Open paper AskPDF arXiv

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

Abstract

Vision-Language Models

Explore similar work

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder