When Does Reusing Layers Help Multimodal Embeddings?
Organizations: Amazon · University of California, Merced
Abstract
We ask whether a compact universal multimodal embedding models can produce better search embeddings by running the same layers again. Looped transformers add computation with nearly unchanged model size, making them an appealing alternative to a larger encoder. Their success in language-model reasoning and recent use for retrieval motivate the idea, but higher retrieval scores alone do not reveal what an extra pass contributes. We study eleven loop designs and training variants on three checkpoints of one vision-language architecture. Our protocol distinguishes whether a model uses its extra pass, whether its image states respond to the query, and whether those updates improve retrieval. We prove that an image placed before the query remains query-independent through repeated layers when information only travels forward. We then open a query-to-image path in three ways, including order-alternating recurrence, which places the query first on the next pass. Each path produces query-sensitive image states. For bidirectional attention and order alternation, freezing later visual updates does not lower accuracy. On all 78 datasets of the Massive Multimodal Embedding Benchmark version 2 (MMEB-V2), a standard added loop loses 0.46 points against its trained control; an intermediate generative stage changes the effect to +0.36. A model pretrained with recurrence gains +4.23 points over its own loop-removed control and benefits from query-conditioned image updates when answering visual questions or locating objects from text.
Figures & tables
| Model and training | Image | Video | Document | Overall | |
| (36) | (18) | (24) | (78) | paired control | |
| Qwen3.5-2B: loops added after pretraining | |||||
| Standard conversion | 69.58 | 51.03 | 67.30 | 64.60 | — |
| + standard causal loop | 69.69 | 50.48 | 66.06 | 64.14 | |
| + pooled-state feedback | 69.78 | 51.00 | 65.86 | 64.24 | |
| + order-alternating recurrence | 69.51 | 50.90 | 65.70 | 64.04 | |
| Loop setting | Seed 0 | Seed 1 | Mean |
|---|---|---|---|
| Standard causal loop | |||
| Bidirectional attention | |||
| Order alternation | |||
| Generative retrofit | |||
| Native recurrence |
Multimodal Embedding
Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.
See all 167 papers