cs.CVNov 15, 2026

When Does Reusing Layers Help Multimodal Embeddings?

Authors: Zhizhang Hu, Yue Zhang

Organizations: Amazon · University of California, Merced

Abstract

We ask whether a compact universal multimodal embedding models can produce better search embeddings by running the same layers again. Looped transformers add computation with nearly unchanged model size, making them an appealing alternative to a larger encoder. Their success in language-model reasoning and recent use for retrieval motivate the idea, but higher retrieval scores alone do not reveal what an extra pass contributes. We study eleven loop designs and training variants on three checkpoints of one vision-language architecture. Our protocol distinguishes whether a model uses its extra pass, whether its image states respond to the query, and whether those updates improve retrieval. We prove that an image placed before the query remains query-independent through repeated layers when information only travels forward. We then open a query-to-image path in three ways, including order-alternating recurrence, which places the query first on the next pass. Each path produces query-sensitive image states. For bidirectional attention and order alternation, freezing later visual updates does not lower accuracy. On all 78 datasets of the Massive Multimodal Embedding Benchmark version 2 (MMEB-V2), a standard added loop loses 0.46 points against its trained control; an intermediate generative stage changes the effect to +0.36. A model pretrained with recurrence gains +4.23 points over its own loop-removed control and benefits from query-conditioned image updates when answering visual questions or locating objects from text.

Figures & tables

Explore similar work

CardsList
  1. ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding

    Jul 30, 2026Shijie Wang, Xiangzhao Hao, Yueti Li +3Efficient Multimodal InferenceMultimodal Embedding

  2. Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining

    Nov 4, 2026Daichi Hotta, Yohei Kobashi, Takeshi Kojima +2Language Model PretrainingRecurrent Transformers

  3. VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

    Sep 27, 2026Peixi Wu, Mingzhou Jiang, Feipeng Ma +11Multimodal IREfficient Multimodal Inference