We ask whether a compact universal multimodal embedding models can produce better search embeddings by running the same layers again. Looped transformers add computation with nearly unchanged model size, making them an appealing alternative to a larger encoder. Their success in language-model reasoning and recent use for retrieval motivate the idea, but higher retrieval scores alone do not reveal what an extra pass contributes. We study eleven loop designs and training variants on three checkpoints of one vision-language architecture. Our protocol distinguishes whether a model uses its extra pass, whether its image states respond to the query, and whether those updates improve retrieval. We prove that an image placed before the query remains query-independent through repeated layers when information only travels forward. We then open a query-to-image path in three ways, including order-alternating recurrence, which places the query first on the next pass. Each path produces query-sensitive image states. For bidirectional attention and order alternation, freezing later visual updates does not lower accuracy. On all 78 datasets of the Massive Multimodal Embedding Benchmark version 2 (MMEB-V2), a standard added loop loses 0.46 points against its trained control; an intermediate generative stage changes the effect to +0.36. A model pretrained with recurrence gains +4.23 points over its own loop-removed control and benefits from query-conditioned image updates when answering visual questions or locating objects from text.
Figures & tables
Figure 1: Using an extra pass is different from improving the model. On the 16-dataset diagnostic, the same trained Qwen3.5-2B loop improves by 1.27 points when its second pass is enabled. A separately trained single-pass control still scores higher. The middle and right points use exactly the same loop checkpoint. Boxes count layer applications, with the repeated eight-layer block sharing its base weights. The plotted score axis is restricted to show the differences.
Figure 2: More depth does not automatically change who can read whom. Arrows show possible information flow, with the arrowhead at the reader. Both passes in the fixed image-first order let the query read the image. Order-alternating recurrence places the already image-informed query before the image on pass 2, opening a reverse path through the same causal block. States and positional indices are permuted together, and the output is restored before the gated residual. The prelude and coda run once; only the middle block is reused. Opening a path permits dependence but does not guarantee that training will exploit it.
Model and training
Image
Video
Document
Overall
Δ
(36)
(18)
(24)
(78)
paired control
Qwen3.5-2B: loops added after pretraining
Standard conversion
69.58
51.03
67.30
64.60
—
+ standard causal loop
69.69
50.48
66.06
64.14
−0.46
+ pooled-state feedback
69.78
51.00
65.86
64.24
−0.36
+ order-alternating recurrence
69.51
50.90
65.70
64.04
−0.56
Table 1: Complete 78-dataset evaluation. Each difference uses the control in its own training group. The overall score averages all datasets, rather than averaging the three family means. The native loop-removed control trains longer to approximately match observed conversion time. These two model groups are not an architecture-matched comparison. Bidirectional and query-sandwich variants are not assigned an unmeasured full-benchmark score.
Loop setting
Seed 0
Seed 1
Mean
Standard causal loop
−0.86
−0.09
−0.47
Bidirectional attention
−0.39
+0.44
+0.02
Order alternation
−0.68
+0.17
−0.26
Generative retrofit
+0.66
+0.62
+0.64
Native recurrence
+3.42
+3.28
+3.35
Table 2: Repeated diagnostic comparisons. Loop minus its trained control on the 16-dataset mean, in points. The bidirectional control has the same attention change; the generative control has the same extra training stage. The native control has approximately matched observed training time.
Figure 3: Query dependence and useful visual updates are separate properties. Left: dataset-averaged relative visual-state differences after pass or cycle 1 and 2 when only the query changes (one checkpoint). This measures hidden states, not retrieval accuracy. Right: changes in the eight-image-task score under visual freeze; circles show seed-matched changes and bars their means. Opening a query path produces sensitivity without an accuracy cost under freeze for the measured added loops. The native loop is strongly affected; Figure 4 tests whether its dependence is query-specific.
Figure 4: Which tasks need the image to be updated under the right query? In the native LoopVL embedder, second-cycle visual states are replaced by states computed from the same image under another instance text; actual text states are retained. Dots show two seeds and short bars their means. Scores use MMEB’s diagnostic candidate pools. Question answering and grounding are more affected than composed retrieval. Generic text-to-image inputs provide a zero-change control.
Figure 5: The same image, two questions. Maps show per-patch relative changes in visual hidden states after pass or cycle 2, using a prespecified Visual7W example [ 31 ] . Models use native grids on a shared, unclipped scale. These are state-difference maps, not attention or saliency maps.
Figure 6: Which task groups benefit from recurrence? Axes partition all 78 MMEB-v2 datasets; points average dataset-score differences from each model’s own trained control. Scores use top-1 hit rate for images and videos and nDCG@5 for documents, on a 0–100 scale. The retrofit uses order-alternating recurrence. The native control uses the same pretrained weights without repetition and trains longer. The zero ring separates gains from losses.
Universal multimodal embedding (UME) maps heterogeneous multimodal inputs into a shared embedding space. Existing UME models either form embeddings through single forward encoding or add computation through explicit rationale tokens and latent autoregressive states. Although token expansion can improve complex matching, serial generation increases retrieval latency and makes the final embedding depend on generated intermediate states. This raises a different question: can useful computation be expanded along model depth while keeping the token workspace fixed? We analyze positive-negative similarity separation at every layer of independently trained UME models and observe a shared progression: early layers contextualize multimodal inputs, a contiguous middle-to-late stage forms retrieval-discriminative features, and the final layers map them into the embedding space. Based on this finding, we propose ReLoop-UME, which executes the early layers once, recurrently reuses a parameter-shared retrieval-forming block, and applies the final mapping layers after the last loop. Learnable Retrieval Registers provide persistent retrieval-specific states that accumulate and exchange evidence across loops, with the final register serving as the embedding readout. On MMEB-V2 and MRMR, ReLoop-UME consistently improves retrieval across different backbones while running 44.9x faster than UME-R1 and 1.5x faster than PLUME.
Shijie Wang, Xiangzhao Hao, Yueti Li +3
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China · AI Platform, evermind
Multimodal models have often scored below text-only baselines in the BabyLM Challenge (Klerings et al., 2024; Takmaz et al., 2025). We test whether additional recurrent computation helps close this gap. Recurrence increases computation by reusing layers without adding parameters or training data. Using a single implementation, we compare standard and looped Transformers under a 100M-word budget, with three seeds per main condition. Each image is represented by one feature vector from a frozen DINOv2 encoder, used as a prefix token. Caption words count toward the budget, so multimodal training also changes the language data. In a pre-registered text-only comparison, a 12-layer stack repeated four times exceeds a standard 48-layer Transformer by 5.4 BLiMP percentage points at matched inference compute. However, multimodal training lowers BLiMP accuracy by about 3.8 points in both architectures, with no clear mitigation from recurrence. An exploratory loop-side no-image control, trained with the multimodal language data and batching schedule, scores 5.7 points below the text-only baseline. Adding the visual prefix raises accuracy by 1.85 points on average, but this contrast is uncertain with three seeds. These results suggest that changes in language data and its presentation are a major source of the drop. Diagnostic tests show that the models use visual information; its contribution to grammatical generalization remains uncertain under this single-vector interface.
Daichi Hotta, Yohei Kobashi, Takeshi Kojima +2
Aoyama Gakuin University · The University of Tokyo
Universal multimodal retrieval requires compact embeddings that preserve task-relevant semantic information across diverse modalities. Prior works have incorporated latent reasoning into multimodal embedding learning to refine this information before embedding extraction. However, most existing approaches remain confined to deterministic latent paths, without exploring alternative trajectories to discover better embeddings. Thus, we propose VaME (Variational Multimodal Embeddings), a framework that models latent reasoning as a learnable distribution over trajectories. Specifically, we first introduce Variational Latent Reasoning (VLR) to enable autoregressive exploration in latent space, guided by answer reconstruction through a lightweight decoder. Meanwhile, we augment the original embedding-token readout with a latent-fused embedding to facilitate exploration during subsequent reinforcement learning. Finally, we optimize latent reasoning over stochastic variational trajectories through reinforcement learning, using Semantic Decoding Reward (SDR) to favor semantically meaningful trajectories with interpretable decoded outcomes. On the 78-task MMEB-V2 benchmark, spanning image, video, and visual-document retrieval, VaME outperforms most explicit CoT-based models and all latent-reasoning baselines. VaME also demonstrates robust performance on reasoning-intensive benchmarks such as MRMR, with substantial gains after reinforcement learning. Importantly, VaME achieves these gains with at least a 4.25x inference speedup over the deterministic latent autoregressive baselines. The code will be made publicly available.
Peixi Wu, Mingzhou Jiang, Feipeng Ma +11
University of Science and Technology of China · Tsinghua University · Kuaishou +1