cs.CLAug 10, 2026

Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation

Authors: Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham, Arman Luthra

Organizations: Eximius Labs · Wabash College · Skop Intelligence Co.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient methods silently changes those outputs; LoRA-style adaptation rewrites the text path whether or not the weights are merged, invalidating every stored embedding. We propose modality-gated deep adapters: bottleneck adapters attached to every decoder layer of a frozen multimodal embedding LLM, grouped into per-modality packs that execute only while their own modality is being encoded. The result is a modality added with zero change to existing outputs: inputs no pack claims traverse the base model's own computation graph, bit-for-bit unchanged, and co-loaded packs compose with an exact-zero isolation matrix. Both properties are stated as propositions, hold after arbitrary training rather than only at initialization, require no task labels or routing metadata at inference, and are verified by exact-equality tests on the released checkpoints. On one frozen 2B base, the audio pack (injected as connector tokens) improves audio-to-text R@10 by +3.4 to +5.4 points over an identically trained control, positive at every seed and reproduced at eleven times the data; the thermal pack, reusing the base's own frozen vision path, clears its pre-registered acceptance gate roughly sevenfold at every seed and lifts thermal-to-text R@10 from 0.224 to 0.785. An encoder swap locates the missing capacity: an external audio encoder that outranks Whisper-family encoders in CLAP-style comparisons loses by 16 R@10 points inside the frozen LLM, so the capacity belongs in the layers, exactly where the gated adapters place it. We release the audio model, the thermal pack, and the training, evaluation and invariance suites: models at huggingface.co/EximiusLabs, code on GitHub.

Figures & tables

Figure 1: One frozen decoder layer with a gated adapter. Left: on an input whose modality no pack claims (text, image, video), the hook returns the frozen layer’s output before any adapter arithmetic executes, so the computation graph is the released base model’s own and the output is bit-for-bit the base’s (Proposition 1 ); ungated methods such as LoRA instead execute on this path and change it. Right: on an input of the pack’s own modality (here audio), the bottleneck adapter is added to the residual stream and trained. Each added modality carries its own pack and gate, keyed on the input rather than on a task label, and co-loaded packs are mutually invisible, so packs compose with exact-zero isolation (Table 6 ).
TowerA →\to T R@10T →\to A R@10
Qwen2.5-Omni ( ∼{\sim} 640M)0.6310.684
Dasheng-base (86M)0.4690.538
Table 1: An encoder that outranks Whisper-family towers in shallow-projection comparisons (Dinkel et al., 2026 ) loses by 16 points inside the frozen-LLM splice. Matched 45K/800-step arms, identical recipe, each arm on frames from its own frozen tower. The towers are not parameter-matched; the probe establishes non-transfer of the external ranking, not encoder superiority per parameter.
Figure 2: The ranking inversion behind Table 1 . Left: under shallow projections, Dasheng outranks Whisper-family encoders on sound retrieval (AudioCaps text-to-audio mAP@10, as reported by Dinkel et al., 2026 ). Right: spliced into the frozen LLM under a matched recipe, the base’s co-trained Whisper-family tower wins by 16 R@10 points. External encoder rankings do not transfer to this architecture class.
ArmTrainedR@1R@10T →\to ALoss
Control (no adapt.)16.4M0.2020.6310.6844.70
+ gated, r=128r{=}12831.2M0.2170.656——
+ gated, r=384r{=}38460.6M0.2290.6650.7084.46
Table 2: Gated in-layer capacity passes the pre-registered acceptance gate ( ≥+3\geq{+}3 R@10) at +3.4+3.4 , improves every measured direction, and has not saturated at the largest rank trained. The controlled experiment: 45K AudioCaps-only, 800 steps, identical recipe, one change per row; in-run protocol; audio-to-text unless marked; final training loss in the last column. Rank 384 is the released configuration. Dashes: not recorded.
CellPredecessorOurs
AudioCaps
A →\to T R@1 / R@100.332 / 0.741 †0.302 / 0.743 †
T →\to A R@1 / R@100.280 † / 0.7460.292 † / 0.775
Clotho (zero-shot)
A →\to T R@1 / R@100.135 † / 0.433 †0.127 † / 0.421 †
T →\to A R@1 / R@100.136 † / 0.4600.151 † / 0.482
Table 3: The adapters improve the point estimate in every text-to-audio cell and the VGGSound audio-to-text R@10, and cede audio-to-text R@1 on the caption benchmarks. Release-protocol comparison against the connector-only predecessor ( fusion-embedding-1 v0.3), matched bf16 lineage and native input template; Clotho strictly zero-shot. Bold: better point estimate per cell; † marks both members of a pair whose difference is below the ∼2{\sim}2 -point single-seed parity threshold (§ 4.1 ), read as parity.
CheckResult
Base parameter drift, max ∣Δ∣|\Delta| , every training run0
Text forward Δ\Delta vs. base, gate closed (unit test)0
Text/image Δ\Delta , hooks attached vs. removed, released checkpoint (smoke test)0
Zero-init adapter stack under an open gate0
Audio forward Δ\Delta , trained adapters, gate open≠0\neq 0
Gradients reaching base or tower parametersnone
Table 4: Verification of Proposition 1 and its supporting machinery. Zeros are exact (maximum absolute difference under matched execution), not tolerances. The open-gate row is the negative control: outputs must change when the branch executes, and that visible, gated change is what ungated adaptation applies to every input. Artifact-level detail in Appendix B .
Thermal cellFrozenEmber (3 seeds)
T →\to text R@100.2240.783/0.785/0.777
T →\to text R@10.0710.399/0.412/0.404
T →\to visible R@10 ∗0.1650.333/0.349/0.341
LLVIP ZS acc. (%)95.494.1/94.3/91.8
Table 5: The released thermal pack (Ember) lifts thermal-to-text R@10 from 0.224 to 0.777–0.785 and more than doubles the emergent thermal-to-visible direction, at every seed. Three seeds versus the frozen base; thermal-to-text is the IR-TD held-out split; the frozen column is the same base with the pack’s zero-initialized up-projection (exact identity). ∗ Thermal-to-visible (LLVIP twin) is emergent: no thermal-RGB pairs appear in training. The zero-shot classifier row shows the pack does not degrade the base’s thermal zero-shot classification (a gate-open readout); exact preservation of non-thermal inputs is the bitwise check in Table 6 .
ReadoutGate(s) openmax ∣Δ∣|\Delta|
Textnone0
Image (RGB)none0
Videonone0
Audioaudio0
Thermalthermal0
Audio, registry pathaudio0
Table 6: Every preserved readout is exactly zero: the released audio and thermal packs, co-loaded on one frozen base, reproduce their reference bit-for-bit. Each readout is compared to the reference the propositions predict (text/image/video to the raw base, audio to the audio-pack-only model, thermal to the thermal-pack-only model); max ∣Δ∣|\Delta| is the exact maximum absolute difference, 00 meaning bitwise-identical. The last two rows are checks: the audio pack through the registry vs. the shipped single-gate path (must be 00 ), and the thermal readout with its gate open vs. the frozen base (must be nonzero, confirming the pack is active). Mixed inputs that would open two gates in one forward are outside the guarantee and untested (§ 8 ).
More in this topic

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

Explore similar work

CardsList
Jul 21, 2026

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.

Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham +1

1Eximius Labs · 2Wabash College · 3Skop Intelligence Co.

Audio-Text AlignmentVLM AdaptationMultimodal EmbeddingCross-Modal RetrievalMultimodal Foundation Models
AskPDF
Oct 27, 2026

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.

Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70

Multimodal EmbeddingUnified Multimodal ModelsMultimodal PretrainingEfficient Multimodal Inference
AskPDF
Oct 16, 2026

Encoder-Free Multimodal LLMs Learn Implicit Encoders

We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as ``implicit encoders,'' encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.

David Harwath, Desh Raj, Xavier Velez +5

The University of Texas at Austin · NVIDIA · Johns Hopkins University +2

Multimodal Large Language ModelsMultimodal Model Interpretability
AskPDF