Injecting non-text modalities into frozen language models often introduces early optimization instability, making it difficult to separate architectural gains from transient training dynamics. We introduce Inductively Gated Cross-Attention (IGCA), a zero-initialized cross-attention bridge that preserves pretrained model behavior at initialization while enabling gradual multimodal conditioning. Integrated across six decoder layers of frozen Falcon-40B, open_llama_30b, and Bloom-7b1 backbones (with 40,404,961,288 trainable adapter parameters used during training), IGCA acts as an exact algebraic no-op at initialization with a maximum output discrepancy of 0.00e+00. Over 59,500 training steps on Falcon-40B, the training loss decreases from 3.9403 to 0.0007 (with final ten step losses of [0.0001, 0.0003, 0.0005, 0.0002, 0.0005, 0.0011, 0.0005, 0.0006, 0.0005, 0.0007]). On held-out evaluation, the best gate-on checkpoint achieves a loss of 2.2201 compared to 4.3469 for the gate-off baseline -- a 48.9% reduction (2.1268 nats) -- while average held-out loss improves from 4.3423 to 2.2498 (1.9075 nats) with a mean gate magnitude of ∣tanh(α)∣=0.0713.
Figure 1: Mechanism and evidence boundary for IGCA. A zero-initialized scalar gate removes the cross-attention update at initialization, matching the reported maximum output difference of 0.00e+00 . The six-layer configuration and 40,404,961,288 trainable-parameter count are transcribed from the logged record; exact historical layer indices and resource accounting are marked N/A because the contemporaneous run manifest is not included in the repository.
Algorithm 1 IGCA Training (per epoch)
Require: Training set T , frozen LLM fθ , IGCA parameters {αl,WQ(l),WK(l),WV(l)} , learning rate η
Ensure: Trained IGCA parameters
1: for each batch (X,q,r∗) in T do
2: Encode modalities: E←[ϕimg;ϕaud;ϕvid;ϕtxt]
3: Forward pass through frozen LLM fθ with IGCA modules active at configured layers
Table 1: Configuration and evidence status for the logged IGCA outcome. Resource fields requiring a contemporaneous run manifest are not inferred from parameter count.
Figure 2: Schematic reconstruction of the early-window training pattern (steps 0–2000): dense noisy per-step band ( ∼0.7–5.5 ) with the 100-step moving average settling near ∼2.1 . This panel is a shape diagnostic only, not raw run data; the logged 59,500-step endpoints ( 3.9403→0.0007 train; 4.3469→2.2201 held-out) are reported separately in Figure 3 and Table 3 .
Readout
Early window (0–2k)
Logged full run (59.5k)
Per-step loss band
∼0.7–5.5 (noisy)
—
Moving avg ( w=100 )
∼2.6→∼2.1
—
Train loss endpoints
—
3.9403→0.0007
Best held-out (gate-on)
—
2.2201
Role
shape diagnostic (not raw run data)
evidence record
Table 2: Early-window diagnostic readout (steps 0–2,000, illustrative shape diagnostic only, not raw run data) listed alongside the logged full-run endpoints (59,500 steps). The diagnostic explains optimization behavior; only the logged record is used as empirical evidence.
Metric
gate-off
gate-on
Initialization maximum output difference
0.00e+00
Training loss: initial → final
3.9403→0.0007
Training duration
59,500 steps
Best held-out loss
4.3469
2.2201
Best held-out reduction
2.1268 nats ( 48.9% )
Average held-out loss
4.3423
2.2498
Table 3: Falcon-40B IGCA outcome transcribed from the logged result record. N/A denotes a value for which the repository contains no contemporaneous run-manifest evidence.
Figure 3: Falcon-40B IGCA summary from the logged record. (a) Training loss endpoints over 59,500 steps ( 3.9403→0.0007 ); the plot intentionally does not invent an intermediate trajectory. (b) Best held-out loss for gate-off ( 4.3469 , plotted as 4.35 ) and gate-on ( 2.2201 , plotted as 2.22 ).
Outcome
Falcon-40B Result
Status
Initialization max. output difference
0.00e+00
Reported
Best held-out loss, gate-on
2.2201
Reported
Best held-out loss, gate-off
4.3469
Reported
Average held-out loss, gate-on
2.2498
Reported
Average held-out loss, gate-off
4.3423
Reported
Full-system hallucination score
N/A
Not evaluated
Table 4: Evidence scope. Only rows supported by the logged Falcon-40B IGCA record contain numerical outcomes.
Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient methods silently changes those outputs; LoRA-style adaptation rewrites the text path whether or not the weights are merged, invalidating every stored embedding. We propose modality-gated deep adapters: bottleneck adapters attached to every decoder layer of a frozen multimodal embedding LLM, grouped into per-modality packs that execute only while their own modality is being encoded. The result is a modality added with zero change to existing outputs: inputs no pack claims traverse the base model's own computation graph, bit-for-bit unchanged, and co-loaded packs compose with an exact-zero isolation matrix. Both properties are stated as propositions, hold after arbitrary training rather than only at initialization, require no task labels or routing metadata at inference, and are verified by exact-equality tests on the released checkpoints. On one frozen 2B base, the audio pack (injected as connector tokens) improves audio-to-text R@10 by +3.4 to +5.4 points over an identically trained control, positive at every seed and reproduced at eleven times the data; the thermal pack, reusing the base's own frozen vision path, clears its pre-registered acceptance gate roughly sevenfold at every seed and lifts thermal-to-text R@10 from 0.224 to 0.785. An encoder swap locates the missing capacity: an external audio encoder that outranks Whisper-family encoders in CLAP-style comparisons loses by 16 R@10 points inside the frozen LLM, so the capacity belongs in the layers, exactly where the gated adapters place it. We release the audio model, the thermal pack, and the training, evaluation and invariance suites: models at huggingface.co/EximiusLabs, code on GitHub.
Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham +1
Eximius Labs · Wabash College · Skop Intelligence Co.
Frozen Gemma 4 31B weights pretrained exclusively on text, unmodified, transfer through a thin trainable interface to non-text modalities the substrate has never processed. On the L24--L29 slice (192 attention heads), an English-text TxtCopy attention probe (95 sentences) and per-head ablation impact on four non-language token-pattern tasks (binary copy, associative recall, 1D cellular automaton Rule 90, binary addition) jointly classify four heads -- L26.28, L27.28, L27.2, L27.3 -- as top-tier on both signals. The slice-level joint coincidence is significant under hypergeometric null (P=0.0013, N=192, K=38, n=4) and survives multiplicity-aware permutation tests (PV4=0.013). Pretrained Gemma L26 reaches 60.22% on OGBench cube-double-play-task1 vs ~1% for random-init Gemma (+59pt at n=3); a FrozenRandom-GPT2 control with correct 1/dk scaling also fails. Head-level causal validation: zeroing L26.28 in the trained cube-task1 IQL agent drops success 63.3%→10.0% vs 46.7% for a layer-matched low-TxtCopy negative control (3.2× specificity at n=30; n=5 paired-tp=0.039). A full L26 sweep places L26.28 at rank 4 of 32. Honest negatives: within-L26 Spearman ρ(TxtCopy, drop)=+0.37 (opposite of within-layer causal reading); single-head activation patching does not transfer the matching variable; the 4 named heads alone do not suffice on any task; Walker2d-DT and scene-task1 recruit L24 outside the named slice and show null head-ablation specificity. We frame the contribution as a cross-distribution importance fingerprint at the slice level plus head-level causal evidence on one cross-modality target.
In this work, we introduce GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a language model, which in turn generates embeddings for all varieties of input. We present the result: the jina-embeddings-v5-omni suite, a pair of models that encode text, image, audio, and video input into a single semantic embedding space. GELATO extends the two Jina Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Training is therefore much more efficient than full-parameter retraining. Additionally, the language model remains effectively unaltered, producing exactly the same embeddings for text inputs as the Jina Embeddings v5 Text models. Our evaluations show that GELATO produces results that are competitive with the state-of-the-art, yielding nearly equal performance to larger multimodal embedding models.
Florian Hönicke, Michael Günther, Andreas Koukounas +4