Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.
Figures & tables
Figure 1: Base (1): The original causal model. Bi+Base (2): The Base model with bidirectional attention enabled. Bi+MNTP (3): The Bi+Base model with an MNTP adaptation phase. Bi+Contrastive (4): The Bi+Base model with a contrastive adaptation phase. Bi+MNTP+Contrastive (5): The Bi+Base model adapted sequentially using MNTP followed by contrastive training. Intermediate dashed blocks denote adaptation phases.
Figure 2: Performance comparison of model variants across downstream tasks. Bars illustrate the absolute performance change relative to the unmodified Base model. Exact point differences are annotated above or below each bar.
Figure 3: Evolution of model performances during long run adaptation. Solid lines depict the absolute score change relative to the initial 10B adaptation, while dotted lines highlight the impact of complementary solutions to retain general knowledge.
Figure 4: Model performance across merging ratios. The first four columns report task scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks. Merging ratio index dictates the interpolation weight.
Figure 5: Model performance across data mix ratios. The first four columns report task scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks. The mix ratio specifies the proportion of multi-domain data.
Figure 6: Multilingual model performance by size. We report the average scores of the latest multilingual models across individual tasks on the XTREME and MTEB benchmarks. The dashed line indicates the open-source performance Pareto frontier.
Figure 7: Evolution of performance during domain specialization . We report test split performance on Beaver, SAFE and Aegis. Solid lines correspond to the exponential moving averaged (EMA) curves ( α=0.4 ), with shaded areas showing raw value deviation.
Figure 8: Evolution of performance during modality specialization . We report test split F1 score on e-SNLI-VE (vision) and Boolq-Audio (Audio). Solid lines correspond to exponential moving average curves ( α=0.4 ), with shaded areas showing raw data deviation.
Figure 9: Embedding model performance by size. Average score across individual tasks on MTEB Multilingual V2, MIEB, and MAEB, as a function of model size. The dashed line shows the Pareto frontier over open training data models.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Gemma3-270M
Gemma3-1B
Qwen3-0.6B
Qwen3-1.7B
Architecture
Parameters
268M
1001M
596M
1721M
Layers
18
26
28
28
Hidden dimension
640
1152
1024
2048
Normalization
RMSNorm (pre & post attention)
RMSNorm (pre attention)
Embedding tying
No
Yes
Appendix
Table 1: Architectural comparison of base models used in this work.
Gemma3-270M
Qwen3-0.6B
Gemma3-1B
Qwen3-1.7B
Learning rate
5×10−4
1×10−4
7×10−5
5×10−5
Warmup steps
0.01 epoch
LR scheduler
Warmup Stable Decay (WSD) ( Hu et al., 2024 )
Optimizer
AdamW (fused)
Max grad norm
1.0
Seed
42
Appendix
Table 2: Masked adaptation hyperparameters.
Gemma3-270M
Qwen3-0.6B
Gemma3-1B
Qwen3-1.7B
Learning rate
6×10−5
3×10−5
1×10−4
1×10−4
Warmup steps
500
LR scheduler
Linear
Optimizer
AdamW (fused)
Max grad norm
1.0
Seed
42
Appendix
Table 3: Contrastive training hyperparameters.
Dataset
Pairs
Dataset
Pairs
KaLM
mMARCO (zh)
379,870
NLLB
26,504
SimCLUE
290,699
ESCI
26,043
Multi-CPR
234,587
Aya Dataset
22,449
SimCSE NLI
217,099
Yahoo Answers
21,724
T2Ranking
188,606
CSL
19,945
Appendix
Table 4: Training dataset composition after domain decontamination ( 10,110,219 training pairs).
Figure 10: The construction of BidirLM-Omni-2.5B relies on a modular composition strategy. We begin with three specialized variants sharing an identical underlying architecture: a vision model (Qwen3-VL-2B), our bidirectional text encoder (Qwen-1.7B Bi+MNTP), and an audio model (Qwen3-ASR-1.7B). First, we isolate their trainable textual backbones and perform a linear weight merge in equal proportions ( 1/3 each) to forge a unified omnimodal representation space. Second, we extract the frozen, modality-specific projection heads (visual and audio) from the specialist models and seamlessly append them to the newly merged backbone. This composition enables cross-modal routing.
Figure 11: Performance comparison of MLM vs. MNTP adaptation. The first four columns report dataset-specific scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks.
Figure 12: Per-layer cosine similarity between causal models and their Bi+MNTP-adapted encoders. Top: Aggregate cosine similarity per layer. Bottom: Comparison broken down by by Self-Attention and MLP weight group.
Figure 13: Per-layer cosine similarity between Bi+MNTP adapted Qwen3 models and their causal multimodal variants. Top: Aggregate cosine similarity per layer. Bottom: Comparison down by Self-Attention and MLP weight group.
Figure 14: Model performance on safety classification benchmarks across merging ratios. We report the resulting scores as a function of the weight allocated to the causal Qwen3Guard-Gen-0.6B model when merged with our Bi+MNTP Qwen3-0.6B encoder.
Figure 15: Model performance on multimodal classification benchmarks across merging ratios. We report the resulting scores as a function of the weight allocated to the causal Qwen3-ASR-0.6B and Qwen3-VL-2B-Instruct models when merged with our Bi+MNTP Qwen3-0.6B and Qwen3-1.7B encoders.
Figure 16: Average score per benchmark: BidirLM-Omni-2.5B (merged) vs. BidirLM-Omni-2.5B (non-merged). Average score across individual tasks. Comparison following the contrastive training phase between the merged BidirLM-Omni variant and the non-merged baseline on: MTEB (Multilingual V2), MIEB (lite), and MAEB (beta).
MTEB (Multilingual V2)
Bitext Mining
Class.
Clust.
Instr. Rerank.
ML Class.
Pair Class.
Rerank.
Retr.
STS
Mean (Task)
Mean (TaskType)
BidirLM-270M
59.5
58.8
45.9
-2.1
19.7
76.7
56.8
48.7
67.7
55.5
48.0
BidirLM-0.6B
66.7
62.2
49.6
0.8
23.8
78.4
58.1
56.5
70.8
59.6
51.9
BidirLM-1B
71.8
65.9
50.3
-0.4
25.1
79.8
58.6
56.5
74.6
62.1
53.6
BidirLM-1.7B
72.2
65.7
51.5
0.5
26.6
80.2
62.1
59.9
74.2
62.9
54.8
BidirLM-Omni-2.5B
72.2
65.5
51.6
0.9
26.6
80.7
63.4
59.4
75.7
63.1
55.1
Appendix
Table 5: Performance per task type on MTEB (Multilingual V2). Best score per column is bolded . Class.: classification, Clust.: clustering, Instr. Rerank.: instruction reranking, ML Class.: multilabel classification, Pair Class.: pair classification, Rerank.: reranking, Retr.: retrieval.
MIEB (lite)
Retr.
Comp.
Doc. Underst.
Img. Class.
Img. Clust.
Vision QA
ZS Class.
Mean (Task)
Mean (TaskType)
BidirLM-Omni-2.5B
28.8
46.0
76.9
73.6
61.1
48.6
48.1
58.1
54.7
Appendix
Table 6: Performance per task type on MIEB (lite) for BidirLM-Omni-2.5B. Comp.: compositionality, Vision QA: vision-centric QA, ZS Class.: zero-shot classification.
MAEB (beta)
Any2Any Retr.
Audio Class.
Audio Clust.
Audio ML Class.
Audio Pair Class.
Audio Rerank.
Audio ZS Class.
Mean (Task)
Mean (TaskType)
BidirLM-Omni-2.5B
32.8
58.2
5.5
32.3
66.4
74.8
55.3
45.2
46.5
Appendix
Table 7: Performance per task type on MAEB (beta) for BidirLM-Omni-2.5B. Any2Any Retr.: any-to-any retrieval, Audio ML Class.: audio multilabel classification, Audio ZS Class.: audio zero-shot classification.
MTEB (Multilingual V2)
Params
Mean Rank
Zero Shot
Bitext Mining
Class.
Clust.
Instr. Rerank.
ML Class.
Pair Class.
Rerank.
Retr.
STS
Mean (TaskType)
Mean (Task)
KaLM-Gemma3-12B
11.8B
1
73%
83.8
77.9
55.8
5.5
33.0
84.7
67.3
75.7
79.0
62.5
72.3
⋯
BidirLM-Omni-2.5B
2.5B
17
100%
72.2
65.5
51.6
0.9
26.6
80.7
63.4
59.4
75.7
55.1
63.1
BidirLM-1.7B
1.7B
18
100%
72.2
65.7
51.5
0.5
26.6
80.2
62.1
59.9
74.2
54.8
62.9
⋯
GTE-Qwen2-7B
7.1B
22
–
73.9
61.5
52.8
4.9
25.5
85.1
65.5
60.1
74.0
55.9
62.5
Appendix
Table 8: Per-task-type performance on MTEB (Multilingual V2) for our models (bold) and open-data baselines, ranked by Mean (Task). ⋯ denotes a jump in leaderboard entries. Zero-shot ratios from the MTEB leaderboard . Best per column in bold .
MIEB (lite)
Params
Mean Rank
Zero Shot
Retr.
Comp.
Doc. Underst.
Img. Class.
Img. Clust.
Vision QA
ZS Class.
Mean (TaskType)
Mean (Task)
BidirLM-Omni-2.5B
2.5B
2
100%
28.8
46.0
76.9
73.6
61.1
48.6
48.1
54.7
58.1
⋯
E5-V
8.4B
6
100%
26.9
39.4
56.0
70.6
51.7
52.6
36.2
47.6
51.9
⋯
Nemotron-Omni-3B
4.7B
8
–
31.8
41.0
46.6
68.3
75.6
47.9
43.7
50.7
51.4
CLIP-bigG
2.5B
9
100%
34.2
35.0
35.5
77.8
80.8
43.0
72.4
54.1
51.3
Appendix
Table 9: Per-task-type performance on MIEB (lite) for our models (bold) and open-data baselines, ranked by Mean (Task). ⋯ denotes a jump in leaderboard entries.
MAEB (beta)
Params
Mean Rank
Zero Shot
Any2Any Retr.
Audio Class.
Audio Clust.
Audio ML Class.
Audio Pair Class.
Audio Rerank.
Audio ZS Class.
Mean (TaskType)
Mean (Task)
LCO-Omni-7B
8.9B
1
–
55.2
57.1
1.7
38.4
67.3
78.7
64.5
51.9
52.0
LCO-Omni-3B
4.7B
2
–
54.5
55.4
1.3
36.8
66.7
75.4
62.2
50.3
50.7
BidirLM-Omni-2.5B
2.5B
3
100%
32.8
58.2
5.5
32.3
66.4
74.8
55.3
46.5
45.2
⋯
Nemotron-Omni-3B
4.7B
5
–
38.8
47.6
1.2
21.1
66.1
81.5
69.8
46.6
43.1
⋯
Appendix
Table 10: Per-task-type performance on MAEB (beta) for our models (bold) and open-data baselines, ranked by Mean (Task). ⋯ denotes a jump in leaderboard entries.
When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.
Rian Touchent, Eric de la Clergerie
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as ``implicit encoders,'' encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.
David Harwath, Desh Raj, Xavier Velez +5
The University of Texas at Austin · NVIDIA · Johns Hopkins University +2
Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.