BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

Nicolas Boizard; Théo Deschamps-Berger; Hippolyte Gisserot-Boukhlef; Céline Hudelot; Pierre Colombo

cs.CLApr 2, 2026

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

Authors: Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo

Organizations: Diabolocom · MICS, CentraleSupélec, Université Paris-Saclay · Artefact Research Center · Cohere

Abstract

Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Base (1): The original causal model. Bi+Base (2): The Base model with bidirectional attention enabled. Bi+MNTP (3): The Bi+Base model with an MNTP adaptation phase. Bi+Contrastive (4): The Bi+Base model with a contrastive adaptation phase. Bi+MNTP+Contrastive (5): The Bi+Base model adapted sequentially using MNTP followed by contrastive training. Intermediate dashed blocks denote adaptation phases.

Figure 2: Performance comparison of model variants across downstream tasks. Bars illustrate the absolute performance change relative to the unmodified Base model. Exact point differences are annotated above or below each bar.

Figure 3: Evolution of model performances during long run adaptation. Solid lines depict the absolute score change relative to the initial 10B adaptation, while dotted lines highlight the impact of complementary solutions to retain general knowledge.

Figure 4: Model performance across merging ratios. The first four columns report task scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks. Merging ratio index dictates the interpolation weight.

Figure 5: Model performance across data mix ratios. The first four columns report task scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks. The mix ratio specifies the proportion of multi-domain data.

Figure 6: Multilingual model performance by size. We report the average scores of the latest multilingual models across individual tasks on the XTREME and MTEB benchmarks. The dashed line indicates the open-source performance Pareto frontier.

Figure 7: Evolution of performance during domain specialization . We report test split performance on Beaver, SAFE and Aegis. Solid lines correspond to the exponential moving averaged (EMA) curves (

\alpha=0.4

), with shaded areas showing raw value deviation.

Figure 8: Evolution of performance during modality specialization . We report test split F1 score on e-SNLI-VE (vision) and Boolq-Audio (Audio). Solid lines correspond to exponential moving average curves (

\alpha=0.4

), with shaded areas showing raw data deviation.

Figure 9: Embedding model performance by size. Average score across individual tasks on MTEB Multilingual V2, MIEB, and MAEB, as a function of model size. The dashed line shows the Pareto frontier over open training data models.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
	Gemma3-270M	Gemma3-1B	Qwen3-0.6B	Qwen3-1.7B
Parameters	268M	1001M	596M	1721M
Layers	18	26	28	28
Hidden dimension	640	1152	1024	2048
Normalization	RMSNorm (pre & post attention)		RMSNorm (pre attention)
Embedding tying	No		Yes

Appendix

Table 1: Architectural comparison of base models used in this work.

	Gemma3-270M	Qwen3-0.6B	Gemma3-1B	Qwen3-1.7B
Learning rate	$5\times 10^{-4}$	$1\times 10^{-4}$	$7\times 10^{-5}$	$5\times 10^{-5}$
Warmup steps	0.01 epoch
LR scheduler	Warmup Stable Decay (WSD) ( Hu et al., 2024 )
Optimizer	AdamW (fused)
Max grad norm	1.0
Seed	42

Appendix

Table 2: Masked adaptation hyperparameters.

	Gemma3-270M	Qwen3-0.6B	Gemma3-1B	Qwen3-1.7B
Learning rate	$6\times 10^{-5}$	$3\times 10^{-5}$	$1\times 10^{-4}$	$1\times 10^{-4}$
Warmup steps	500
LR scheduler	Linear
Optimizer	AdamW (fused)
Max grad norm	1.0
Seed	42

Appendix

Table 3: Contrastive training hyperparameters.

Dataset	Pairs	Dataset	Pairs
KaLM
mMARCO (zh)	$379,870$	NLLB	$26,504$
SimCLUE	$290,699$	ESCI	$26,043$
Multi-CPR	$234,587$	Aya Dataset	$22,449$
SimCSE NLI	$217,099$	Yahoo Answers	$21,724$
T2Ranking	$188,606$	CSL	$19,945$

Appendix

Table 4: Training dataset composition after domain decontamination (

10,110,219

training pairs).

Panel of Figure 10 — Figure 10: The construction of BidirLM-Omni-2.5B relies on a modular composition strategy. We begin with three specialized variants sharing an identical underlying architecture: a vision model (Qwen3-VL-2B), our bidirectional text encoder (Qwen-1.7B Bi+MNTP), and an audio model (Qwen3-ASR-1.7B). First, we isolate their trainable textual backbones and perform a linear weight merge in equal proportions ( $1/3$ each) to forge a unified omnimodal representation space. Second, we extract the frozen, modality-specific projection heads (visual and audio) from the specialist models and seamlessly append them to the newly merged backbone. This composition enables cross-modal routing.

Figure 11: Performance comparison of MLM vs. MNTP adaptation. The first four columns report dataset-specific scores, while the rightmost column reports the model ranking based on average normalized performance across all tasks.

Figure 12: Per-layer cosine similarity between causal models and their Bi+MNTP-adapted encoders. Top: Aggregate cosine similarity per layer. Bottom: Comparison broken down by by Self-Attention and MLP weight group.

Figure 13: Per-layer cosine similarity between Bi+MNTP adapted Qwen3 models and their causal multimodal variants. Top: Aggregate cosine similarity per layer. Bottom: Comparison down by Self-Attention and MLP weight group.

Figure 14: Model performance on safety classification benchmarks across merging ratios. We report the resulting scores as a function of the weight allocated to the causal Qwen3Guard-Gen-0.6B model when merged with our Bi+MNTP Qwen3-0.6B encoder.

Figure 15: Model performance on multimodal classification benchmarks across merging ratios. We report the resulting scores as a function of the weight allocated to the causal Qwen3-ASR-0.6B and Qwen3-VL-2B-Instruct models when merged with our Bi+MNTP Qwen3-0.6B and Qwen3-1.7B encoders.

Figure 16: Average score per benchmark: BidirLM-Omni-2.5B (merged) vs. BidirLM-Omni-2.5B (non-merged). Average score across individual tasks. Comparison following the contrastive training phase between the merged BidirLM-Omni variant and the non-merged baseline on: MTEB (Multilingual V2), MIEB (lite), and MAEB (beta).

MTEB (Multilingual V2)	Bitext Mining	Class.	Clust.	Instr. Rerank.	ML Class.	Pair Class.	Rerank.	Retr.	STS	Mean (Task)	Mean (TaskType)
BidirLM-270M	59.5	58.8	45.9	-2.1	19.7	76.7	56.8	48.7	67.7	55.5	48.0
BidirLM-0.6B	66.7	62.2	49.6	0.8	23.8	78.4	58.1	56.5	70.8	59.6	51.9
BidirLM-1B	71.8	65.9	50.3	-0.4	25.1	79.8	58.6	56.5	74.6	62.1	53.6
BidirLM-1.7B	72.2	65.7	51.5	0.5	26.6	80.2	62.1	59.9	74.2	62.9	54.8
BidirLM-Omni-2.5B	72.2	65.5	51.6	0.9	26.6	80.7	63.4	59.4	75.7	63.1	55.1

Appendix

Table 5: Performance per task type on MTEB (Multilingual V2). Best score per column is bolded . Class.: classification, Clust.: clustering, Instr. Rerank.: instruction reranking, ML Class.: multilabel classification, Pair Class.: pair classification, Rerank.: reranking, Retr.: retrieval.

MIEB (lite)	Retr.	Comp.	Doc. Underst.	Img. Class.	Img. Clust.	Vision QA	ZS Class.	Mean (Task)	Mean (TaskType)
BidirLM-Omni-2.5B	28.8	46.0	76.9	73.6	61.1	48.6	48.1	58.1	54.7

Appendix

Table 6: Performance per task type on MIEB (lite) for BidirLM-Omni-2.5B. Comp.: compositionality, Vision QA: vision-centric QA, ZS Class.: zero-shot classification.

MAEB (beta)	Any2Any Retr.	Audio Class.	Audio Clust.	Audio ML Class.	Audio Pair Class.	Audio Rerank.	Audio ZS Class.	Mean (Task)	Mean (TaskType)
BidirLM-Omni-2.5B	32.8	58.2	5.5	32.3	66.4	74.8	55.3	45.2	46.5

Appendix

Table 7: Performance per task type on MAEB (beta) for BidirLM-Omni-2.5B. Any2Any Retr.: any-to-any retrieval, Audio ML Class.: audio multilabel classification, Audio ZS Class.: audio zero-shot classification.

MTEB (Multilingual V2)	Params	Mean Rank	Zero Shot	Bitext Mining	Class.	Clust.	Instr. Rerank.	ML Class.	Pair Class.	Rerank.	Retr.	STS	Mean (TaskType)	Mean (Task)
KaLM-Gemma3-12B	11.8B	1	73%	83.8	77.9	55.8	5.5	33.0	84.7	67.3	75.7	79.0	62.5	72.3
$\cdots$
BidirLM-Omni-2.5B	2.5B	17	100%	72.2	65.5	51.6	0.9	26.6	80.7	63.4	59.4	75.7	55.1	63.1
BidirLM-1.7B	1.7B	18	100%	72.2	65.7	51.5	0.5	26.6	80.2	62.1	59.9	74.2	54.8	62.9
$\cdots$
GTE-Qwen2-7B	7.1B	22	–	73.9	61.5	52.8	4.9	25.5	85.1	65.5	60.1	74.0	55.9	62.5

Appendix

Table 8: Per-task-type performance on MTEB (Multilingual V2) for our models (bold) and open-data baselines, ranked by Mean (Task).

\cdots

denotes a jump in leaderboard entries. Zero-shot ratios from the MTEB leaderboard . Best per column in bold .

MIEB (lite)	Params	Mean Rank	Zero Shot	Retr.	Comp.	Doc. Underst.	Img. Class.	Img. Clust.	Vision QA	ZS Class.	Mean (TaskType)	Mean (Task)
BidirLM-Omni-2.5B	2.5B	2	100%	28.8	46.0	76.9	73.6	61.1	48.6	48.1	54.7	58.1
$\cdots$
E5-V	8.4B	6	100%	26.9	39.4	56.0	70.6	51.7	52.6	36.2	47.6	51.9
$\cdots$
Nemotron-Omni-3B	4.7B	8	–	31.8	41.0	46.6	68.3	75.6	47.9	43.7	50.7	51.4
CLIP-bigG	2.5B	9	100%	34.2	35.0	35.5	77.8	80.8	43.0	72.4	54.1	51.3

Appendix

Table 9: Per-task-type performance on MIEB (lite) for our models (bold) and open-data baselines, ranked by Mean (Task).

\cdots

denotes a jump in leaderboard entries.

MAEB (beta)	Params	Mean Rank	Zero Shot	Any2Any Retr.	Audio Class.	Audio Clust.	Audio ML Class.	Audio Pair Class.	Audio Rerank.	Audio ZS Class.	Mean (TaskType)	Mean (Task)
LCO-Omni-7B	8.9B	1	–	55.2	57.1	1.7	38.4	67.3	78.7	64.5	51.9	52.0
LCO-Omni-3B	4.7B	2	–	54.5	55.4	1.3	36.8	66.7	75.4	62.2	50.3	50.7
BidirLM-Omni-2.5B	2.5B	3	100%	32.8	58.2	5.5	32.3	66.4	74.8	55.3	46.5	45.2
$\cdots$
Nemotron-Omni-3B	4.7B	5	–	38.8	47.6	1.2	21.1	66.1	81.5	69.8	46.6	43.1
$\cdots$

Appendix

Table 10: Per-task-type performance on MAEB (beta) for our models (bold) and open-data baselines, ranked by Mean (Task).

\cdots

denotes a jump in leaderboard entries.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
Encoder-Free Multimodal LLMs Learn Implicit Encoders
Oct 16, 2026David Harwath, Desh Raj, Xavier Velez +5Multimodal Large Language Models Multimodal Model Interpretability
The University of Texas at Austin · NVIDIA · Johns Hopkins University +2
UltraBERT: Beyond the Limit of BERT Pre-Training
Oct 10, 2026Jungwoo Park, Taewhoo Lee, Jaewoo KangLanguage Model Pretraining Masked Language Modeling
Korea University · AIGEN Sciences · Amazon

May 12, 2026J/K move · Enter open · S save

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Rian Touchent, Eric de la Clergerie

Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris

Language Model Pretraining Masked Language Modeling Autoregressive Language Modeling Language Modeling Continued Pretraining

Open paper AskPDF arXiv

When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.

Abstract

Language Modeling

Explore similar work

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Encoder-Free Multimodal LLMs Learn Implicit Encoders

UltraBERT: Beyond the Limit of BERT Pre-Training