NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Aurélien Lac; Tony Wu

cs.IRAug 31, 2026

NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Authors: Aurélien Lac, Tony Wu

Organizations: H Company

Abstract

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

Figures & tables

Figure 2: Dynamic-resolution image processing with a variable side-length cap.

Figure 3: Alternating sliding-window and global-attention layers in the NeoMME encoder stack.

Figure 4: Pre-normalized attention and MLP paths in one NeoMME encoder layer.

Group	Dataset	Within-stream weight	Expected tokens
Text only: 55% of packed input tokens
Web and PDF	FineWeb-Edu	0.167660	48.346B
	Nemotron-CC-v2.1	0.072896	21.020B
	FinePDFs	0.131214	37.837B
	FineWeb2-HQ	0.349900	100.897B
Synthetic, reference, and math	Cosmopedia-v2	0.102054	29.428B

Table 2: Configured pretraining mixture by logical source family.

Hardware and batch
Setting	NeoMME -260M	NeoMME -800M
Nodes	2	4
H100 accelerators	16	32
Packed sequences per GPU	4	2
Multimodal sequence share	45%	45%
Compute precision	bfloat16	bfloat16

Table 3: Pretraining hardware, batch, and optimization settings.

Figure 6: Pretraining loss and image gain at 90% masking for both model sizes. Lines show raw and smoothed measurements, with dashed lines marking observed loss transitions.

Figure 7: Natural-image input and generated text from the step-500,000 pretrained NeoMME -260M model.

Figure 8: Dense, cross-encoder, and late-interaction retrieval scoring patterns.

Group	Dataset	Within-stream weight	Expected batch share
Text only
LightOn retrieval	Filtered English retrieval	0.09	3.00%
	mLateOn multilingual	0.73	24.33%
	Code retrieval	0.13	4.33%
Other retrieval	Additional multilingual retrieval	0.03	1.00%
	Organic retrieval (en) and Organic retrieval (fr)	0.02	0.67%

Table 4: Configured retrieval training mixture by logical source family.

Figure 10: ViDoRe v3 nDCG@10 versus model size. NeoMME -Retriever 260M and 800M sit on the Pareto frontier. Arrows mark the gain of each NeoMME late-interaction head over its dense head. Table 19 gives the full results.

Table 12

Panel of Figure 11 — Figure 11: Hierarchical token pooling on ViDoRe v3: impact of the pool factor on retained retrieval quality for both NeoMME -Retriever models.

Figure 14: Similarity map for the query token hour , overlaid on a document from the ViDoRe syntheticDocQA energy test set. 26 26 26 vidore/syntheticDocQA_energy_test .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Dataset family	Description
Web and PDF	FineWeb-Edu	English web text filtered for educational quality.
	Nemotron-CC-v2.1	High-quality English Common Crawl text curated by NVIDIA.
	FinePDFs	Extracted PDF text in English, French, German, Spanish, Russian, Arabic, and Chinese.
	FineWeb2-HQ	High-quality web text in 20 languages.
Synthetic, reference, and math	Cosmopedia-v2	Synthetic educational text generated from web and reference material.
	Wikipedia	Encyclopedic articles in 10 languages.

Appendix

Table 10: Pretraining text source descriptions.

Dataset family	Weight	Description
FineVision	0.50	A selected mixture of 45 vision–language and document datasets, predominantly in English.
PDFA and LightOnOCR	0.30	PDF page images paired with extracted text and model-generated recaptions where available.
PixelProse	0.10	URL-fetched natural images paired with Gemini-generated captions.
DocAtlas	0.07	Rendered document images paired with Markdown targets in 82 languages.
Synthetic multilingual OCR	0.03	Synthetic document images and text in Japanese, Korean, simplified Chinese, and traditional Chinese.
Total	1.00

Appendix

Table 11: Pretraining visual–text source weights and descriptions.

Initialization and schedule
Setting	NeoMME -260M	NeoMME -800M
Pretraining initialization	Step 500,000 3	Step 500,000 4
Nodes	1	1
H100 accelerators	8	8
Steps	20,000	20,000
Sequences per rank	8	8

Appendix

Table 12: Retrieval-training settings.

Context	Formatting
Query prefix	[QUERY]
Document prefix	[DOC]
Image stream prefix	[IMG]
Patch-row boundary	[ROW]
Query expansion	[MASK]
Image patch at grid position $(r,c)$	$p_{r,c}$

Appendix

Table 13: NeoMME processor format. Colors mark structural roles rather than model inputs.

Source pool	Weight	Coverage
Filtered English retrieval	0.09	English, 10 candidates per query
mLateOn multilingual	0.73	Multilingual retrieval splits
Code retrieval	0.13	Code-search splits
Additional multilingual retrieval	0.03	13 languages, eight candidates per query
Organic retrieval (en) Organic retrieval (fr)	0.02	English and French
Total	1.00

Appendix

Table 14: Retrieval text-source weights.

Source	Weight	# Queries	# Documents	Languages
ColPali	0.25	148,123	118,195	en
Multilingual document images	0.25	351,651	280,679	en, it, fr, de, es
VisRAG	0.25	154,187	122,752	en
VisRAG synthetic	0.25	298,780	239,200	en
Total	1.00	952,741	760,826	6

Appendix

Table 15: Image retrieval mixture.

Source	NeoMME 3	ModernBERT 27 27 27 answerdotai/ModernBERT-base .	LFM2.5 28 28 28 LiquidAI/LFM2.5-Encoder-230M .	mmBERT 29 29 29 jhu-clsp/mmBERT-base .	EuroBERT 30 30 30 EuroBERT/EuroBERT-210m .
Vocabulary	131,072	50,368	65,536	256,000	128,256
English	0.1930	0.2075	0.2097	0.2044	0.2057
French	0.2122	0.2926	0.2434	0.2243	0.2652
German	0.2306	0.3194	0.2452	0.2165	0.2748
Spanish	0.2154	0.2942	0.2478	0.2136	0.2617
Italian	0.2265	0.3107	0.2685	0.2313	0.2830

Appendix

Table 16: Tokenizer compression on the 14 target languages in FLORES-200 devtest. All tokenizers encode identical texts without added special tokens. Values are tokens per UTF-8 byte, and lower values are better.

Source	NeoMME 3	ModernBERT 27	LFM2.5 28	mmBERT 29	EuroBERT 30
Vocabulary	131,072	50,368	65,536	256,000	128,256
English PDF	0.2439	0.2535	0.2552	0.2616	0.2390
Math	0.2510	0.2865	0.2889	0.2989	0.2792
Python	0.2148	0.3092	0.2869	0.4059	0.2428
JavaScript	0.2092	0.3086	0.2850	0.3964	0.2462
SQL	0.2665	0.3605	0.3533	0.3852	0.2924

Appendix

Table 17: Tokenizer compression on document, math, and code sources. All tokenizers encode the same 2,000 documents per source without added special tokens. Values are tokens per UTF-8 byte, and lower values are better.

Image	Predicted text
	This image displays: A pair of red Nike sneakers with black laces and white Nike logo accents. The sneakers are displayed on a black background.
	This image displays: A sunset over a river. The sky is orange and yellow, with a few clouds. The river is blue and still. There are no people in the image. The image is a photograph.
	Further Suspension and Restriction of Hearings June 8, 2021 Page 2 ## Out of Custody Trials All out of custody trials will be suspended. Counsel should discuss re-setting these matters for trial and advise the Court of the new trial date on the date currently set for trial [...]
	# Main Terms of TERP Settlement Proposal ## Settlement effective at closing of a jointly approved strategic transaction ### TERP shareholders participate ratably in transaction - After transaction, regardless of form, SUNE and all pre-transaction shareholders receive equal value per share --- 9 [...]

Appendix

Table 18: Selected image inputs and text generated by the NeoMME -260M model. [...] marks truncation.

Model details				ViDoRe
Model	Late-interaction	Params. (B)	Emb. dim.	v1 nDCG@5	v2 nDCG@5	v3 nDCG@10
<300M
BiModernVBERT 31 31 31 ModernVBERT/bimodernvbert ( Teiletche et al., 2026 ) . †		0.25	768	–	–	0.1541
ColModernVBERT 13	✓	0.25	128	0.8058 $\ddagger$	0.4068 $\ddagger$	0.2612 †
ColSmol-256M 14 †	✓	0.26	128	0.7974	0.3482	0.2073
NeoMME -260M, dense 15 $\ddagger$		0.26	1,024	0.7552	0.4075	0.3907

Appendix

Table 19: Visual document retrieval performance on the ViDoRe benchmarks for all compared models. A dash marks a result that was not reported. Emb. dim. is the per-token vector width for late-interaction models and the single-vector width for dense models.

Longest side (pixels)	Square-page representation		ViDoRe v1		ViDoRe v2		ViDoRe v3
Longest side (pixels)	Vectors	Raw float32 (MB)	260M	800M	260M	800M	260M	800M
768	602	0.31	0.7231	0.7771	0.3897	0.4454	0.3340	0.4049
1,024	1,058	0.54	0.8278	0.8533	0.4843	0.5251	0.4562	0.5029
1,536	2,354	1.21	0.8633	0.8775	0.5419	0.5728	0.5134	0.5496
2,048	4,162	2.13	0.8666	0.8801	0.5505	0.5814	0.5226	0.5560

Appendix

Table 21: Retrieval quality and uncompressed square-page representation size across image resolutions. All results use the evaluation protocol in subsection 5.6 .

<300M
Model	Params.	ViDoRe v3 domains
Model	Params.	HR	Fin.-EN	Industrial	Pharma.	CS	Energy	Physics	Fin.-FR
ColModernVBERT 13 †	250M	0.1982	0.2876	0.1567	0.3219	0.3761	0.3251	0.2262	0.1977
ColSmol-256M 14 †	256M	0.1646	0.2323	0.1287	0.2785	0.2880	0.2483	0.1614	0.1568
NeoMME -260M 15 $\ddagger$	260M	0.5520	0.5639	0.3989	0.5963	0.6743	0.5922	0.4246	0.3782
300M to 1B

Appendix

Table 22: ViDoRe v3 nDCG@10 results by task.

Model details		ViDoRe (nDCG@10)
Model	Dense width	v3	v2	v1
NeoMME -260M 15	128	0.3607	0.4063	0.7461
	256	0.3761	0.4133	0.7568
	512	0.3864	0.4228	0.7668
	1,024	0.3907	0.4334	0.7691
NeoMME -800M 18	128	0.3974	0.4332	0.7786

Appendix

Table 24: nDCG@10 results on ViDoRe v3, v2, and v1 at every trained Matryoshka representation size. All results use the evaluation protocol in subsection 5.6 .

Panel of Figure 16 — Figure 16: Image indexing throughput over the resolution grid on NVIDIA H100 and Apple M5 Pro.

Model	Processor behavior	Squares shown (pixels)	Vectors at cap
NeoMME -260M 15	Encodes each requested square	768 / 1,024 / 1,536 / 2,048	4,162
NeoMME -800M 18	Encodes each requested square	768 / 1,024 / 1,536 / 2,048	4,162
ColModernVBERT 13	Fixed 2,048-pixel tiling	2,048	1,149
ColSmol-256M 14	Fixed 2,048-pixel tiling	2,048	1,139
Vultron Flash 17	Caps at its visual-token budget	768 / 1,024 / 1,344	1,775
Qwen3-VL-Embedding-2B 51 51 51 Qwen/Qwen3-VL-Embedding-2B .	Caps at its visual-token budget	768 / 1,024 / 1,344	1

Appendix

Table 26: Processor behavior and effective image settings in the indexing benchmark.

Model	Square (pixels)	Vectors per page	H100	L40S	M5 Pro
NeoMME -260M 15	2,048	4,162	76.8	51.3	3.2
NeoMME -800M 18	2,048	4,162	40.4	21.2	1.4
Vultron Flash 17	1,344	1,775	35.2	20.2	–
Qwen3-VL-Embedding-2B	1,344	1	15.5	8.1	0.4
ColModernVBERT 13	2,048	1,149	53.3	26.0	2.9 †
ColSmol-256M 14	2,048	1,139	50.7	24.8	2.6 †

Appendix

Table 27: Document-encoding throughput at each model’s largest effective square. All values use the same measurement protocol.

Figure 17: Batch-one text query-encoding latency for visual retrievers on a 128-core CPU host, one NVIDIA L40S, and an Apple M5 Pro. Bars show the mean, and error bars extend to the 95th percentile.

System	Workload	Measure	Baseline	LIK	Improvement
ColQwen2 55 55 55 vidore/colqwen2-base .	MaxSim, batch 128	Peak memory	7.81 GiB	61 MiB	$130\times$ lower
ColQwen2 55 55 55 vidore/colqwen2-base .	Whole training	Maximum batch	64	128	$2\times$ higher
PyLate 56 56 56 lightonai/GTE-ModernColBERT-v1 .	Whole training, batch 1,024	Peak memory	62.18 GiB	57.67 GiB	7% lower
PyLate 56 56 56 lightonai/GTE-ModernColBERT-v1 .	Whole training, batch 1,024	Step time	6.02 s	4.81 s	$1.25\times$ faster

Appendix

Table 29: Selected LIK training results from the LIK benchmark report.

Multimodal Foundation Models

Multimodal foundation models learn shared capabilities across text, images, audio, video, and other signals for understanding, generation, and action.

See all 349 papers

Mar 13, 2026

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval

Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32

\times

fewer parameters and 50

\times

lower CPU query latency, at a total training cost under 13 GPU-hours.

Zhuchenyang Liu, Yao Zhang, Yu Xiao

Aalto University, Espoo, Finland

Visual Document Retrieval Vision-Language Retrieval VLM Distillation Efficient VLM Inference

Ask PDF

Sep 28, 2026

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around

10^{22}

FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

Lin Chen, Bolin Ni, Qi Yang +6

CASIA · UCAS · Foundation Model Department, Tencent

Multimodal Pretraining Multimodal Large Language Models Scaling Laws

Ask PDF

Apr 13, 2026

Bottleneck Tokens for Unified Multimodal Retrieval

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).

Siyu Sun, Jing Ren, Zhaohe Liao +8

Shanghai Jiao Tong University · ByteDance

Multimodal IR Multimodal Embedding Multimodal Large Language Models Multimodal Token Compression

Ask PDF

Abstract

Multimodal Foundation Models

Explore similar work

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Bottleneck Tokens for Unified Multimodal Retrieval