MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Jianpeng Cheng; Guangyu Sun; Aashu Singh; Benyu Zhang; Haixing Dai; Hossein Mansour; Jiangfan Zhang; Shlok Kumar Mishra

cs.CLOct 8, 2026

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Authors: Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, +6 more

Organizations: Meta AI

Abstract

System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.

Figures & tables

Capability	Benchmark	Modality	Candidates	Tasks	Metric
Decision making	JEVBench	Text	Closed	3	Accuracy
Decision making	ImaJEV-Bench	Multimodal	Closed	3	Accuracy
Multimodal understanding	MMLU	Text	Closed	57	Hit@1
	MMMU	Multimodal	Closed	30	Hit@1
	Video-MMMU	Multimodal	Closed	3	Hit@1
	NaturalBench	Multimodal	Closed	1	Hit@1

Table 1: Evaluation benchmarks. Tasks counts the scored task units for each suite. Closed-set tasks score each request against its own option set (fewer than 256 candidates). Open-set tasks score against a shared and larger candidate pool.

Category	Benchmark	MetaEncoder	OpenJev (27B)	WeMM-Embedding (9B)	Qwen3-VL-Embedding (8B)
Decision making	JevBench (orig)	0.9444	0.9861	0.6250	0.6389
	JevBench (easy)	1.0	1.0	0.9792	0.9375
	JevBench (hard)	0.7387	0.7658	0.4865	0.4324
	Jev Intelligence	81.4	84.5	45.8	42.1
Multimodal decision making	ImaJev (dev)	0.8555	–	0.8117	0.7792
General knowledge understanding	MMLU	0.7767	0.8601	0.7325	0.6544

Table 2: Benchmark Evaluation Results for MetaEncoder compared to SOTA System One decision makers and SOTA retrieval models.

Benchmark	Without options	With options	$\Delta$
MMLU (57 subjects)	0.5645	0.7456	$+0.1811$
MMMU (30 subjects)	0.4875	0.5629	$+0.0754$
MMMU-Pro (4 options)	0.4699	0.5553	$+0.0854$
MMMU-Pro (10 options)	0.3503	0.4237	$+0.0734$

Table 3: Effect of appending candidate option lists to the request prompt. Scores reflect unweighted subject means (MMLU, MMMU) or Hit@1 (MMMU-Pro).

Benchmark	C0	C1	C2	$\Delta_{\text{C1}}$	$\Delta_{\text{C2}}$
MMLU (57 subjects)	0.7456	0.6512	0.7767	$-0.0944$	$+0.0311$

Table 4

Explore similar work

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Jul 28, 2026Mingqiao Ye, Zhaochong An, Zhitong Gao +11Unified Multimodal Models Decoder-Only Language Models
1EPFL · University of Copenhagen · 4CUHK +3
MAD-Guard: Controlled Study of Autoregressive Generation versus Direct Decision Interfaces for Closed Multimodal Forensic Tasks
Sep 27, 2026Hao ChenEfficient Multimodal Inference Image Forgery Detection
School of Cyber Security, Guangdong Polytechnic Normal University, Guangzhou, China
M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal Grounding Efficient Multimodal Inference
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

Jul 28, 2026, cs.CVJ/K move · Enter open · S save

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu+10

1EPFL · University of Copenhagen · 4CUHK+3

Unified Multimodal Models Decoder-Only Language Models Multimodal Generation

Open paper AskPDF arXiv

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

Abstract

Explore similar work

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

MAD-Guard: Controlled Study of Autoregressive Generation versus Direct Decision Interfaces for Closed Multimodal Forensic Tasks

M3^33Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions