From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

Pritam Deka

cs.CLOct 1, 2026

From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

Authors: Pritam Deka

Organizations: Queen’s University Belfast Belfast, United Kingdom

Abstract

Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. A System One request pairs a state with typed questions. A biomedical sentence encoder, converted with one of four heads (§ 3 ), returns a calibrated distribution over each question’s options in one forward pass, and downstream code thresholds it to act or escalate. Example values are illustrative.

Task	Family	Type	#Train	#Test
Grounded
PubMedQA-L	Evidence QA	choice	400	500
PubMedQA-A §	Evidence QA	noul	20,000	–
SciFact	Claim verif.	choice	780	321
HealthVer	Claim verif.	choice	10,073	1,823
PUBHEALTH †	Claim verif.	choice	–	1,231

Table 1: BioDecide and MEDLINE-S1 , in decisions (a state may carry several questions).

\dagger

test only (never trained on);

\ddagger

held-out task family;

\lx@sectionsign

training only. Historical calibration slices are carved from the training data before training and are not counted. Clinical training counts apply only to the +clinical variant. Counts describe the original splits; the integrity audit reports exclusions separately.

		Seen tasks, acc ${}_{\text{cn}}$			Held-out, acc ${}_{\text{cn}}$		Clinical	ECE ${}_{\text{TS}}$
Encoder	Type	C	PFR	C, 10%	C	PFR	C	C
PubMedBERT	MLM	67.3 $\pm$ 0.7	66.9 $\pm$ 0.8	59.1 $\pm$ 0.6	29.5 $\pm$ 3.3	20.7 $\pm$ 0.6	20.7 $\pm$ 0.8	4.8 $\pm$ 0.9
S-PubMedBERT-MS-MARCO	contr.	67.7 $\pm$ 0.9	65.3 $\pm$ 1.5	61.0 $\pm$ 1.4	32.8 $\pm$ 1.5	23.6 $\pm$ 1.5	20.5 $\pm$ 1.8	3.9 $\pm$ 0.7
MedCPT-Query	contr.	17.9 $\pm$ 4.3	60.4 $\pm$ 0.7	15.0 $\pm$ 10.2	3.9 $\pm$ 0.5	23.6 $\pm$ 3.2	12.5 $\pm$ 3.4	14.0 $\pm$ 2.3
MPNet	MLM	66.4 $\pm$ 0.9	63.3 $\pm$ 0.4	61.5 $\pm$ 0.5	30.2 $\pm$ 2.7	20.8 $\pm$ 2.5	22.7 $\pm$ 5.2	4.8 $\pm$ 0.4
all-MPNet-base-v2	contr.	66.4 $\pm$ 0.9	67.8 $\pm$ 0.8	60.5 $\pm$ 2.5	27.5 $\pm$ 0.8	24.7 $\pm$ 2.3	13.7 $\pm$ 5.0	5.2 $\pm$ 0.3

Table 2: Main results (

\times 100

; mean

\pm

sd over 3 seeds; 10% also uses 3 seeds). Encoders are grouped by matched pair (MLM parent, then contrastive children). acc

{}_{\text{cn}}

: chance-normalised accuracy, averaged over question groups within a task and then over tasks. Seen: the 8 grounded training tasks. Held-out: PUBHEALTH, BIOSSES, MTSamples. Clinical: MedNLI, MIMIC-III trial questions and MIMIC-IV name gate, zero-shot except “+clinical train”. ECE

{}_{\text{TS}}

: top-label ECE after temperature scaling on seen tasks. All rows are trained with the released RLCD (PG) recipe except “C trained with CE”. In the lower block the C columns hold the row’s own model. Knowledge-track results, raw ECE and all architectures per encoder are in Appendix G .

Figure 2: Seen-task accuracy versus training data for the six parent-retriever comparisons (grey: MLM parent, blue: retrieval child; solid: C, dashed: PFR). Error bars show the sd over three seeds.

	Z = PFR at step 0		C at step 0
Encoder	content	label	content	label
PubMedBERT	1.0	13.0	1.1	24.0
S-PubMedBERT-MS-MARCO	15.2	$-$ 20.4	2.3	$-$ 2.8
MedCPT-Query	19.8	$-$ 20.2	3.5	17.6
MPNet	1.0	3.6	$-$ 1.2	12.5
all-MPNet-base-v2	21.2	$-$ 1.2	$-$ 1.2	12.1

Table 3: Untrained (step-0) chance-normalised accuracy (

\times 100

, after temperature scaling on the calibration split) on questions whose options differ in content (MEDLINE design and disease area, MTSamples, exam answers, Druglib and BIOSSES rubrics) or only by a label word ( noul , PubMedQA, verification, DDI, MedNLI). C at step 0: 3 seeds.

	Seen acc ${}_{\text{cn}}$		Held-out acc ${}_{\text{cn}}$		ECE ${}_{\text{TS}}$
Objective	PFR	C	PFR	C	PFR	C
CE	68.3	70.2	20.8	31.0	4.0	3.8
Proper	68.5	70.7	22.6	28.8	4.4	3.9
PG (released RLCD)	65.3	67.7	23.6	32.8	3.8	3.9
PG w/o CE	65.5	–	20.3	–	4.0	–
PG-LOO (unbiased)	67.8	69.2	23.8	30.0	3.9	3.7

Table 4: Head

\times

objective on S-PubMedBERT-MS-MARCO (

\times 100

, mean over 3 seeds; sd and further metrics in Table 7 ). PG is the released RLCD recipe, used for all other trained models. PG-LOO replaces its same-sample mean baseline and reward normalisation with a leave-one-out baseline. PG w/o CE was run for PFR only.

Figure 3: Retrospective routing to Gemma-4-31B: the least-confident fraction of questions (by temperature-scaled confidence) is escalated. Lines: mean over three encoder seeds; bands: sd.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task	State	Question(s) and type	Options and target	Splits
PubMedQA-L	abstract contexts without the conclusion	the original question ( choice )	yes / no / maybe with short descriptions; one-hot	1,000 labelled items: 500 test, 50 dev, 50 calibration, rest train (seeded)
PubMedQA-A	abstract contexts	the original question ( noul )	yes $\to$ true; one-hot	20k sampled for training, excluding every evaluation PMID
SciFact	title and abstract of the evidence (or first cited) document	“Does this abstract support or refute the claim: c ” ( choice )	supports / refutes / not_enough_info	official train; validation as test; calibration grouped by claim
HealthVer	evidence passage	“Does this evidence support or refute the claim: c ” ( choice )	as SciFact	official train / dev / test; calibration grouped by topic
PUBHEALTH	article text (fact-checkers’ explanation removed)	“Based on this article, what is the veracity of the health claim: c ” ( choice )	false / mixture / true / unproven	official test only
DDI-2013	sentence with the two drugs marked	“What kind of interaction between @DRUG1 $and @DRUG2$ does the sentence describe?” ( choice )	mechanism / effect / advice / int / none	official dev as test (test is gated); test sentences seen in training removed

Appendix

Table 5: Conversion of each public source dataset into typed decisions. Every training set loses a calibration slice of

\min(400,\max(10,\lfloor n/10\rfloor))

states, carved before any run and used only for temperature fitting. Clinical conversions are listed in the text below.

		Slope vs. $\nabla J_{\sigma}$			Score : CE weight		Cosine	SNR
Checkpoint (head, objective)	Estimator	$\sigma{=}0.4$	$0.25$	$0.1$	$\sigma{=}0.4$	$0.1$	$\sigma{=}0.25$	$\sigma{=}0.25$
PFR, PG	released PG	3.59	5.72	14.27	4.74	18.87	0.982	0.861
	mean baseline, unnormalised	0.75	0.75	0.75	0.99	1.00	0.982	0.816
	PG-LOO	1.01	1.00	1.00	1.33	1.33	0.983	0.791
	pathwise (Reparam)	1.00	1.00	1.00	1.32	1.32	1.000	206
PFR, PG-LOO	released PG	3.66	5.84	14.57	4.83	19.23	0.982	0.854

Appendix

Table 6: Matched gradient-estimator probe. For each saved checkpoint, the logits of the same 40 training batches (32 items each) are fixed, and 32 independent estimates of the smoothed-score gradient (

G{=}4

, no CE term) are drawn per estimator and noise level. Slope: projection of the mean estimate onto a 2,048-sample pathwise reference

\nabla J_{\sigma}

(1 is the correct scale). Score : CE weight: slope

\times\lVert\nabla J_{\sigma}\rVert/\lVert\nabla\mathrm{CE}\rVert

, the effective weight of the smoothed-score term relative to the CE term in a training step. SNR: per-item

\lVert\nabla J_{\sigma}\rVert^{2}

divided by the scale-matched variance of one estimate.

Figure 5: Matched probe, averaged over the three checkpoints of Table 6 . Left: scale of each estimator relative to the true smoothed-score gradient (dotted line: correct scale). Right: per-item signal-to-noise ratio after matching scales. The released estimator’s scale grows as

\sigma

anneals; all score-function estimators have SNR below one, the pathwise estimator two to three orders of magnitude more.

Figure 6: Median global gradient norm before clipping during training (S-PubMedBERT-MS-MARCO, three seeds, rolling median). The clipping threshold is 1 (dotted), so every step of every objective is clipped and update length is fixed; the released PG recipe’s norm is an order of magnitude above CE and grows as

\sigma

anneals.

Objective	Head	Seen	Held-out	Clinical	Knowledge	ECE	ECE ${}_{\text{TS}}$	NLL ${}_{\text{TS}}$	MAE ${}_{\texttt{score}}$	Perm. flip	$\lVert\nabla\rVert$
CE	PFR	68.3 $\pm$ 0.4	20.8 $\pm$ 1.6	21.9 $\pm$ 2.9	13.1 $\pm$ 0.8	9.3 $\pm$ 0.9	4.0 $\pm$ 0.4	0.510 $\pm$ 0.004	0.588 $\pm$ 0.012	0.4 $\pm$ 0.1	7.1
	C	70.2 $\pm$ 0.4	31.0 $\pm$ 1.7	20.0 $\pm$ 4.1	13.6 $\pm$ 0.9	7.4 $\pm$ 0.3	3.8 $\pm$ 0.2	0.491 $\pm$ 0.001	0.639 $\pm$ 0.020	2.9 $\pm$ 0.4	6.0
Proper	PFR	68.5 $\pm$ 0.2	22.6 $\pm$ 1.9	23.4 $\pm$ 0.7	13.6 $\pm$ 0.4	9.1 $\pm$ 0.6	4.4 $\pm$ 0.4	0.510 $\pm$ 0.006	0.582 $\pm$ 0.004	0.2 $\pm$ 0.1	8.7
	C	70.7 $\pm$ 0.3	28.8 $\pm$ 0.4	22.6 $\pm$ 1.1	12.8 $\pm$ 0.7	7.6 $\pm$ 0.5	3.9 $\pm$ 0.3	0.493 $\pm$ 0.013	0.627 $\pm$ 0.007	2.6 $\pm$ 0.2	7.7
PG (released RLCD)	PFR	65.3 $\pm$ 1.5	23.6 $\pm$ 1.5	18.3 $\pm$ 8.2	11.4 $\pm$ 1.2	7.0 $\pm$ 0.6	3.8 $\pm$ 0.2	0.548 $\pm$ 0.018	0.638 $\pm$ 0.016	0.4 $\pm$ 0.1	88.1
	C	67.7 $\pm$ 0.9	32.8 $\pm$ 1.5	20.5 $\pm$ 1.8	11.2 $\pm$ 0.7	6.5 $\pm$ 0.0	3.9 $\pm$ 0.7	0.527 $\pm$ 0.010	0.746 $\pm$ 0.031	3.2 $\pm$ 0.5	50.2

Appendix

Table 7: Complete head

\times

objective grid on S-PubMedBERT-MS-MARCO (mean

\pm

sd over 3 seeds; accuracy, ECE and flip rates

\times 100

). Columns 3–6: chance-normalised accuracy by track. ECE, NLL: seen tasks, top-label, before and after temperature scaling. MAE

{}_{\texttt{score}}

: error of the expected Druglib level. Perm. flip: share of seen-task answers that change under a random option permutation.

\lVert\nabla\rVert

: median global gradient norm before clipping; the clipping threshold is 1, so every step of every objective is clipped.

Figure 7: Top-label reliability diagrams on the seen tasks (items pooled over tasks and seeds; 10 bins, bins with fewer than 30 items omitted), before (dashed) and after (solid) temperature scaling. Pooled ECE differs from the task-averaged ECE in the tables. Fine-tuned encoders are over-confident before scaling and close to the diagonal after it; the zero-shot LLM remains miscalibrated.

Contrast ( $a-b$ )	Seeds	Seen $\Delta$ [95% CI]	$p_{\text{Holm}}$	Held-out $\Delta$ [95% CI]	$p_{\text{Holm}}$
C full: S-PubMedBERT $-$ PubMedBERT	3/3	+0.4 [ $-$ 0.4, +1.1]	1	+3.2 [+0.3, +6.1]	.956
C full: all-MPNet $-$ MPNet	3/3	+0.0 [ $-$ 0.8, +0.9]	1	$-$ 2.6 [ $-$ 5.6, +0.3]	1
C full: Nomic-embed $-$ ModernBERT	3/3	$-$ 1.0 [ $-$ 1.9, $-$ 0.1]	.708	+2.5 [ $-$ 1.2, +6.2]	1
C full: GTE-ModernBERT $-$ ModernBERT	3/3	$-$ 2.5 [ $-$ 3.5, $-$ 1.5]	.012	$-$ 2.7 [ $-$ 5.9, +0.6]	1
C full: BioClinical-MB-emb $-$ BioClinical-MB	3/3	$-$ 0.2 [ $-$ 1.0, +0.5]	1	+0.3 [ $-$ 3.2, +3.7]	1
PFR full: S-PubMedBERT $-$ PubMedBERT	3/3	$-$ 1.6 [ $-$ 2.5, $-$ 0.8]	.014	+2.9 [ $-$ 0.1, +6.1]	1

Appendix

Table 8: Initialisation contrasts (contrastive child minus MLM parent) for every available pair, head and training-data fraction. Document-clustered paired bootstrap (10,000 draws; differences in chance-normalised points). Intervals are pointwise and condition on the observed tasks and on seed-averaged correctness. Holm adjustment is applied within each family (RQ1; RQ2; RQ3; baselines) across both tracks. The smallest attainable unadjusted

p

is 0.0002.

Contrast ( $a-b$ )	Seeds	Seen $\Delta$ [95% CI]	$p_{\text{Holm}}$	Held-out $\Delta$ [95% CI]	$p_{\text{Holm}}$
C $-$ PFR (CE)	3/3	+1.9 [+0.8, +3.1]	.011	+10.2 [+5.8, +14.6]	.003
C $-$ PFR (Proper)	3/3	+2.2 [+1.1, +3.4]	.004	+6.2 [+2.0, +10.5]	.041
C $-$ PFR (PG)	3/3	+2.4 [+1.2, +3.6]	.003	+9.2 [+5.0, +13.4]	.003
C $-$ PFR (LOO)	3/3	+1.5 [+0.3, +2.6]	.079	+6.2 [+2.0, +10.3]	.041
C $-$ PFR (Reparam)	3/3	+1.3 [+0.1, +2.4]	.136	+3.8 [ $-$ 0.6, +8.2]	.247
B $-$ C (PG)	3/3	$-$ 0.6 [ $-$ 1.8, +0.7]	.353	$-$ 9.3 [ $-$ 13.5, $-$ 5.0]	.003

Appendix

Table 9: Conversion (RQ2), objective (RQ3) and baseline contrasts on S-PubMedBERT-MS-MARCO. Document-clustered paired bootstrap (10,000 draws; differences in chance-normalised points). Intervals are pointwise and condition on the observed tasks and on seed-averaged correctness. Holm adjustment is applied within each family (RQ1; RQ2; RQ3; baselines) across both tracks. The smallest attainable unadjusted

p

is 0.0002.

Encoder	Arch	Seen acc ${}_{\text{cn}}$	ECE	ECE ${}_{\text{TS}}$	Held-out	Clinical	Knowledge
PubMedBERT	C	67.3 $\pm$ 0.7	6.7 $\pm$ 0.6	4.8 $\pm$ 0.9	29.5 $\pm$ 3.3	20.7 $\pm$ 0.8	13.4 $\pm$ 0.7
	PFR	66.9 $\pm$ 0.8	8.3 $\pm$ 0.4	4.1 $\pm$ 0.3	20.7 $\pm$ 0.6	23.9 $\pm$ 0.6	12.5 $\pm$ 0.9
	B	67.0 $\pm$ 0.5	8.3 $\pm$ 0.4	4.1 $\pm$ 0.1	17.8 $\pm$ 6.5	23.6 $\pm$ 1.4	11.5 $\pm$ 0.7
S-PubMedBERT-MS-MARCO	C	67.7 $\pm$ 0.9	6.5 $\pm$ 0.0	3.9 $\pm$ 0.7	32.8 $\pm$ 1.5	20.5 $\pm$ 1.8	11.2 $\pm$ 0.7
	PFR	65.3 $\pm$ 1.5	7.0 $\pm$ 0.6	3.8 $\pm$ 0.2	23.6 $\pm$ 1.5	18.3 $\pm$ 8.2	11.4 $\pm$ 1.2
	B	67.1 $\pm$ 0.5	7.8 $\pm$ 0.6	3.9 $\pm$ 0.2	23.4 $\pm$ 1.4	25.9 $\pm$ 2.1	11.8 $\pm$ 0.7

Appendix

Table 10: All encoders and architectures (

\times 100

; mean

\pm

sd over 3 seeds).

	Seen (grounded training tasks)								Held-out
Model	PubMedQA	SciFact	HealthVer	DDI	HoC	ADE	Druglib	MEDLINE	PUBHEALTH	BIOSSES	MTSamples
PubMedBERT C (PG)	51.6	51.2	48.8	89.3	88.1	87.3	35.6	86.5	23.1	0.4	65.2
S-PubMedBERT C (PG)	47.6	55.3	49.1	91.0	88.7	88.0	35.4	86.4	24.2	5.4	68.7
S-PubMedBERT C (CE)	47.7	60.6	55.0	92.4	89.1	88.1	41.6	87.0	20.5	1.7	70.9
S-PubMedBERT PFR (PG)	44.9	34.9	52.9	91.5	85.8	86.9	42.8	82.7	17.0	2.5	51.3
S-PubMedBERT PFR (CE)	46.9	39.9	61.9	93.3	87.3	87.3	45.7	83.8	8.4	2.5	51.5

Appendix

Table 11: Per-task chance-normalised accuracy (

\times 100

; mean over seeds; question groups averaged within a task). The training objective is given in parentheses.

	Perm.	Trial Qs: answer (acc ${}_{\text{cn}}$ )				Trial Qs: answerable (acc ${}_{\text{cn}}$ )				Name gate AUROC
Model	flip %	trunc	win-mean	win-conf	RtD	trunc	win-mean	win-conf	RtD	test	common-word
PubMedBERT C	16.6	2.4	3.4	2.3	5.9	4.5	$-$ 3.7	$-$ 2.8	7.9	72.9	71.8
S-PubMedBERT C	15.7	0.0	$-$ 3.8	0.1	$-$ 1.1	7.1	1.0	4.0	20.2	72.6	71.4
S-PubMedBERT PFR	0.3	8.2	15.0	16.0	7.1	4.7	$-$ 0.6	$-$ 1.5	5.8	74.8	74.5
S-PubMedBERT PFR, frozen	13.1	$-$ 0.5	$-$ 0.2	2.9	0.4	7.8	0.4	2.3	29.9	75.0	74.1
S-PubMedBERT B	0.0	10.9	19.7	20.5	8.3	9.8	4.2	4.7	3.8	77.6	77.7

Appendix

Table 12: Robustness (

\times 100

; mean over 3 seeds where available). Perm. flip: share of answers that change under a random option permutation (5 permutations), averaged over all evaluation splits; on the seen tasks alone C heads flip 2–3% and PFR below 0.5% (Table 7 ). Trial Qs: MIMIC-III trial-eligibility questions over notes longer than the 512-token context, decided by truncation, sliding windows (mean of logits or most confident window) or retrieve-then-decide (RtD). Name gate: MIMIC-IV windows, standard and common-word-name splits. Both LLMs reach AUROC 100.0 on both name-gate splits.

Model	Historical	Filtered	Difference
S-PubMedBERT C	67.69	67.32	$-$ 0.37
S-PubMedBERT PFR (PG)	65.30	64.91	$-$ 0.40
S-PubMedBERT PFR (CE)	68.26	67.89	$-$ 0.37
S-PubMedBERT PFR (Reparam)	68.53	68.19	$-$ 0.35
BioClinical-ModernBERT C	73.10	72.81	$-$ 0.29
Laya-large FT	74.99	74.69	$-$ 0.30

Appendix

Table 13: Seen-task accuracy sensitivity to removal of exact test inputs appearing in public training/calibration data, and repeated test inputs. Seed means, in chance-normalised points. Checkpoints and temperatures are unchanged; this does not repair calibration contamination.

	nDCG@10 ( $\times 100$ )		Latency
Model (S-PubMedBERT)	SciFact	NFCorpus	ms
Z (no training)	67.6	31.3	10.2
B	34.3	11.0	9.7
C	43.3	13.0	6.6
C, trained with CE	24.6	10.3	6.6
PFR, shared prior	35.6	11.2	15.2

Appendix

Table 14: Retrieval retention of the pooled embedding after conversion (seed 0), and p50 latency of one request with 10 questions at batch size 1 on an MI300X (base-encoder latencies measured on the PubMedBERT architecture, which S-PubMedBERT shares). Times cover the model forward pass only, excluding tokenisation and host/device transfers.

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 224 papers

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
Oct 8, 2026Jianpeng Cheng, Guangyu Sun, Aashu Singh +11Text Embedding Models Cross-Modal Retrieval
Meta AI
System One models Are Not Better Rerankers
Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM Reranking Information Retrieval
University of Passau, Passau, Germany · Interdisciplinary Transformation University Austria (IT:U), Linz, Austria
Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages
May 28, 2026David Rey-Blanco, Roberto CruzData Augmentation Healthcare
TietAI

Oct 8, 2026J/K move · Enter open · S save

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang+10

Meta AI

Text Embedding Models Cross-Modal Retrieval Multimodal Understanding

Open paper AskPDF arXiv

System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.

Abstract

Text Classification

Explore similar work

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

System One models Are Not Better Rerankers

Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages