Seeing the Options: Typed Decisions over Visual Candidates with Reusable Option Memory

Fengxin Li

cs.AIOct 24, 2026

Seeing the Options: Typed Decisions over Visual Candidates with Reusable Option Memory

Authors: Fengxin Li

Organizations: Harbin Institute of Technology, Shenzhen

Abstract

Agents end many steps in a typed decision over a set of candidates: which screen element to click, which product to pick. System One models answer in one forward pass, and Jeva showed that four properties the setting demands - a "none of the above" probability, invariance to candidate order, scalability in the number of candidates, and reuse of candidate computation - follow from encoding the request and each candidate separately and letting a request state read a candidate memory. Jeva is text-only, and existing multimodal System One models (OneJev, Jev-Omni) put images in the request while candidates stay text in letter slots. This paper asks what happens to the four properties, and to accuracy, when the candidates themselves are images. Iris admits image, text, or mixed candidates and leaves Jeva's decision module untouched. On a visual-candidate benchmark built from Multimodal-Mind2Web (every candidate is the element's screenshot crop, K=16, three splits), independent encoding reaches 78.4/72.1/74.8% with a flip rate of exactly zero, while joint encoding (OneJev-4B fine-tuned on the same data) reaches 90.6/89.5/91.6% but changes its answer on 8-10% of records when the crops are reordered. Most of the gap is the candidate not seeing the request: a third design that encodes each candidate after the state - but never after another candidate - keeps order invariance exactly, shares the state pass through a forked cache, and reaches 92.1/87.0/90.2% at a third of OneJev's latency with a calibrated null probability (AUROC 0.93-0.97). Letting pooled candidates see each other (set attention, token-level reads, multi-vector memory) buys nothing, because the encoder maps visually distinct elements to vectors within cosine 0.999 of each other. Reusing image candidates across requests cuts latency 37x at K=256 and 191x at K=2048. Code: https://github.com/lllfx/Iris

Figures & tables

Panel of Figure 1 — Figure 1: Jeva encodes the request and each text candidate independently and lets the state read a candidate memory. Iris keeps that decision module and the four properties it guarantees, and lets the state and every candidate carry images: a screenshot crop, a chart, a product photo. The vision tower that Jeva loads and discards is used; nothing in the decision path changes.

Design	Order-invariant	Cacheable	Null	Encodings / request	$K$ in practice
Letter slots ( OneJev , Liu et al.)	no	no	no	$K$	$\leq 16$ – $256$
Runoff (inference-time)	by construction	no	no	$K+m^{2}$	tens
Bi-encoder (MetaEncoder)	approx.	candidates only ∗	no	$(1{-}r)K$	millions (open set)
Jeva	structural	both sides	yes	$(1{-}r)K$	thousands (text)
Iris , independent	structural	both sides	yes	$(1{-}r)K$	thousands (images)
Iris , state-conditioned	structural	within a request	yes	$1+K$ (state once)	thousands (images)

Table 1: How System One designs handle visual candidates.

r

: fraction of candidates already cached. ∗ With the candidate list inside the request (needed for closed-set accuracy) the request itself is not reusable. The state-conditioned Iris encoder (§ 3.5 ) shares the state pass across a request’s candidates and questions but not across requests.

Panel of Figure 2 — Figure 2: What a candidate’s vector may depend on. (a) Joint encoding: the answer depends on the order and number of candidates and nothing is reusable. (b) Independent encoding: $u_{i}$ is a function of $c_{i}$ alone, so the decision is order-invariant, unbounded in $K$ and cacheable across requests (Proposition 1 ). (c) State-conditioned encoding: each candidate continues from a fork of the state’s cache—order invariance and null handling are unchanged, the state pass is shared within a request, and cross-request reuse is given up. (d) Readouts that let the pooled candidate vectors see each other keep every guarantee of (c) and, in our experiments, add nothing (§ 5.5 ).

Figure 3: A Multimodal-Mind2Web step as Iris sees it. Left: the task, the last previous actions and the question form the text state. Right: each candidate (14 here, after deduplication) is the element’s own crop with a short label; the gold element is outlined in red, at a shuffled position.

			Accuracy (%)			Flip rate (%)
Model	Candidates	Train data	Task	Website	Domain	Task	Website	Domain
Jeva -4B (reported)	HTML text	Jeva curriculum	79.99	62.25	66.23	0.00	0.00	0.00
OneJev -4B	labels (page in state)	OneJev-Data	26.40	24.88	24.20	60.13	67.75	69.97
OneJev -4B	crops in state	OneJev-Data	37.85	37.46	34.83	50.64	52.80	54.30
OneJev -4B retrained	crops in state	+ Mind2Web crops	90.58	89.48	91.59	7.63	10.13	7.99
Iris -4B zero-shot	crops	–	31.11	25.07	32.73	0.00	0.00	0.00

Table 2: Element selection on Multimodal-Mind2Web with

K{=}16

: first-choice accuracy and decision flip rate over five candidate orders (original, reversed, three seeded random), on the cross-task / cross-website / cross-domain splits. Jeva ’s row uses text candidates and is the reference point for the same positives. OneJev cannot take image candidates: “labels” gives it each candidate’s text label with the page screenshot in the state (its native setting); “crops in state” lists every candidate’s crop inside the state and keeps letter slots for the candidates, the only way a letter-slot model can look at the crops; “retrained” fine-tunes OneJev -4B with LoRA on the same 7,751 training steps in that format (Appendix A ). Trained rows report the final checkpoint; the “selected” row is the periodic checkpoint with the best cross-task accuracy (epoch 2, step 50; Appendix C.2 ), so its cross-task number (

\dagger

) is a selection-split number, not a test number; the “abstention stage” row continues that checkpoint for one epoch on a null-oversampled subset of the training steps, and the released row applies to it an abstention margin selected on cross-task (§ 5.4 ); the last three rows add candidate-side readouts to these models and are discussed in § 5.5 . Chance is 6.25%. For Iris , all-order accuracy equals first-order accuracy on every split: a record answered correctly is answered correctly in every order.

Figure 4: Accuracy against order dependence on Mind2Web, one marker per model and split. The two released Iris models sit at a flip rate of exactly zero; the retrained OneJev buys 1–3 points of accuracy on two splits with an 8–10% flip rate, and the released OneJev answers largely by position.

Figure 5: The collapse diagnosis on cross-website (released conditioned model). Left: cosine between the gold candidate’s option vector and its hardest negative, by outcome (bars: 10th–90th percentile). Right: pixel RMSE between the gold crop and the chosen crop on wrong records, and to the nearest negative on correct ones (bars: quartiles). The crops differ; the vectors do not.

Model	JevBench orig.	JevBench easy	JevBench hard	ImaJev-Bench dev
OpenJev-27B (joint encoding; reported)	98.61	100.0	76.58	–
MetaEncoder-30B (bi-encoder; reported)	94.44	100.0	73.87	85.55
Iris -4B ( Jeva -4B weights, no training)	100.0	100.0	60.36	70.13
Iris -4B stage 1 (OneJev-Data)	95.83	100.0	48.65	74.68
Iris -4B stage 2 (+ Mind2Web crops only)	75.00	100.0	29.73	61.04
Iris -4B mixture (Mind2Web + M-BEIR + OneJev-Data replay)	73.61	97.92	44.14	65.58

Table 3: Accuracy (%) on the typed-decision benchmarks used by MetaEncoder. Reported numbers are from [ 4 ] , Table 2. ImaJev-Bench: the 154 dev rows with public labels (of 173). Iris ’s flip rate is 0.00 on every split.

	MSCOCO i2t	VisNews i2t	VisDial	MSCOCO t2i	VisNews t2i	WebQA	EDIS	Wiki-SS-NQ	CIRR	NIGHTS	OVEN	FashionIQ	mean
first stage (Qwen3-VL-Emb-2B)	74.6	75.4	81.5	78.0	73.1	90.1	92.6	83.7	65.1	69.2	76.5	38.8	74.9
Qwen3-VL-Reranker-8B (supervised) [ 9 ]	77.7	79.6	89.5	82.2	75.1	92.7	92.4	86.4	74.1	70.8	78.6	41.2	78.4
Qwen3.8-27B choice, zero-shot [ 9 ]	70.5	74.2	84.8	80.0	76.2	92.4	95.0	87.3	72.0	68.8	78.0	41.9	76.8
Iris -4B stage 1, zero-shot	41.4	40.2	15.4	21.6	27.2	34.4	42.1	13.5	4.4	4.9	41.6	5.5	24.4
Iris -4B mixture (retrieval-tuned)	47.6	49.4	30.4	44.8	47.0	82.0	69.3	48.8	4.5	6.0	49.3	4.6	40.3

Table 4: Reranking the top 16 of Qwen3-VL-Embedding-2B on MMEB-V2 image retrieval (hit@1, 1,000 queries per task). Iris ’s flip rate is 0.00 on every task. The zero-shot row has never seen a retrieval candidate; the tuned row is the released mixture model (§ 5.8 ), trained on 4,000 M-BEIR train queries of each of ten tasks (all but VisDial and Wiki-SS-NQ, which measure transfer) with hard negatives from the same first stage.

Figure 6: MMEB-V2 reranking per task. Retrieval tuning helps wherever the query or the candidate is text and does nothing where both are images: a per-candidate vector cannot carry a comparison with the query image.

	AUROC of $p(\bot)$			Abstention on null steps (%)
Model	Task	Website	Domain	Task	Website	Domain
Iris -4B mixture (independent)	0.867	0.780	0.785	11.1	13.6	13.1
Iris -4B conditioned, stage 2	0.943	0.919	0.947	40.7	43.2	33.8
Iris -4B conditioned + abstention stage	0.969	0.935	0.949	79.0	77.3	77.5

Table 5: Null detection on the Mind2Web null steps (81 / 44 / 222 steps per split).

Model	Accuracy (all)	noul MAE	noul Brier	score MAE
Iris -4B ( Jeva -4B weights, no training)	53.0	0.386	0.199	0.934
Iris -4B stage 1 (OneJev-Data)	74.1	0.263	0.127	0.597
Iris -4B mixture	67.5	0.312	0.153	0.654

Table 6: Typed outputs on 2,000 held-out OneJev-Data rows.

Figure 7: Latency against

K

with none, half, or all of the image candidates already cached (solid: measured; dashed: Proposition 2 with the measured constants). At full reuse a request costs the state pass plus host-side assembly whatever

K

is.

$K$	$r{=}0$	$r{=}0.25$	$r{=}0.5$	$r{=}0.75$	$r{=}1$	$S(1)$
	p50 latency, measured / predicted (ms)
16	208 / 201	166 / 164	147 / 127	135 / 90	57 / 54	3.6 $\times$
64	652 / 628	497 / 485	339 / 341	198 / 197	58 / 54	11 $\times$
256	2,342 / 2,275	1,768 / 1,720	1,208 / 1,164	645 / 609	64 / 54	37 $\times$
1,024	12,108 / 11,907	9,118 / 8,944	6,083 / 5,980	3,051 / 3,017	91 / 54	133 $\times$
2,048	23,894 / 23,894	17,835 / 17,934	11,909 / 11,974	5,933 / 6,014	125 / 54	191 $\times$

Table 7: Latency under candidate reuse:

K

image candidates of which a fraction

r

is already in the hybrid cache, fresh state every request; one A800, 5 repetitions. Predicted is

T(r)

of Proposition 2 with the measured constants.

S(1)

is the measured speedup at full reuse.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Qwen3.5-4B (vision tower + language model), bf16, SDPA attention
Vision tower	frozen; patch embedding run as a matrix product (§ 3.6 )
LoRA	rank 64, $\alpha$ 128, dropout 0.05, on q/k/v/o and gate/up/down projections of the language model; initialised from Jeva -4B
Pooling / projection	last token; two linear maps $4096\to 768$ , no bias
Decision stack	$N{=}1$ block, 8 heads, SwiGLU ratio 4; $\tau_{\text{set}}$ init 0.15, $\tau_{\text{dec}}$ 0.10, $\tau_{\text{cov}}$ 0.10
Pixel budget	$50{,}176$ – $200{,}704$ px per image ( $\approx$ 64–256 tokens)
Token limits	state 16,384 (stage 1) / 4,096 (stage 2 and evaluation); candidate 2,048 / 1,024

Appendix

Table 8: Settings. Everything not listed is Jeva -4B’s default.

$\delta$	cross-task accuracy	positive accuracy	abstention on null steps
$-0.2$	91.02	91.64	81.5
$0$ (saved decision)	91.70	92.52	79.0
$+0.1$	91.77	92.75	76.5
$+0.24$ (selected)	92.07	93.23	74.1
$+0.4$	91.92	93.23	71.6
cross-website at $\delta{=}0$ / $+0.24$	86.53 / 87.02	86.9 / 87.5	77.3 / 77.3

Appendix

Table 9: Abstention margin for the released conditioned model (%).

Figure 8: Cross-task accuracy of the conditioned stage-2 checkpoints: total, on steps with a positive, and correct abstention on the 81 null steps.

Checkpoint	Rows seen	Accuracy	Positive	Null
epoch 1, step 100	3,200	86.69	91.5	12.3
epoch 1, step 150	4,800	88.33	92.0	32.1
epoch 1, step 200	6,400	87.51	91.4	27.2
epoch 2, step 50	9,351	89.75	92.4	48.1
epoch 2, step 100	10,951	89.68	91.2	65.4
epoch 2, step 150	12,551	88.71	90.8	55.6

Appendix

Table 10: Cross-task accuracy (%) of the conditioned stage-2 checkpoints, original order. From 1.2 epochs on, accuracy on steps with a positive is stable and the total is driven by the volatile abstention rate on the 81 null steps.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 487 papers

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
Oct 8, 2026Jianpeng Cheng, Guangyu Sun, Aashu Singh +11Text Embedding Models Cross-Modal Retrieval
Meta AI
Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling
Oct 15, 2026Tong Zhu, Zhe Xu, Zefeng He +7LLM Decision-Making Efficient Multimodal Inference
Shanghai AI Lab · Shanghai Jiao Tong University · Fudan University +1
Jeva: System One Models from First Principles
Oct 13, 2026Qi Li, Xinchao WangLLM Decision-Making Efficient Language Model Inference
National University of Singapore

Oct 8, 2026J/K move · Enter open · S save

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang+10

Meta AI

Text Embedding Models Cross-Modal Retrieval Multimodal Understanding

Open paper AskPDF arXiv

System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.

Abstract

Efficient VLM Inference

Explore similar work

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

Jeva: System One Models from First Principles