What Can't Jev Decide? Probing the Limits of One-Pass Decision Models

Wang Yang; Xinpeng Li; Yanyan Zhang; Chuang Ma; Shouren Wang; Jianzhe Wei; Liqin Ye; Xiaoran Cai

cs.CLOct 18, 2026

What Can't Jev Decide? Probing the Limits of One-Pass Decision Models

Authors: Wang Yang, Xinpeng Li, Yanyan Zhang, Chuang Ma, Shouren Wang, Jianzhe Wei, Liqin Ye, Xiaoran Cai,

Organizations: Case Western Reserve University · NII LLMC (Japan) · Georgia Institute of Technology · Columbia University

Abstract

Jev and its open replicas choose among the options a caller supplies with a single forward pass, fast enough to decide inside an agent loop. We ask what such a model cannot decide, and find a simple limit: Jev cannot make a decision that an LLM makes only by writing out intermediate steps. On a benchmark that varies one factor at a time, Jev finds and selects records without error up to 20k tokens and 64 options, but once the answer depends on two or more updates of one user, accuracy falls, to 53-80% at sixteen updates, because similar records interfere. Given a stale summary of the state, Jev follows it. On real decision items graded by how many thinking tokens a reference LLM needs, Jev answers 97.5% of the items that one forward pass of that LLM solves, but only 74% of the others. Jev also does not know its limit: its confidence flags the errors it makes itself but not those it inherits from a stale input, and neither an explicit nor a trained hand-off option catches the latter. The open models we evaluate share these limits. Checking the state such models rely on requires a model that reasons.

Figures & tables

Panel of Figure 1 — Figure 1: Where Jev’s limit lies. Jev decides what one record says, but not what several records imply together or whether a state it is given is correct; an LLM that writes intermediate steps decides all three. Accuracy on the controlled benchmark up to 8k tokens: event lookup, final state after 16 updates, and rejection of a stale summary (Table 2 ).

Panel of Figure 2 — Figure 2: The three task families. Thin marks are log entries or records; blue marks belong to the target. Options are drawn from the context, so every wrong option also occurs in it.

Figure 3: Jev 1.13.0 on the controlled benchmark. (a) Retrieval is perfect at every depth and length. (b) Final state fails from two updates on, although event lookup on the same logs is perfect. (c) With a stale summary appended, the share of answers that repeat the stale state; only items whose options include the stale state, which is rare for $k\geq 8$ . At $k=1$ the stale state is the empty set. Blue panels show accuracy (higher is better); the orange panel shows a failure rate (higher is worse). Each cell: 10 scenarios $\times$ 3 option orders. All panels in %; colour scales start at 25%, chance with four options.

	$k\geq 2$			$k=16$
Variant	Jev	K-27B	K-4B	Jev	K-27B	K-4B
P0 original	80	90	66	61	86	41
P1 step by step	81	90	67	61	88	39
P2 procedure	79	92	65	54	87	38
P3 name failure	78	88	62	64	86	38
R1 target only	100	100	85	100	100	67

Table 1: Final-state accuracy in % when the question (P1–P3) or the log (R1–R4) is changed. R3 and R4 have the same length and target positions as P0; R4 replaces the other users’ entries by essay text. Jev: 1k–20k tokens; Kev-27B (K-27B) and Kev-4B (K-4B): up to 4k. Kev-9B is in Appendix B .

	Find and select			Final state			With summary		Verify
Model	Retr.	Select	Event	$k{=}1$	$k{\geq}2$	$k{=}16$	correct	stale	reject stale
Jev 1.13.0	100	100	100	100	84	64	100	58	70
Open-Jev-27B	100	100	100	100	89	78	100	72	74
OpenJev-27B	100	100	100	100	95	93	100	66	56
Kev-27B †	100	100	100	100	91	88	100	60	62
Kev-4B †	100	100	100	98	66	41	100	53	38

Table 2: Accuracy in % of Jev and representative open models on contexts up to 8k tokens. “Select” pools 2–64 options. “With summary” is final-state accuracy when a correct or a stale summary is appended; “reject stale” is the share of stale summaries judged incorrect (chance 50%). † Kev-4B and Kev-27B accept at most 4k tokens and are evaluated up to 4k. ‡ Not a decision model: the same LLM generates its reasoning before answering; shown for reference and run on the state-tracking items only. All models are in Appendix B .

Panel of Figure 4 — Figure 4: Two real-task items, with the state abbreviated. (a) An item that needs a short calculation, which Jev gets wrong in one pass. (b) One of the 93 items excluded because their gold label is disputed: both models give the same other label.

Figure 5: Accuracy on the 738 real-task items against the number of thinking tokens Qwen3.8-27B needs to answer them. Jev and the direct readout answer in one forward pass; the thinking model is averaged over 10 samples. Items are selected by the thinking model (Section 4 ), so its curve is an upper reference, not a competitor. Error bars: 95% Wilson intervals.

	By demand		By source
Model	One pass	Reasoning	J-easy	J-original	J-hard	PubMedQA	Banking77	All
	646	92	48	72	108	275	235	738
Jev 1.13.0	97.5	73.9	100.0	98.6	73.1	98.5	97.4	94.6
Kev-27B	97.4	69.6	100.0	100.0	72.2	96.7	97.4	93.9
llm2jev-27B	95.4	68.5	100.0	95.8	74.1	94.5	94.5	92.0
LitJev-27B	95.4	63.0	100.0	98.6	75.9	93.5	91.9	91.3

Table 3: Accuracy in % on the 738 real-task items; the second header row gives the number of items after the exclusions above. “One pass”: items Qwen3.8-27B’s direct readout answers correctly; “Reasoning”: items it does not. J-: JevBench tiers. The reference rows define the split and the item selection, so their first two columns and the thinking row hold by construction. All models and all items are in Appendix B .

Figure 6: What confidence and a hand-off option catch. (a) AUROC of

1-p_{\max}

for a model’s own final-state errors and for errors after a stale summary; synthetic logs up to 4k tokens; dashed line: chance. (b) Share of real-task items routed on a held-out source (J: JevBench, P: PubMedQA, B: Banking77) by a threshold on the untrained model’s confidence, chosen on the other two sources to match the trained option’s rate there, and by the trained hand-off option. Banking77 is part of the Kev training data. (c) Share of errors handed off on the synthetic logs before and after hand-off training. (b) and (c): mean over three seeds.

Model	Acc.	Rate	Errors	Option	Threshold
		Hand-off		Acc. of the rest
Jev 1.13.0	94.6	1.6	8	94.5	95.5
Kev-27B	93.9	9.6	20	93.9	97.0
LitJev-27B	91.3	3.1	16	92.3	93.6
llm2jev-27B	92.0	3.8	14	93.5	94.1
JevK5-9B	90.2	3.1	10	90.8	92.6

Table 4: Hand-off on the 738 real-task items; all columns in %. Acc.: accuracy without the hand-off option. Rate: share of items handed off when the option is offered; Errors: share of the model’s errors (without the option) that it hands off. Acc. of the rest: accuracy on the items not handed off, by the option or by a threshold on the confidence of the untrained model that routes the same share. Bottom: our training runs, mean over three seeds; “(ours)” reproduces the released recipe and “+ hand-off” adds hand-off training. Kev-0.8B is in Appendix B .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Backbone	How the answer is read	Input limit
Jev 1.13.0 ( TypeSafe AI, 2026 )	undisclosed	undisclosed (hosted API)	32k, state + longest question
Kev 0.8B, 4B, 9B, 27B ( Palmer, 2026 )	Qwen3.5-Base; Qwen3.8-27B	LoRA and a pointer head over the options	4B, 27B: $<$ 8k
JevK5 4B v0.2, v0.3; 9B v0.3.3 ( alibiserikbay, 2026 )	Qwen3.5-4B, 9B	merged LoRA, option-letter logits	–
Decider 0.8B–35B-A3B ( Mapika, 2026 )	Qwen3.5-Base	fine-tuned, one forward pass	32,768 per row
Open-Jev-27B v1.1 ( Zefan-Cai, 2026 )	Qwen3.8-27B	fine-tuned, one forward pass	16,384
OpenJev-27B ( OpenJev, 2026 )	Qwen3.8-27B	fine-tuned, one forward pass	16,384; $\leq$ 52 options

Appendix

Table 5: Models and how they are served; repositories are given in the references. Input limit: the largest input the model accepts in our runs, in tokens of its own tokenizer; “–”: no input in our benchmarks exceeded it. OpenJev-SGLang is run with the unquantised Qwen3.6-35B-A3B ( Qwen Team, 2026b ) rather than the NVFP4 default of its repository.

	Find and select				Final state			With summary		Verify		AUROC
Model	Retr.	Sel. $\leq$ 26	Sel. 32–64	Event	$k{=}1$	$k{\geq}2$	$k{=}16$	correct	stale	accept	reject	own	stale
Jev 1.13.0	100	100	100	100	100	84	64	100	58	92	70	0.83	0.57
Kev-27B †	100	100	100	100	100	91	88	100	60	100	62	0.90	0.62
Kev-9B	100	100	100	100	100	70	54	100	55	81	26	0.70	0.38
Kev-4B †	100	100	100	100	98	66	41	100	53	88	38	0.76	0.29
Kev-0.8B	86	99	97	66	18	32	18	91	25	66	41	0.56	0.46

Appendix

Table 6: Controlled benchmark, all models, contexts up to 8k tokens; accuracies and shares in %, AUROC in

[0,1]

. Groups: Jev; fine-tuned; training-free; plain Qwen with the letter-logit readout (no items with more than 26 options); contrastive. Verify: share of correct summaries accepted and of stale summaries rejected. AUROC of

1-p_{\max}

for the model’s own final-state errors and for errors after a stale summary. † Up to 4k tokens. ‡ Generates its reasoning; state-tracking items only, no probabilities.

Variant	$k\geq 2$	$k=16$
P0 original	69	48
P1 step by step	69	56
P2 procedure	69	56
P3 name failure	67	48
R1 target only	71	50
R2 newest first	20	7

Appendix

Table 7: Final-state accuracy in % of Kev-9B under the variants of Table 1 , up to 4k tokens.

	JevBench						Pooled
Model	easy	original	hard	PubMedQA	Banking77	HelpSteer2	831 items	738 items
	48	72	111	300	300	300
Jev 1.13.0	100.0	98.6	71.2	92.0	80.3	41.7	86.0	94.6
Kev-27B	100.0	100.0	70.3	89.7	83.3	44.3	86.3	93.9
Kev-4B	100.0	93.1	51.4	83.7	82.7	44.0	80.7	86.4
JevK5-4B v0.2	100.0	94.4	73.9	85.0	69.7	35.7	79.7	87.7

Appendix

Table 8: Real-task accuracy in % before exclusion; the second header row gives the number of items. 831 items: all sources except HelpSteer2. 738 items: additionally without the 93 disputed items (Section 4 ). The sampled modes are averaged over 10 samples.

		Real tasks			Final state
	Backbone	All	One pass	Reasoning	$k\geq 2$
4B	Base	87.2 $\pm$ 0.5	91.8 $\pm$ 0.4	54 $\pm$ 3	70 $\pm$ 5
4B	Reasoning	87.4 $\pm$ 0.4	91.4 $\pm$ 0.3	58 $\pm$ 1	50 $\pm$ 3
9B	Base	90.1 $\pm$ 0.2	94.1 $\pm$ 0.2	60 $\pm$ 3	76 $\pm$ 0
9B	Reasoning	90.1 $\pm$ 0.5	94.2 $\pm$ 0.5	59 $\pm$ 2	71 $\pm$ 2

Appendix

Table 9: Kev trained with the same recipe from Qwen3.5-Base or from the post-trained, thinking Qwen3.5; accuracy in %, mean

\pm

standard deviation over three seeds. Real tasks: the 738 items, restricted to those every seed of a size answers (717 at 4B, 712 at 9B). Final state: Section 3 , up to 4k tokens.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 237 papers

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
Oct 8, 2026Xing Li, Qingcheng Chang, Jinzhong Ning +5LLM Evaluation Multiple-Choice Question Answering
Dalian Maritime University · Dalian University of Technology
General Decision Models: Benchmarking and Insights Beyond Jev
Oct 2, 2026Feiyu Duan, Jiayu Lin, Jia Wang +6LLM Decision-Making Uncertainty Calibration
Shanghai Innovation Institute · Fudan University · Tongji University +2
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Oct 1, 2026Yinheng Li, Justin WagleLLM Inference LLM Fine-Tuning
Microsoft

Oct 8, 2026J/K move · Enter open · S save

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu+4

Dalian Maritime University · Dalian University of Technology

LLM Evaluation Multiple-Choice Question Answering

Open paper AskPDF arXiv

Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.

Abstract

LLM Decision-Making

Explore similar work

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

General Decision Models: Benchmarking and Insights Beyond Jev

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them