Jev and its open replicas choose among the options a caller supplies with a single forward pass, fast enough to decide inside an agent loop. We ask what such a model cannot decide, and find a simple limit: Jev cannot make a decision that an LLM makes only by writing out intermediate steps. On a benchmark that varies one factor at a time, Jev finds and selects records without error up to 20k tokens and 64 options, but once the answer depends on two or more updates of one user, accuracy falls, to 53-80% at sixteen updates, because similar records interfere. Given a stale summary of the state, Jev follows it. On real decision items graded by how many thinking tokens a reference LLM needs, Jev answers 97.5% of the items that one forward pass of that LLM solves, but only 74% of the others. Jev also does not know its limit: its confidence flags the errors it makes itself but not those it inherits from a stale input, and neither an explicit nor a trained hand-off option catches the latter. The open models we evaluate share these limits. Checking the state such models rely on requires a model that reasons.
Figures & tables
Figure 1: Where Jev’s limit lies. Jev decides what one record says, but not what several records imply together or whether a state it is given is correct; an LLM that writes intermediate steps decides all three. Accuracy on the controlled benchmark up to 8k tokens: event lookup, final state after 16 updates, and rejection of a stale summary (Table 2 ).
Figure 2: The three task families. Thin marks are log entries or records; blue marks belong to the target. Options are drawn from the context, so every wrong option also occurs in it.
Figure 3: Jev 1.13.0 on the controlled benchmark. (a) Retrieval is perfect at every depth and length. (b) Final state fails from two updates on, although event lookup on the same logs is perfect. (c) With a stale summary appended, the share of answers that repeat the stale state; only items whose options include the stale state, which is rare for k≥8 . At k=1 the stale state is the empty set. Blue panels show accuracy (higher is better); the orange panel shows a failure rate (higher is worse). Each cell: 10 scenarios × 3 option orders. All panels in %; colour scales start at 25%, chance with four options.
k≥2
k=16
Variant
Jev
K-27B
K-4B
Jev
K-27B
K-4B
P0 original
80
90
66
61
86
41
P1 step by step
81
90
67
61
88
39
P2 procedure
79
92
65
54
87
38
P3 name failure
78
88
62
64
86
38
R1 target only
100
100
85
100
100
67
Table 1: Final-state accuracy in % when the question (P1–P3) or the log (R1–R4) is changed. R3 and R4 have the same length and target positions as P0; R4 replaces the other users’ entries by essay text. Jev: 1k–20k tokens; Kev-27B (K-27B) and Kev-4B (K-4B): up to 4k. Kev-9B is in Appendix B .
Find and select
Final state
With summary
Verify
Model
Retr.
Select
Event
k=1
k≥2
k=16
correct
stale
reject stale
Jev 1.13.0
100
100
100
100
84
64
100
58
70
Open-Jev-27B
100
100
100
100
89
78
100
72
74
OpenJev-27B
100
100
100
100
95
93
100
66
56
Kev-27B †
100
100
100
100
91
88
100
60
62
Kev-4B †
100
100
100
98
66
41
100
53
38
Table 2: Accuracy in % of Jev and representative open models on contexts up to 8k tokens. “Select” pools 2–64 options. “With summary” is final-state accuracy when a correct or a stale summary is appended; “reject stale” is the share of stale summaries judged incorrect (chance 50%). † Kev-4B and Kev-27B accept at most 4k tokens and are evaluated up to 4k. ‡ Not a decision model: the same LLM generates its reasoning before answering; shown for reference and run on the state-tracking items only. All models are in Appendix B .
Figure 4: Two real-task items, with the state abbreviated. (a) An item that needs a short calculation, which Jev gets wrong in one pass. (b) One of the 93 items excluded because their gold label is disputed: both models give the same other label.
Figure 5: Accuracy on the 738 real-task items against the number of thinking tokens Qwen3.8-27B needs to answer them. Jev and the direct readout answer in one forward pass; the thinking model is averaged over 10 samples. Items are selected by the thinking model (Section 4 ), so its curve is an upper reference, not a competitor. Error bars: 95% Wilson intervals.
By demand
By source
Model
One pass
Reasoning
J-easy
J-original
J-hard
PubMedQA
Banking77
All
646
92
48
72
108
275
235
738
Jev 1.13.0
97.5
73.9
100.0
98.6
73.1
98.5
97.4
94.6
Kev-27B
97.4
69.6
100.0
100.0
72.2
96.7
97.4
93.9
llm2jev-27B
95.4
68.5
100.0
95.8
74.1
94.5
94.5
92.0
LitJev-27B
95.4
63.0
100.0
98.6
75.9
93.5
91.9
91.3
Table 3: Accuracy in % on the 738 real-task items; the second header row gives the number of items after the exclusions above. “One pass”: items Qwen3.8-27B’s direct readout answers correctly; “Reasoning”: items it does not. J-: JevBench tiers. The reference rows define the split and the item selection, so their first two columns and the thinking row hold by construction. All models and all items are in Appendix B .
Figure 6: What confidence and a hand-off option catch. (a) AUROC of 1−pmax for a model’s own final-state errors and for errors after a stale summary; synthetic logs up to 4k tokens; dashed line: chance. (b) Share of real-task items routed on a held-out source (J: JevBench, P: PubMedQA, B: Banking77) by a threshold on the untrained model’s confidence, chosen on the other two sources to match the trained option’s rate there, and by the trained hand-off option. Banking77 is part of the Kev training data. (c) Share of errors handed off on the synthetic logs before and after hand-off training. (b) and (c): mean over three seeds.
Hand-off
Acc. of the rest
Model
Acc.
Rate
Errors
Option
Threshold
Jev 1.13.0
94.6
1.6
8
94.5
95.5
Kev-27B
93.9
9.6
20
93.9
97.0
LitJev-27B
91.3
3.1
16
92.3
93.6
llm2jev-27B
92.0
3.8
14
93.5
94.1
JevK5-9B
90.2
3.1
10
90.8
92.6
Table 4: Hand-off on the 738 real-task items; all columns in %. Acc.: accuracy without the hand-off option. Rate: share of items handed off when the option is offered; Errors: share of the model’s errors (without the option) that it hands off. Acc. of the rest: accuracy on the items not handed off, by the option or by a threshold on the confidence of the untrained model that routes the same share. Bottom: our training runs, mean over three seeds; “(ours)” reproduces the released recipe and “+ hand-off” adds hand-off training. Kev-0.8B is in Appendix B .
Table 5: Models and how they are served; repositories are given in the references. Input limit: the largest input the model accepts in our runs, in tokens of its own tokenizer; “–”: no input in our benchmarks exceeded it. OpenJev-SGLang is run with the unquantised Qwen3.6-35B-A3B ( Qwen Team, 2026b ) rather than the NVFP4 default of its repository.
Find and select
Final state
With summary
Verify
AUROC
Model
Retr.
Sel. ≤ 26
Sel. 32–64
Event
k=1
k≥2
k=16
correct
stale
accept
reject
own
stale
Jev 1.13.0
100
100
100
100
100
84
64
100
58
92
70
0.83
0.57
Kev-27B †
100
100
100
100
100
91
88
100
60
100
62
0.90
0.62
Kev-9B
100
100
100
100
100
70
54
100
55
81
26
0.70
0.38
Kev-4B †
100
100
100
100
98
66
41
100
53
88
38
0.76
0.29
Kev-0.8B
86
99
97
66
18
32
18
91
25
66
41
0.56
0.46
Appendix
Table 6: Controlled benchmark, all models, contexts up to 8k tokens; accuracies and shares in %, AUROC in [0,1] . Groups: Jev; fine-tuned; training-free; plain Qwen with the letter-logit readout (no items with more than 26 options); contrastive. Verify: share of correct summaries accepted and of stale summaries rejected. AUROC of 1−pmax for the model’s own final-state errors and for errors after a stale summary. † Up to 4k tokens. ‡ Generates its reasoning; state-tracking items only, no probabilities.
Variant
k≥2
k=16
P0 original
69
48
P1 step by step
69
56
P2 procedure
69
56
P3 name failure
67
48
R1 target only
71
50
R2 newest first
20
7
Appendix
Table 7: Final-state accuracy in % of Kev-9B under the variants of Table 1 , up to 4k tokens.
JevBench
Pooled
Model
easy
original
hard
PubMedQA
Banking77
HelpSteer2
831 items
738 items
48
72
111
300
300
300
Jev 1.13.0
100.0
98.6
71.2
92.0
80.3
41.7
86.0
94.6
Kev-27B
100.0
100.0
70.3
89.7
83.3
44.3
86.3
93.9
Kev-4B
100.0
93.1
51.4
83.7
82.7
44.0
80.7
86.4
JevK5-4B v0.2
100.0
94.4
73.9
85.0
69.7
35.7
79.7
87.7
Appendix
Table 8: Real-task accuracy in % before exclusion; the second header row gives the number of items. 831 items: all sources except HelpSteer2. 738 items: additionally without the 93 disputed items (Section 4 ). The sampled modes are averaged over 10 samples.
Real tasks
Final state
Backbone
All
One pass
Reasoning
k≥2
4B
Base
87.2 ± 0.5
91.8 ± 0.4
54 ± 3
70 ± 5
Reasoning
87.4 ± 0.4
91.4 ± 0.3
58 ± 1
50 ± 3
9B
Base
90.1 ± 0.2
94.1 ± 0.2
60 ± 3
76 ± 0
Reasoning
90.1 ± 0.5
94.2 ± 0.5
59 ± 2
71 ± 2
Appendix
Table 9: Kev trained with the same recipe from Qwen3.5-Base or from the post-trained, thinking Qwen3.5; accuracy in %, mean ± standard deviation over three seeds. Real tasks: the 738 items, restricted to those every seed of a size answers (717 at 4B, 712 at 9B). Final state: Section 3 , up to 4k tokens.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.
Xing Li, Qingcheng Chang, Jinzhong Ning +5
Dalian Maritime University · Dalian University of Technology
General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on τ-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.
Feiyu Duan, Jiayu Lin, Jia Wang +6
Shanghai Innovation Institute · Fudan University · Tongji University +2
Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without training. Fine-tuning provides targeted gains for smaller models and complex routing, while our KL anchors prevent behavioral degradation in general text generation.