General Decision Models: Benchmarking and Insights Beyond Jev

Feiyu Duan; Jiayu Lin; Jia Wang; Jun Xiang; Jialiang Wu; Xinnong Zhang; Hanqi Yan; Siyuan Wang

cs.CLOct 2, 2026

General Decision Models: Benchmarking and Insights Beyond Jev

Authors: Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, +1 more

Organizations: Shanghai Innovation Institute · Fudan University · Tongji University · King’s College London · Chinese University of Hong Kong

Abstract

General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on $τ$ -bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.

Figures & tables

Figure 1: A conceptual view of general and decisional intelligence. The left panel contrasts open-ended language capabilities with decision-making in constrained output spaces. The right panel illustrates the evolution from expert systems and task-specific models to encoder models, chat and reasoning LLMs, and general decision models represented by Jev.

Figure 2: Overview of JEVal. (a) Data processing pipeline. (b) Share of the 11,257 JEVal instances contributed by each domain. (c) Input length over JEVal measured over the state, instruction, and candidates in cl100k_base tokens. The frequency axis is logarithmic.

Model	ACC (%) $\uparrow$	ECE $\downarrow$	$\mathbf{Brier}_{\mathbf{raw}}$ $\downarrow$
General Decision models
Jev	78.38	0.0532	0.287
Kev-0.8b	49.63	0.0432	0.617
Kev-4b	68.90	0.0843	0.435
Kev-9b	70.87	0.0579	0.404
Kev-27b	77.17	0.0356	0.304

Table 1: The strongest decision models match thinking LLMs in accuracy and are better calibrated than every generative LLM. We evaluate 25 configurations on the 10,938 choice and noul questions of JEVal. Accuracy counts invalid responses as incorrect, while ECE and

\mathrm{Brier}_{\mathrm{raw}}

are computed over valid responses.

Figure 3: Decision models lead when the evidence is in the input and trail generative LLMs in medicine, finance, and law. We break the JEVal results of all 25 configurations down by the ten application domains. (a) Accuracy, (b) ECE, and (c)

\mathrm{Brier}_{\mathrm{raw}}

; N and T denote non-thinking and thinking. GKR: general knowledge and reasoning; MED: medicine; LAW: law; FIN: finance; CS: commonsense; ATU: agent and tool use; FH: factuality and hallucination; LCDU: long-context and document understanding; SS: social science; PP: personalization and preferences.

Figure 4: Local decision models are faster than hosted Jev on short inputs and slower on long ones. We measure mean latency on a 200-question test stratified into six input-length bins of up to 16K tokens. (a) 4B decision models, (b) 27B decision models, and (c) the other decision models. Jev’s latency is that of its hosted API and includes network transport, whereas local models run on RTX 4090 GPUs and exclude queueing and transport.

		Repeating division		Area inversion		Prior probability			Posterior probability
Model	Inference mode	Acc (%) $\uparrow$	Acc ±1 (%) $\uparrow$	Acc (%) $\uparrow$	Acc ±1 (%) $\uparrow$	Acc (%) $\uparrow$	$\bm{E}_{\mathrm{mode}}$ (pp) $\downarrow$	$\bm{E}_{\mathrm{all}}$ (pp) $\downarrow$	Acc (%) $\uparrow$	$\bm{E}_{\mathrm{mode}}$ (pp) $\downarrow$	$\bm{E}_{\mathrm{all}}$ (pp) $\downarrow$
Jev	Native	44	97	39	73	100	53.75	26.91	67	28.12	18.13
Qwen3.5-4B	Non-thinking	34	68	24	45	48	18.66	13.12	38	21.84	17.40
Qwen3.8-27B	Non-thinking	53	99	45	95	98	2.85	2.06	46	12.37	9.87
DeepSeek-V4-Flash	Non-thinking	60	100	69	98	66	9.90	6.54	43	18.67	14.99
Qwen3.5-4B	default think	72	73	82	84	81	0.41	0.38	82	0.74	0.63

Table 2: Jev lands near the exact answer and picks the most likely label, while thinking LLMs recover both the exact value and the probabilities. We construct four tasks of 100 items each, two with a unique numerical answer and two with an analytically determined distribution, and compare Jev with four LLMs, three of them in both inference modes. Acc and Acc ±1 are the share of answers that are exact and within one integer of the gold value;

E_{\mathrm{mode}}

and

E_{\mathrm{all}}

are the mean absolute errors, in percentage points, of the probability of the gold option and of all four options.

Figure 5: Jev ranks the options correctly but concentrates the probability on its top choice. We compare Jev’s output probabilities with analytically determined distributions over four options, using 100 items per task. Options are ordered by true probability. (a) Prior items, where no clue is observed. (b) Posterior items, where the probabilities follow from an observed clue.

Figure 6: BABILong

Figure 10: Jev

Env.	System	Pass (%)	Turns	P50 (s)	P95 (s)
Retail	Base (tool call)	86.1	13.22	257.3	661.9
Retail	Jev + LLM	73.9	13.90	147.1	702.3
Airline	Base (tool call)	58.0	13.68	277.8	795.2
Airline	Jev + LLM	48.0	12.35	242.6	1147.7
Overall	Base (tool call)	77.6	13.36	267.1	695.5
Overall	Jev + LLM	66.1	13.44	160.6	816.0

Table 3: With Jev selecting actions, the typical episode is faster, but fewer episodes succeed and the slowest ones take longer. We replace the action-selection step of the

\tau

-bench tool-calling agent with Jev, keep the same base LLM for arguments and responses, and run each test task once. Pass is the episode success rate, Turns is the mean number of agent turns, and P50 and P95 are percentiles of episode wall-clock time.

Compared System	Mean Jaccard $\uparrow$	Exact Match $\uparrow$
Jev 1.13	53.88%	15.03%
Qwen3.8-27B non-thinking	65.24%	25.06%
Qwen3.8-27B thinking	65.86%	28.84%

Table 4: Jev agrees less with the reference profiler than Qwen3.8-27B does. We profile 898 social-media accounts as multi-label sets over 20 interest categories and compare each system with a frozen Qwen3-Max reference. Jaccard similarity is computed per user and averaged, and an exact match requires identical sets of categories.

	Jev		Qwen3.5-4B				DeepSeek-V4-Flash-0731				Ours: SII-Decision-27B
	Native		Non-thinking		Thinking		Non-thinking		Thinking ( high )		Native
Domain	ACC $\uparrow$	Latency $\downarrow$	ACC $\uparrow$	Latency $\downarrow$	ACC $\uparrow$	Latency $\downarrow$	ACC $\uparrow$	Latency $\downarrow$	ACC $\uparrow$	Latency $\downarrow$	ACC $\uparrow$	Latency $\downarrow$
Citizenship	43.00	0.478	39.46	6.999	41.84	102.333	43.60	268.818	45.41	302.180	45.09	8.412
Environment	34.85	0.480	30.30	6.832	33.62	101.452	35.51	234.424	38.29	283.856	36.21	8.919
Family	37.23	0.493	29.17	6.728	33.55	106.760	40.31	260.248	42.94	328.831	40.77	8.490
Health	35.44	0.494	30.46	6.816	35.03	97.687	34.83	250.956	37.44	292.322	39.99	8.623

Table 5: Decision models predict individual survey responses about as accurately as generative LLMs at a fraction of their latency. We predict responses from demographic profiles on SocioBench, using the same 277,500 respondent–question pairs across ten domains for all systems. SII-Decision-27B is close to thinking DeepSeek-V4-Flash, and Jev to its non-thinking mode. Latency is the mean time in seconds per item.

Figure 14: SII-Decision-27B calls 47 of 51 states correctly, and all four misses are Trump-won states called for Harris. We simulate the 2024 U.S. presidential election in ElectionSim, with SII-Decision-27B driving about 330,000 voter agents sampled at a ratio of 1/1,000. (a) Actual two-party margin. (b) Simulated margin. Appendix E.3 lists the per-state results.

Figure 15: Overview of how we build a decision model from an open-weight LLM. (a) First-Token Decision Readout: the next-token distribution at the answer position is restricted to the labels of the valid options. One forward pass gives the selected option for choice , the probability of true for noul , and the expected level for score . (b) Reasoning-to-Readout Self-Distillation: the model first reasons, and its readouts at the end of the reasoning chains are averaged into a target distribution. The same model without reasoning is trained to match this target at its first answer token, with a KL term that keeps it close to the readout of the frozen original model.

Figure 16: SII-Decision-4B and SII-Decision-27B lie on the accuracy–latency Pareto frontier, and SII-Decision-27B exceeds Jev in accuracy at a fraction of its latency. We compare open reproductions of Jev, and Jev itself, on pooled JEVal accuracy and on mean latency for inputs of up to 1K tokens. Open models are measured on a single H100 GPU, and Jev through its hosted API, including network transport.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: Decision formats across all 11,257 instances. Slice labels show percentages; the legend gives counts.

Domain	Datasets	Subtasks	Instances	Share (%)	English	Chinese
General knowledge	12	13	2,598	23.1	2,198	400
Medicine	3	4	800	7.1	200	600
Law	3	9	1,800	16.0	1,400	400
Finance	2	2	59	0.5	59	0
Commonsense	3	3	600	5.3	600	0
Agents and tools	4	4	800	7.1	800	0

Appendix

Table 6: Domain-level composition of the final evaluation set. Language follows each source subtask’s designation, rather than a separate label for every input field. Empty planned domains are omitted.

Domain	Mean	Maximum
General knowledge	348	3,447
Medicine	136	653
Law	2,831	20,480
Finance	565	1,591
Commonsense	85	276
Agents and tools	267	3,523

Appendix

Table 7: Input lengths by domain in cl100k_base tokens. Each instance is capped at 20,480 tokens before calculating the mean and maximum for this descriptive table. Stored inputs above the cap are not claimed to have been truncated during evaluation.

Dataset and format	Evaluation instance
AI2 ARC Science question choice	State: It was once thought that living organisms could come from non-living matter. For example, people believed that flies would develop from rotting meat. Instruction: This idea was later disproved primarily because of Criteria: opt_1 : the discovery of the atom; opt_2 : better surgical techniques; opt_3 : continued experimentation; opt_4 : the invention of the microscope. Gold label: opt_3 (continued experimentation).
MedQA Medical examination choice	State: A 72-year-old anthropologist with long-standing hypertension visits your office for a routine exam. You notice an abnormality on his laboratory results caused by his regimen of captopril and triamterene. What abnormality did you most likely find? Instruction: Based on the question, choose the correct option. Criteria: A: Hyperkalemia; B: Hypernatremia; C: Thrombocytopenia; D: Anemia. Gold label: A (Hyperkalemia).
LexGLUE Unfair contract clause choice	State: We may modify this contract, our privacy policy and our cookies policies from time to time. Instruction: Which type of unfair clause does this terms-of-service excerpt contain? Choose None of the 8 if it contains none. Criteria: Limitation of liability; Unilateral termination; Unilateral change; Content removal; Contract by using; Choice of law; Jurisdiction; Arbitration; None of the 8. Gold label: Unilateral change.
HaluEval Response grounding noul	State: User query: “Sketch a logo for the company International Business Company.” Response: “Unfortunately, as an AI language model, I do not have the capability to generate visual content. Can I assist you with anything else?” Instruction: Does this response contain hallucinated content not supported by the query? Criteria: false : The response is faithful to the source and contains no hallucination; true : The response contains content not supported by the source (hallucinated). Gold label: false .
LongBench Paragraph retrieval choice	State (excerpt): Paragraph 16: “This species of lizard has a large head that is elongated and depressed, with the cheeks swollen in adult males.” Paragraph 17: “Biometrika was established in 1901 by Francis Galton, Karl Pearson, and Raphael Weldon to promote the study of biometrics.” The other 28 paragraphs are omitted here. Instruction (excerpt): Which paragraph in the passage below matches this description? The text describes the physical characteristics of a species of lizard. It focuses on details such as the large head with swollen cheeks in adult males, the length of the snout, … Criteria: Paragraph 1 through Paragraph 30 (30 candidate identifiers). Gold label: Paragraph 16.
JevBench Incident impact score	State: The icon is misaligned. Every function works. Instruction: Rate incident impact using only reported facts. Use the highest fully supported level. Criteria: 0: No function impaired; cosmetic only. 1: One user or a nonessential function impaired, with a workaround. 2: Many users blocked from a core function, no data loss. 3: Confirmed irreversible data loss or physical harm. Gold label: 0 (cosmetic only).

Appendix

Table 8: Representative JEVal test cases. Each row shows one evaluation instance in the shared decision format.

Figure 20: Prompt templates and response structures for the nine LLM configurations. Panel (a) presents the shared system message; panels (b)–(d) present the user templates and schematic responses for choice , noul , and score , respectively. Placeholders are instantiated per question. Formatting notes, value constraints, and the binary prediction rule are explanatory annotations, not additional prompt text.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
aida-testc $N=200$
ACC $\uparrow$	95.00	86.00	97.50	38.00	80.00	96.00	98.50	6.50
Latency $\downarrow$	2.11	5.80	14.02	18.15	57.79	94.05	55.95	6.38
ECE $\downarrow$	0.009	0.043	0.012	0.169	0.532	0.302	0.077	0.051
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.032	0.193	0.040	0.807	0.600	0.191	0.037	0.980

Appendix

Table 10: General knowledge and reasoning: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
aida-testc $N=200$
ACC $\uparrow$	13.00	83.50	19.00	87.50	94.00	1.50	42.00	16.00
Latency $\downarrow$	6.32	10.07	12.40	229.93	23.87	40.14	266.04	216.20
ECE $\downarrow$	0.203	0.314	0.625	0.542	0.046	0.001	0.406	0.125
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	1.004	0.376	1.387	0.521	0.094	1.000	0.980	0.977

Appendix

Table 11: General knowledge and reasoning: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
aida-testc $N=200$
ACC $\uparrow$	77.00	96.50	91.00	99.50	96.50	89.00	97.50	98.50	99.00
Latency $\downarrow$	25.21	90.29	54.52	45.27	13.75	74.46	248.88	206.46	211.86
ECE $\downarrow$	0.204	0.024	0.087	0.005	0.035	0.035	0.015	0.020	0.008
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	4.291	0.064	0.180	0.010	0.070	0.099	0.028	0.035	0.022

Appendix

Table 12: General knowledge and reasoning: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
cmb $N=200$
ACC $\uparrow$	84.50	67.50	80.50	33.50	63.50	68.00	76.00	36.50
Latency $\downarrow$	0.82	5.63	12.45	17.62	57.19	94.69	54.87	6.35
ECE $\downarrow$	0.056	0.095	0.048	0.066	0.076	0.104	0.145	0.073
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.198	0.425	0.269	0.740	0.484	0.424	0.366	0.714

Appendix

Table 13: Medicine: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
cmb $N=200$
ACC $\uparrow$	45.00	67.00	26.00	63.00	75.50	20.00	33.50	24.00
Latency $\downarrow$	6.35	9.72	12.62	233.39	23.79	39.32	19.22	221.44
ECE $\downarrow$	0.112	0.056	0.133	0.140	0.112	0.092	0.090	0.078
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.672	0.460	0.838	0.529	0.342	0.816	0.814	0.810

Appendix

Table 14: Medicine: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
cmb $N=200$
ACC $\uparrow$	60.50	72.50	82.00	90.50	90.50	67.00	87.00	90.50	91.50
Latency $\downarrow$	0.98	3.09	3.23	12.59	5.63	30.37	96.36	63.81	11.64
ECE $\downarrow$	0.273	0.184	0.106	0.077	0.075	0.170	0.066	0.088	0.031
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.698	0.496	0.295	0.181	0.179	0.488	0.216	0.180	0.142

Appendix

Table 15: Medicine: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
cuad $N=200$
ACC $\uparrow$	55.50	81.00	83.50	77.00	83.50	83.00	81.50	69.00
Latency $\downarrow$	1.27	6.08	14.37	17.70	58.57	95.20	56.49	6.58
ECE $\downarrow$	0.035	0.046	0.052	0.181	0.123	0.079	0.030	0.143
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.122	0.153	0.108	0.283	0.159	0.146	0.118	0.350

Appendix

Table 16: Law: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
cuad $N=200$
ACC $\uparrow$	75.00	81.00	18.50	88.00	83.00	22.00	22.00	67.50
Latency $\downarrow$	6.55	9.77	12.22	232.10	24.25	70.99	229.12	217.61
ECE $\downarrow$	0.035	0.019	0.395	0.125	0.036	0.518	0.340	0.182
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.233	0.158	0.731	0.246	0.133	0.912	0.606	0.433

Appendix

Table 17: Law: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
cuad $N=200$
ACC $\uparrow$	70.00	77.00	76.00	63.00	80.50	62.50	73.50	80.50	76.00
Latency $\downarrow$	2.56	13.36	15.48	13.76	4.72	43.12	106.46	50.38	8.51
ECE $\downarrow$	0.196	0.114	0.142	0.282	0.092	0.271	0.146	0.099	0.099
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.379	0.262	0.287	0.566	0.196	0.548	0.322	0.201	0.234

Appendix

Table 18: Law: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
finqa-yesno $N=22$
ACC $\uparrow$	95.45	72.73	86.36	68.18	81.82	86.36	90.91	54.55
Latency $\downarrow$	0.59	4.94	11.34	11.29	42.06	62.85	36.49	6.61
ECE $\downarrow$	0.051	0.194	0.098	0.113	0.082	0.079	0.056	0.120
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.102	0.263	0.193	0.336	0.227	0.162	0.122	0.439

Appendix

Table 19: Finance: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
finqa-yesno $N=22$
ACC $\uparrow$	81.82	81.82	68.18	90.91	86.36	59.09	54.55	36.36
Latency $\downarrow$	5.37	8.31	12.49	99.97	16.88	25.08	44.75	68.59
ECE $\downarrow$	0.236	0.135	0.275	0.156	0.069	0.103	0.023	0.589
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.400	0.267	0.433	0.178	0.202	0.509	0.489	1.147

Appendix

Table 20: Finance: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
finqa-yesno $N=22$
ACC $\uparrow$	59.09	86.36	81.82	81.82	95.45	77.27	81.82	95.45	90.91
Latency $\downarrow$	2.67	6.25	68.36	19.01	5.09	58.64	72.74	182.33	11.23
ECE $\downarrow$	0.417	0.126	0.190	0.177	0.045	0.210	0.193	0.045	0.091
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.773	0.261	0.366	0.358	0.091	0.439	0.362	0.091	0.180

Appendix

Table 21: Finance: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
commonsense-qa $N=200$
ACC $\uparrow$	89.00	82.50	91.00	56.50	83.50	80.50	88.50	54.50
Latency $\downarrow$	0.49	5.61	12.55	17.44	57.00	93.18	54.62	6.54
ECE $\downarrow$	0.043	0.060	0.072	0.038	0.161	0.145	0.157	0.133
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.172	0.244	0.160	0.560	0.295	0.287	0.214	0.615

Appendix

Table 22: Commonsense: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
commonsense-qa $N=200$
ACC $\uparrow$	63.50	78.00	44.50	72.00	82.00	17.00	29.50	24.50
Latency $\downarrow$	6.23	9.59	12.10	231.86	23.43	37.33	17.60	217.77
ECE $\downarrow$	0.066	0.098	0.115	0.195	0.070	0.120	0.267	0.136
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.500	0.332	0.709	0.441	0.255	0.815	0.951	0.824

Appendix

Table 23: Commonsense: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
commonsense-qa $N=200$
ACC $\uparrow$	75.00	84.50	75.00	86.50	86.50	78.50	90.50	85.50	85.00
Latency $\downarrow$	2.61	3.21	3.97	11.44	4.11	41.66	79.32	65.53	9.54
ECE $\downarrow$	0.163	0.093	0.173	0.087	0.071	0.143	0.030	0.120	0.048
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.429	0.275	0.438	0.227	0.208	0.393	0.161	0.267	0.226

Appendix

Table 24: Commonsense: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
agentharm $N=97$ , $N_{\mathrm{score}}=103$
ACC $\uparrow$	84.54	87.63	87.63	77.32	86.60	87.63	89.69	75.26
Latency $\downarrow$	0.79	5.61	12.26	18.20	56.38	95.20	56.37	6.54
ECE $\downarrow$	0.068	0.077	0.069	0.054	0.105	0.072	0.050	0.075
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.201	0.215	0.200	0.328	0.239	0.213	0.158	0.315

Appendix

Table 25: Agent and tool use: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
agentharm $N=97$ , $N_{\mathrm{score}}=103$
ACC $\uparrow$	85.57	73.20	42.27	82.47	90.72	52.58	48.45	47.42
Latency $\downarrow$	6.38	9.94	12.12	230.13	23.67	39.07	19.53	224.29
ECE $\downarrow$	0.074	0.113	0.328	0.112	0.078	0.330	0.111	0.480
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.243	0.331	0.765	0.306	0.194	0.711	0.522	0.956

Appendix

Table 26: Agent and tool use: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
agentharm $N=97$ , $N_{\mathrm{score}}=103$
ACC $\uparrow$	61.86	74.23	78.35	76.29	71.13	70.10	84.54	77.32	85.57
Latency $\downarrow$	1.09	3.62	3.74	10.19	20.26	27.16	98.40	58.86	16.14
ECE $\downarrow$	0.349	0.242	0.201	0.204	0.243	0.256	0.114	0.220	0.089
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.688	0.484	0.409	0.417	0.485	0.501	0.261	0.437	0.246

Appendix

Table 27: Agent and tool use: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
halueval-dialogue $N=200$
ACC $\uparrow$	92.00	87.00	92.00	64.00	81.50	85.00	91.50	71.00
Latency $\downarrow$	1.17	5.63	12.08	16.83	56.94	92.24	54.17	6.28
ECE $\downarrow$	0.031	0.050	0.067	0.041	0.063	0.114	0.083	0.070
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.093	0.169	0.136	0.422	0.274	0.227	0.124	0.411

Appendix

Table 28: Factuality and hallucination: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
halueval-dialogue $N=200$
ACC $\uparrow$	70.00	87.00	59.00	84.50	92.00	47.50	84.00	38.00
Latency $\downarrow$	5.74	9.79	12.83	237.61	23.58	36.33	23.26	223.89
ECE $\downarrow$	0.052	0.043	0.031	0.034	0.059	0.151	0.236	0.246
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.365	0.186	0.432	0.216	0.149	0.537	0.359	0.652

Appendix

Table 29: Factuality and hallucination: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
halueval-dialogue $N=200$
ACC $\uparrow$	77.50	83.50	89.50	95.50	95.00	90.50	93.50	90.00	94.50
Latency $\downarrow$	2.21	4.46	3.25	15.06	5.50	43.90	130.57	113.87	20.57
ECE $\downarrow$	0.144	0.079	0.098	0.030	0.028	0.137	0.090	0.133	0.047
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.346	0.231	0.185	0.083	0.086	0.241	0.164	0.238	0.097

Appendix

Table 30: Factuality and hallucination: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
helmet $N=200$
ACC $\uparrow$	74.50	95.00	100.00	32.50	92.50	93.00	100.00	20.50
Latency $\downarrow$	0.71	6.56	15.56	17.89	58.22	96.69	56.96	7.17
ECE $\downarrow$	0.001	0.139	0.007	0.283	0.787	0.584	0.027	0.154
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.000	0.098	0.000	0.944	0.771	0.493	0.002	0.924

Appendix

Table 31: Long-context and document understanding: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
helmet $N=200$
ACC $\uparrow$	33.50	82.00	4.00	99.50	73.00	0.50	19.50	1.00
Latency $\downarrow$	6.76	10.70	11.83	357.11	24.34	123.96	437.34	277.30
ECE $\downarrow$	0.197	0.323	0.107	0.797	0.034	0.014	0.182	0.019
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.775	0.407	0.971	0.729	0.030	0.990	0.988	0.991

Appendix

Table 32: Long-context and document understanding: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
helmet $N=200$
ACC $\uparrow$	40.00	74.50	98.00	73.00	74.00	20.50	46.00	64.00	99.00
Latency $\downarrow$	36.02	130.46	73.71	47.77	17.22	62.96	228.30	206.40	183.52
ECE $\downarrow$	0.391	0.000	0.030	0.000	0.007	0.087	0.077	0.000	0.000
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.840	0.000	0.055	0.000	0.013	0.232	0.155	0.015	0.000

Appendix

Table 33: Long-context and document understanding: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
bbq $N=200$
ACC $\uparrow$	98.00	98.00	96.50	68.50	86.50	73.00	94.50	66.50
Latency $\downarrow$	1.03	5.62	12.66	17.93	56.97	95.56	55.72	6.41
ECE $\downarrow$	0.012	0.061	0.062	0.154	0.160	0.065	0.154	0.138
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.036	0.052	0.067	0.472	0.235	0.343	0.157	0.454

Appendix

Table 34: Social science: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
bbq $N=200$
ACC $\uparrow$	85.50	96.50	30.00	76.50	73.50	28.50	24.50	48.00
Latency $\downarrow$	6.23	9.93	12.09	233.57	23.90	39.77	18.52	218.71
ECE $\downarrow$	0.129	0.097	0.393	0.093	0.104	0.333	0.239	0.111
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.261	0.076	0.944	0.339	0.355	0.837	0.877	0.626

Appendix

Table 35: Social science: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
bbq $N=200$
ACC $\uparrow$	67.50	90.50	86.50	96.00	94.00	97.00	96.00	96.50	91.00
Latency $\downarrow$	0.88	2.86	2.68	8.62	6.36	41.40	100.79	53.93	12.73
ECE $\downarrow$	0.160	0.049	0.080	0.034	0.057	0.023	0.020	0.034	0.029
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.445	0.162	0.222	0.079	0.117	0.018	0.071	0.069	0.138

Appendix

Table 36: Social science: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Jev	SII-Decision-4B (ours)	SII-Decision-27B (ours)	kev-0.8b	kev-4b	kev-9b	kev-27b	Intern-Decision 0.8B
prism-alignment $N=200$
ACC $\uparrow$	50.50	50.00	54.00	41.00	45.00	47.00	49.50	43.50
Latency $\downarrow$	0.84	5.88	14.00	19.46	63.08	105.86	62.30	7.82
ECE $\downarrow$	0.174	0.183	0.106	0.169	0.127	0.147	0.098	0.100
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.607	0.642	0.562	0.638	0.603	0.621	0.572	0.577

Appendix

Table 37: Personalization and preferences: model band 1/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Intern-Decision 2B	Intern-Decision 4B	laya	openjev-4b-v5	nimble-9b	nanojev	lightjev	jevforge-0.8b
prism-alignment $N=200$
ACC $\uparrow$	44.00	50.50	2.50	43.50	51.00	46.00	47.50	51.00
Latency $\downarrow$	8.04	12.27	4.31	228.04	26.07	38.84	219.78	211.15
ECE $\downarrow$	0.158	0.144	0.472	0.068	0.199	0.062	0.033	0.083
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.632	0.611	0.455	0.573	0.618	0.577	0.568	0.587

Appendix

Table 38: Personalization and preferences: model band 2/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

N: non-thinking; T: thinking. Parentheses specify reasoning effort.
Metric	Qwen3.5-4B N	Qwen3.8-27B N	DeepSeek V4-Flash N	GPT-6-Sol N	GPT-6-Luna N	Qwen3.5-4B T	Qwen3.8-27B T (xhigh)	DeepSeek V4-Flash T (high)	GLM-5.3-Flash T (high)
prism-alignment $N=200$
ACC $\uparrow$	39.50	44.50	50.50	52.00	50.50	49.00	52.50	48.00	50.00
Latency $\downarrow$	1.09	4.15	5.25	9.41	26.85	37.44	126.47	127.13	16.12
ECE $\downarrow$	0.301	0.311	0.284	0.300	0.331	0.282	0.129	0.348	0.128
$\mathrm{Brier}_{\mathrm{raw}}\,\downarrow$	0.757	0.728	0.713	0.729	0.777	0.692	0.574	0.818	0.599

Appendix

Table 39: Personalization and preferences: model band 3/3. ACC (%)

\uparrow

; benchmark-run latency (s), ECE, and

\mathrm{Brier}_{\mathrm{raw}}

\downarrow

. Rankings span all bands.

Figure 21: Examples of the four tasks, with gold choices in green and true probabilities shown. Numerical options are abbreviated.

Panel of Figure 22 — Figure 22: Matched $\tau$ -bench workflow for the baseline and JEV arms.

Voter model	States $\uparrow$	Battleground $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	Bias
ElectionSim (GPT-4o-mini)	46/51	11/15	3.06	3.80	+2.31
SII-Decision-27B (decision model)	47/51	12/15	6.52	9.42	+3.73

Appendix

Table 40: Summary of the 2024 U.S. presidential election simulated in ElectionSim with voters sampled at a ratio of 1/1,000. Errors are in percentage points. Bold marks the better value in each column, with the bias closer to zero counted as better.

				SII-Decision-27B		ElectionSim, GPT-4o-mini
State	BG	Voters	Actual	Share	Error	Share	Error
Alabama		5,031	34.5	32.6	$-$ 1.9	34.4	$-$ 0.2
Alaska		737	43.2	12.6	$-$ 30.5	38.0	$-$ 5.2
Arizona	$\bullet$	7,159	47.2	41.4	$-$ 5.8	46.6	$-$ 0.6
Arkansas		3,014	34.3	35.5	+1.2	33.6	$-$ 0.7
California		39,577	60.6	66.2	+5.6	64.1	+3.4

Appendix

Table 41: Per-state results of the 2024 ElectionSim simulation with a 1/1,000 voter sample. BG marks the 15 battleground states, and † marks a state called incorrectly.

Setting	Full-rank (main)	LoRA
Updated weights	attention and MLP projections	LoRA adapters on the same modules
LoRA rank and $\alpha$	—	64 and 128
Learning rate	$2\times 10^{-6}$	$5\times 10^{-5}$
Frozen weights	embeddings and output layer
Optimizer and schedule	AdamW, 3% warmup, cosine decay
Global batch size and epochs	64, one epoch

Appendix

Table 42: Training settings for Reasoning-to-Readout Self-Distillation, shared by both backbones.

Figure 23: Development score against the number of training questions under Reasoning-to-Readout Self-Distillation. Filled markers with solid lines show full-rank training, and open markers with dashed lines show LoRA. Points for Qwen3.5-4B are means over two seeds. For Qwen3.8-27B, sizes up to 20,000 questions use one seed and larger sizes average two or three seeds; the shaded region marks sets that add new sources, and the largest set also contains questions written by the model itself.

Capability group	5,000	10,000	20,000	29,326	39,279	48,358
Language understanding	1,199	2,398	4,795	7,031	7,031	7,031
General knowledge	938	1,875	3,751	5,500	5,500	5,500
Commonsense	693	1,387	2,774	4,068	4,068	4,068
Logic	520	1,040	2,081	3,051	3,051	3,051
Medicine, English	520	1,040	2,080	3,050	3,050	3,050
Medicine, Chinese	498	996	1,992	2,921	2,921	2,921

Appendix

Table 43: Number of training questions per capability group. The sets of 5,000 to 29,326 questions are nested and keep the same mixture. The sets of 39,279 and 48,358 questions keep all 29,326 questions, add new sources, and are drawn independently of each other; SII-Decision-27B is trained on the set of 39,279 questions.

Explore similar work

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
Oct 8, 2026Xing Li, Qingcheng Chang, Jinzhong Ning +5LLM Evaluation Multiple-Choice Question Answering
Dalian Maritime University · Dalian University of Technology
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Oct 1, 2026Yinheng Li, Justin WagleLLM Inference LLM Fine-Tuning
Microsoft
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
Oct 12, 2026Jonathan Geuter, Andrew Cheng, David Alvarez-MelisLLM Decision-Making LLM Fine-Tuning
Harvard University · Kempner Institute

Oct 8, 2026J/K move · Enter open · S save

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu+4

Dalian Maritime University · Dalian University of Technology

LLM Evaluation Multiple-Choice Question Answering

Open paper AskPDF arXiv

Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.

Abstract

Explore similar work

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Releasing the Inner Jev: A Simple Recipe for Open Decision Models