Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Jonathan Geuter; Andrew Cheng; David Alvarez-Melis

cs.AIOct 12, 2026

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Authors: Jonathan Geuter, Andrew Cheng, David Alvarez-Melis

Organizations: Harvard University · Kempner Institute

Abstract

LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.

Figures & tables

Panel of Figure 1 — Figure 1: A Jev-style decision in one forward pass. The request is rendered as a multiple-choice prompt with lettered options. The LLM reads it once; the decision head maps the last hidden state $h$ to one logit per option, starting from the language-model head’s rows for the letters, so the untrained model is the zero-shot readout. Training adds LoRA adapters and a learned correction $\Delta$ to the head. A softmax over the real options gives the probabilities.

Family	Code	Authored	Real	Total
LLM routing	22	391	281	694
Multilingual		393	29	422
Fairness pairs		236		236
Business use cases	20	193		213
Counting traps	162			162
Negation pairs	160			160

Table 1: OpenDecisions families and label sources. Programmatic items are written and labeled by code; authored items are written by an LLM; real requests are user requests from public datasets. Calibration: questions with exact answer probabilities.

Part	Rows	Labels
decider public mixture (79 tasks)	300,042	dataset (hard)
Civil Comments	30,000	annotators (soft)
Measuring Hate Speech	30,000	annotators (soft)
STS-B	4,760	similarity (soft)
ChaosNLI	2,886	100 votes (soft)
synthetic documents (ours)	30,641	teachers (soft)

Table 2: Training pool (417,825 questions). Soft = the target is a distribution (annotator votes or teacher probabilities). Teacher labels come from Qwen3.5-122B-A10B and Qwen3.6-27B.

Figure 2: Zero-shot reaches Jev; a short training run goes beyond. OpenDecisions Capability of the four open models without training (orange) and after training (blue, mean of three seeds; dots: seeds; Section 4 ). Dashed: Jev. Dotted: the strongest public system (Eikos-27B). The two models of about 30B parameters reach Jev’s level zero-shot, and training moves Gemma 4 31B clearly beyond it.

Ours
			OpenDecisions			JevBench-Open
System	Backbone	Trained	$I$	$C$	Capability	Capability	Accuracy (%)
Zero-shot	Gemma 4 31B	–	71.9	87.4	79.6	73.4	91.8
Jeff-1.0-Large	Gemma 4 31B	✓	76.4	92.3	84.4	80.1	92.1
Zero-shot	Qwen3.6-27B	–	67.5	90.5	79.0	77.1	88.3
Trained	Qwen3.6-27B	✓	72.6	89.9	81.2	78.7	88.5

Table 3: Main results. OpenDecisions Intelligence, Calibration and Capability, and Capability and top-1 accuracy on JevBench-Open (the 231 items JevBench released publicly). Ours: letter readout with fitted temperatures; trained = mean of three seeds of the recipe (Section 4 ; Qwen3.5-4B with three changes); Jeff-1.0-Large is the trained Gemma 4 31B (we release seed 1). Public systems: as served or with fitted temperatures, whichever scores higher on OpenDecisions. Trained: ✓ fine-tuned, – frozen readout, ? unknown. Bold: best per column.

System	TREC	CB	WANLI	Auth.	Pert.	TypeSafe	decider	Macro	ECE
Items	500	56	256	144	108	348	7,800	9,212
Jev 1.13.0 (API)	92.4	82.1	76.6	97.2	100.0	87.0	80.6	88.0	.066
Gemma 4 31B, zero-shot	92.8	85.7	78.9	95.1	97.2	87.5	81.9	88.4	.077
Jeff-1.0-Large (Gemma 4 31B, trained)	92.8	87.5	78.5	95.8	100.0	85.8	82.8	89.0	.047
Qwen3.6-27B, zero-shot	93.0	83.9	77.7	97.9	100.0	84.3	81.4	88.3	.052
Qwen3.6-27B, trained	93.4	87.5	77.0	97.2	100.0	88.3	82.2	89.4	.064

Table 4: Performance on seven held-out public decision datasets. Accuracy (%) for all models evaluated on the same questions. TREC ( Li & Roth, 2002 ) : question classification; CB ( de Marneffe et al., 2019 ) : natural language inference; WANLI ( Liu et al., 2022 ) : natural language inference; Auth. and Pert.: SemIf’s authored and perturbation sets ( SemIf contributors, 2026 ) ; TypeSafe: public decision examples, measured by agreement with the most likely reference answer; decider ( Marosi, 2026 ) : 29 held-out classification tasks, averaged across tasks. Macro: average accuracy across the seven datasets. ECE: mean expected calibration error across the six datasets with hard labels (excluding TypeSafe); lower is better. Our models use fitted temperatures; trained results use seed 1. Bold indicates the best result in each column.

Variant	Capability	$\Delta$
Recipe (mean of 3 seeds)	84.4
No training (zero-shot)	79.6	$-$ 4.8 $\pm$ 2.0
What is trained
Head only, frozen backbone	78.9	$-$ 5.5 $\pm$ 2.5
LoRA, head frozen at its initialization	84.9	$+$ 0.6 $\pm$ 2.1
Backbone

Table 5: What matters (Gemma 4 31B, OpenDecisions Capability; 400 steps and seed 1 unless noted; 800, 1,500 and 3,000 steps: mean of seeds 2 and 3).

\Delta

: difference to the recipe’s three-seed mean (84.4),

\pm

1.96 standard errors (question sampling and seed variation). Pretrained head: trained first on frozen features.

Figure 3: Training length for Gemma 4 31B. Capability on OpenDecisions (Table 9 ): mean over seeds (line) and single seeds (dots). At 400 steps: the three seeds of Table 3 ; at 800 to 3,000 steps: seeds 2 and 3, each after a final learning-rate decay branched from one longer run per seed. Each step processes 65,536 padded tokens. Dashed: calibrated zero-shot baseline. Smaller-model results appear in Figure 4 (Appendix B ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Letter	Option	Head logit	Raw	Served
team (choice)	A	billing	26.41	0.933	0.842
	B	technical	19.65	0.001	0.010
	C	refund	23.77	0.066	0.148
money_owed (yes/no)	A	true	24.13	0.998	0.992
	B	false	17.74	0.002	0.008
urgency (score)	A	0: not urgent	23.03	0.008	0.020

Appendix

Table 6: Output of Jeff-1.0-Large for the example request. Head logit: the decision head’s logit for the option (Gemma’s logits are soft-capped at 30). Raw: softmax over the question’s options. Served: after the temperature.

Setting	Value
LoRA	rank 32, $\alpha=64$ , dropout 0; q/k/v/o, gate/up/down projections (Qwen: also in_proj_qkv, in_proj_z, out_proj)
Head	decision head (Eq. 2 ), 16 slots, initialized from the language-model head, trained jointly
Optimizer	AdamW, LR $2\cdot 10^{-5}$ (LoRA), $10^{-4}$ (head), weight decay 0
Schedule	250 warmup steps, constant to step 300, cosine decay to 0 at step 400 (WSD)
Batch	65,536 padded tokens per step: 4 GPUs $\times$ 8,192 tokens $\times$ 2 accumulation steps
Inputs	at most 8,192 tokens (longer rows skipped); options shuffled; bf16; gradient checkpointing

Appendix

Table 7: Training settings of the recipe. Qwen3.5-4B differs in three settings (last row).

Figure 4: Training length for the smaller models. OpenDecisions Capability against training steps (65,536 tokens each), for checkpoints with a learning-rate decay at the end; each point is the mean of three seeds (Table 9 ; Gemma 4 12B at 3,000 steps: seeds 2 and 3). (a) Qwen3.6-27B, recipe. (b) Gemma 4 12B, recipe. (c) Qwen3.5-4B: with the three changes for this model (Section 4 ) and with the 31B recipe. Dashed: zero-shot.

Model	Steps	Zero-shot	Seed 1	Seed 2	Seed 3	Mean
Gemma 4 31B	400	79.6	85.3	84.0	83.8	84.4
Qwen3.6-27B	400	79.0	81.8	81.7	80.2	81.2
Gemma 4 12B	400	75.4	75.6	74.6	74.8	75.0
Qwen3.5-4B (4B changes)	400	63.6	68.0	66.4	65.5	66.6

Appendix

Table 8: Per-seed OpenDecisions Capability of the trained models in Table 3 .

Model and variant	400	800	1,500	3,000
Gemma 4 31B	84.4	84.7	83.6	83.3
Qwen3.6-27B	81.2	80.9	80.3	–
Gemma 4 12B	75.0	76.3	75.5	76.2
Qwen3.5-4B, 4B changes	66.6	65.5	65.4	63.9
Qwen3.5-4B, 31B recipe	63.0	64.0	63.5	62.2

Appendix

Table 9: Training length (OpenDecisions Capability, decayed checkpoints, mean of three seeds; Gemma 4 31B at 800 to 3,000 steps and Gemma 4 12B at 3,000 steps: seeds 2 and 3; the numbers of Figures 3 and 4 ).

Type	Easy	Standard	Judge	Hard	Total
Choice	261 (264)	136 (136)	255 (256)	526 (544)	1,178 (1,200)
Yes/no	110 (112)	144 (144)	158 (160)	174 (184)	586 (600)
Score	72 (72)	127 (128)	168 (168)	221 (232)	588 (600)

Appendix

Table 10: OpenDecisions composition by question type and tier (target in brackets: the composition of the public JevBench v1.6 split times 8).

Set	Items	Spearman	Pairs in order	LOO
10 systems used while building OpenDecisions
JevBench-Open	231	0.85 [0.56, 0.94]	0.84	4.9
OpenDecisions pilot	283	0.90 [0.67, 0.92]	0.89	4.3
OpenDecisions	2,352	0.98 [0.92, 0.99]	0.96	2.6
Training-distribution validation set	3,884	0.45 [0.31, 0.53]	0.67	10.9
Adding 4 systems not used to build OpenDecisions

Appendix

Table 11: Does a set predict the official leaderboard? Agreement between Capability on each set and the official v1.6.1 Capability of public systems with official Capability

\geq 55

, each run as its authors serve it and scored with the leaderboard’s scorer. LOO = mean absolute error when predicting each system’s official score from the others by a linear fit. Gray brackets: 95% intervals from resampling item groups (correlations are bounded, so we show the interval itself rather than

\pm

Family	Gemma 4 31B		Qwen3.6-27B	Gemma 4 12B	Eikos-27B	Quyet-L	deck31b	Jev	Torchcast
	trained	zero-shot	trained	zero-shot
LLM routing	78.3	76.1	75.0	72.9	76.6	77.7	76.5	79.3	74.3
Multilingual	80.8	76.8	75.0	66.4	78.4	76.6	78.0	67.5	69.2
Fairness pairs	78.8	75.4	72.4	61.8	80.5	86.6	76.9	72.0	69.7
Use cases	73.4	68.6	65.8	50.7	74.4	71.6	68.1	54.4	59.4
Counting traps	56.7	53.4	48.0	41.7	44.8	57.4	56.8	60.4	41.3

Appendix

Table 12: Competence by item family on OpenDecisions (chance-corrected accuracy; same items for all systems; ours with temperatures, trained = mean of three seeds; public systems as in Table 3 ).

Query set	Questions	Median similarity
OpenDecisions	2,389	0.156
Held-out training ( $H$ )	2,389	0.763
$H$ without shared contexts	1,110	0.509

Appendix

Table 13: Median nearest-neighbor TF–IDF cosine similarity to the same training reference set

R

. The last row excludes training questions with an identical supporting context in

R

System	Backbone	Draw	$I$	$C$	Capability
Quyet-1.0-Large	Gemma 4 31B	v1.6.0	73.4	90.0	81.7
Jeff-1.0-Large (ours)	Gemma 4 31B	v1.6.2	71.9	91.1	81.5
decisio v0.8.0	Gemma 4 31B	v1.6.1	70.4	88.7	79.6
deck31b	Gemma 4 31B	v1.6.0	73.0	82.2	77.6
Jev 1.13.0 (API)	closed	v1.6.1	63.6	90.6	77.1

Appendix

Table 14: Official JevBench leaderboard ( JevBench maintainers, 2026 ) : the systems with the highest Capability, as published in October 2026 (revision v1.6.7). Each system is measured on its own draw of 1,500 items (300 public, 1,200 sealed); draw: the revision in which the system was measured. Jev is listed as a reference and not ranked.

System	BoolQ	MMLU	MMLU-Pro	ARC-C	WinoGr.	SciQ	Macro	ECE
Jev 1.13.0 (API)	92.6	91.8	84.0	97.8	92.2	98.7	92.9	.032
Quyet-1.0-Large	92.2	85.3	65.9	97.5	90.0	99.0	88.3	.035
deck31b	92.4	85.8	68.8	97.2	86.0	98.7	88.1	.077
decider-4b-v2	91.7	75.0	52.3	92.7	83.0	98.2	82.1	.038
Torchcast-decision-12B	90.0	79.0	52.8	94.3	75.8	98.0	81.6	.049
Winnow-12B	89.0	77.3	54.0	94.7	74.5	98.0	81.3	.084

Appendix

Table 15: Knowledge-heavy multiple-choice benchmarks : accuracy (%) on our seeded samples without overlapping items. Macro: mean of six tasks. ECE: 15 bins, raw probabilities, mean over tasks. For reference, Li & Wagle (2026) report macro accuracies on their own samples of 78.6 (4B, training-free), 78.9 (4B, LoRA), 82.2 (Winnow-12B), 75.4 (SemIf) and 91.5 (GPT-5.6 Sol, text output).

Figure 5: Zero-shot accuracy against model size : macro accuracy (%) over the seven held-out decision sets of Table 4 , letter readout, no training. Mixture-of-experts models are placed at their 3B active parameters. Dashed: Jev.

Qwen3.5-4B, option encoding (LoRA and head, 600 steps)
Variant	Seeds	Acc.	$\Delta$	ECE	ECE +T
Letters, given order	2	85.4 $\pm$ 2.6	–	.100	.096
Letters, sorted at test time	1	86.3 $\pm$ 3.3	+1.0 $\pm$ 3.5	.103	.096
Letters, sorted in training and test	2	84.9 $\pm$ 2.6	$-$ 0.4 $\pm$ 2.9	.092	.089
Option-parallel	1	83.4 $\pm$ 3.6	$-$ 1.9 $\pm$ 3.8	.092	.092
+ set layer	1	83.1 $\pm$ 3.6	$-$ 2.3 $\pm$ 3.8	.102	.102

Appendix

Table 16: Order-invariant encoding and calibration losses , rescored on the seven held-out decision sets of Table 4 . Acc.: macro accuracy (%).

\Delta

: paired difference to the first row of the block. ECE: mean over the six hard-label sets, raw and with one temperature fitted on the run’s own validation predictions (+T). Rows with several seeds show the seed mean.

	Zero-shot		Trained
Model	Raw	Temperatures	Raw	Temperatures
Gemma 4 31B	72.9	79.6	81.9	84.4
Qwen3.6-27B	78.2	79.0	81.2	81.2
Gemma 4 12B	65.4	75.4	74.0	75.0
Qwen3.5-4B	59.0	63.6	66.6	66.6

Appendix

Table 17: Raw readout vs. fitted temperatures (OpenDecisions Capability; trained = mean of three seeds; temperatures never below 1).

Model	Version	Choice	Yes/no	Score
Gemma 4 31B	zero-shot	5.96	7.24	7.24
	trained	1.47 $\pm$ 0.09	1.31 $\pm$ 0.07	1.25 $\pm$ 0.07
Qwen3.6-27B	zero-shot	1.38	1.52	1.85
	trained	1.01 $\pm$ 0.03	0.83 $\pm$ 0.03	0.77 $\pm$ 0.04
Gemma 4 12B	zero-shot	3.84	4.67	5.68
	trained	1.19 $\pm$ 0.07	1.06 $\pm$ 0.07	1.10 $\pm$ 0.04

Appendix

Table 18: Fitted temperatures per question type, fitted by NLL on our validation set (3,884 questions). Trained: mean of the three seeds of the models in Table 3 ,

\pm

1.96 standard errors. Italic: below 1; temperatures below 1 are set to 1 when serving (we never sharpen).

System	As served	With temperatures	Fitted temperatures (choice / yes-no / score)
Eikos-27B	83.1	83.1	0.61 / 0.69 / 0.79
Quyet-1.0-Large	81.8	81.7	0.94 / 0.82 / 1.06
deck31b	78.1	81.3	1.68 / 2.59 / 2.37
AutoJev-27B	80.5	80.5	0.82 / 0.77 / 0.94
Jev 1.13.0 (API)	80.1	78.8	1.73 / 0.69 / 2.37
Torchcast-decision-12B	75.6	78.4	1.12 / 4.22 / 1.41

Appendix

Table 19: Public systems with and without fitted temperatures (OpenDecisions Capability). Table 3 uses the higher of the two. Fitted temperatures before the rule; gray: below 1, so set to 1.

Competence
	Items	Zero-shot	Trained
Choice	1,178	84.5 / 84.5	86.4 / 86.4
Yes/no	586	77.0 / 77.0	78.4 / 78.4
Score	588	64.5 / 54.1	66.2 / 64.5
Calibration
Choice	1,178	58.6 / 84.0	81.2 / 92.6

Appendix

Table 20: Gemma 4 31B with and without fitted temperatures, by question type and subset (OpenDecisions; raw / with temperatures; trained = mean of three seeds). Subsets: Capability of their items under the same scorer.

Backbone	Mean outside	99th percentile	Top token not an option	Mean outside	Mean outside
	OpenDecisions			JevBench-Open	Our validation set
Gemma 4 31B	0.01%	0.2%	0	0.19%	0.24%
Qwen3.6-27B	0.37%	4.9%	0	0.13%	0.67%
Gemma 4 12B	0.52%	6.1%	9	$<$ 0.01%	0.28%
Qwen3.5-4B	0.09%	1.5%	0	0.05%	0.05%

Appendix

Table 21: Next-token probability outside the option letters , zero-shot (full vocabulary softmax at the answer position, including Gemma’s soft-capping).

				OpenDecisions			JevBench-Open
Backbone	Version	Prompt	Option mass	$I$	$C$	Capability	Accuracy (%)
Gemma 4 31B	instruction-tuned	chat	99.99%	71.9	87.4	79.6	91.8
	instruction-tuned	plain	69.6%	67.8	90.0	78.9	89.6
	pretrained	chat	0.13%	7.9	66.4	37.2	41.1
	pretrained	plain	66.1%	42.1	85.9	64.0	76.6
	pretrained + recipe	chat	–	58.4	82.1	70.3	81.0

Appendix

Table 22: Backbones with and without instruction tuning , zero-shot letter readout with fitted temperatures (never below 1). Chat: our prompt (Appendix A ). Plain: the same text without chat markers, followed by “Answer:”. Option mass: mean next-token probability on the option letters, on OpenDecisions. + recipe: trained with our recipe (400 steps, seed 1; the decision head reads out only the option letters).

Explore similar work

Oct 1, 2026

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without training. Fine-tuning provides targeted gains for smaller models and complex routing, while our KL anchors prevent behavioral degradation in general text generation.

Yinheng Li, Justin Wagle

Microsoft

LLM Inference LLM Fine-Tuning Language Model Calibration LLM Decision-Making

Ask PDF

Oct 8, 2026

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.

Xing Li, Qingcheng Chang, Jinzhong Ning +5

Dalian Maritime University · Dalian University of Technology

LLM Evaluation Multiple-Choice Question Answering

Ask PDF

Oct 2, 2026

General Decision Models: Benchmarking and Insights Beyond Jev

General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on

τ

-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.

Feiyu Duan, Jiayu Lin, Jia Wang +6

Shanghai Innovation Institute · Fudan University · Tongji University +2

LLM Decision-Making Uncertainty Calibration Knowledge Distillation

Ask PDF