Evaluating and Benchmarking the System One Model Jev

Tobias Deußer; Lorenz Sparrenberg; Rafet Sifa

cs.CLSep 29, 2026

Evaluating and Benchmarking the System One Model Jev

Authors: Tobias Deußer, Lorenz Sparrenberg, Rafet Sifa

Organizations: University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany

Abstract

Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities the vendor describes as calibrated. Such models target small decisions in information access pipelines, such as routing queries, checking grounding, moderating content, or rating against a rubric. We evaluate Jev (jev-1.13.0) zero-shot on 37 datasets spanning classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal clause analysis and rubric scoring, with one frozen template per dataset and full evaluation splits: 346,009 requests for under USD 10. For reference, we score Qwen3.8-27B and Gemma-4-E4B on identical requests via their exact next-token probabilities over the options. Jev reaches 95-99% accuracy on IMDB, SST-2, HellaSwag and ARC and 86.7% on Belebele across 122 languages. It beats Qwen on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals, and Gemma on all 37. All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments. Jev's choice probabilities are well calibrated and support selective prediction. Binary probabilities rank well but are poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder. Rotating the options leaves Jev's accuracy unchanged and withholding the question drops it to near chance, ruling out shallow memorization but not memorized question-answer pairs. We release the code, harness and all raw responses.

Figures & tables

Dataset	Hugging Face source	Split	$n$	Lang.	Primitive
Text classification
AG News [ 29 ]	fancyzhx/ag_news	test	7,600		Choice (4)
IMDB [ 30 ]	stanfordnlp/imdb	test	25,000		Choice (2)
Rotten Tomatoes [ 31 ]	cornell-movie-review-data/rotten_tomatoes	test	1,066		Choice (2)
SST-2 [ 32 ]	stanfordnlp/sst2	val.	872		Choice (2)
Emotion [ 33 ]	dair-ai/emotion	test	2,000		Choice (6)

Table 1: Benchmark suite: 37 datasets in seven categories, the split we evaluate, the number of evaluated examples

n

, the number of languages for multilingual datasets, and the Jev primitive used (number of options or levels in parentheses;

\times k

k

questions per request).

Dataset	$n$	Metric	Score $\uparrow$	95% CI	ECE $\downarrow$
Text classification
AG News	7,600	Acc.	0.885	[0.878, 0.891]	0.077
IMDB	25,000	Acc.	0.965	[0.963, 0.967]	0.019
Rotten Tomatoes	1,066	Acc.	0.933	[0.917, 0.947]	0.038
SST-2	872	Acc.	0.964	[0.951, 0.975]	0.015
Emotion	2,000	Acc.	0.585	[0.563, 0.605]	0.279

Table 2: Main results on the evaluation splits. Score: primary metric (see text); 95% bootstrap confidence interval; ECE of the first Choice or Noul question (– for Score-only tasks).

		Score $\uparrow$			ECE $\downarrow$
Dataset	Metric	Jev	Qwen3.8-27B	Gemma-4-E4B	Jev	Qwen3.8-27B	Gemma-4-E4B
Text classification
AG News	Acc.	0.885	0.867	0.862	0.077	0.081	0.122
IMDB	Acc.	0.965	0.965	0.956	0.019	0.018	0.039
Rotten Tomatoes	Acc.	0.933	0.932	0.901	0.038	0.039	0.092
SST-2	Acc.	0.964	0.954	0.952	0.015	0.021	0.042

Table 3: Jev and the two open-weight reference models on identical requests: primary metric (best per dataset in bold) and ECE of the first Choice or Noul question (– for Score-only tasks).

		Jev		Qwen3.8-27B		Gemma-4-E4B
Dataset	Metric	0.5	tuned	0.5	tuned	0.5	tuned
GoEmotions	Macro-F ${}_{\text{1}}$	0.243	0.353	0.255	0.323	0.211	0.267
	Micro-F ${}_{\text{1}}$	0.239	0.387	0.236	0.317	0.207	0.268
UNFAIR-ToS	Micro-F ${}_{\text{1}}$	0.499	0.748	0.582	0.757	0.302	0.510
	Macro-F ${}_{\text{1}}$	0.577	0.739	0.682	0.768	0.492	0.614
AGB-DE	F ${}_{\text{1}}$	0.204	0.204	0.000	0.179	0.066	0.142

Table 4: F

{}_{\text{1}}

with the fixed 0.5 threshold vs. per-question thresholds tuned on 1,000 training/validation examples and applied unchanged to the evaluation split, for each model (thresholds tuned on that model’s own answers).

		Spearman $\rho\uparrow$
Dataset	Dimension	Jev	Qwen3.8-27B	Gemma-4-E4B	MAE $\downarrow$ (Jev)
STS-B	–	0.890	0.896	0.814	0.56
SST-5	–	0.851	0.857	0.779	0.50
ToxiGen	Toxicity	0.841	0.836	0.780	0.57
SummEval	Coherence	0.639	0.651	0.515	0.61
	Consistency	0.555	0.546	0.487	0.46

Table 5: Score questions: Spearman

\rho

between each model’s probability-weighted score and the gold ratings (summary-level for SummEval; best in bold), and Jev’s mean absolute error in rubric levels.

Benchmark	Subjects / condition	$n$	Jev	Qwen3.8-27B	Gemma-4-E4B
MMLU	calc.-heavy, original	2,207	0.943	0.747	0.546
	other, original	11,835	0.913	0.830	0.698
	calc.-heavy, options rotated	2,207	0.943	0.744	0.551
	calc.-heavy, question withheld	2,207	0.315	0.351	0.304
C-Eval	calc.-heavy, original	3,389	0.814	0.744	0.462
	other, original	8,953	0.849	0.812	0.566

Table 6: Calculation-heavy vs. other subjects of MMLU and C-Eval for each model: accuracy with the original requests, and on the calculation-heavy subjects with the answer options rotated so that every option changes its letter, and with the question withheld (only the subject and the options are shown). Chance is 0.25.

Category	Requests	Input tok. (M)	Tok./req.	Cost (USD)	Latency (s)
Text classification	57,223	32.0	559	1.34	0.36
Intent and topic routing	50,208	28.5	567	1.19	0.35
NLI, paraphrase and grounding	51,094	39.9	780	1.67	0.36
Reading comprehension and knowledge	140,425	91.8	654	3.86	0.37
Commonsense and reasoning	30,837	15.7	509	0.66	0.37
Safety, moderation and legal	10,048	5.6	557	0.23	0.36

Table 7: Requests, input tokens, cost and mean client-side latency per request (at 32 concurrent requests) for the evaluation run, by category. Requests counts unique requests sent during the evaluation run. Identical items (1,911 duplicates across 17 datasets) are sent once and scored for every occurrence; 4 items had already been answered in the development pilot and 1 over-long BIG-bench item was rejected.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	Acc. $\uparrow$	ECE $\downarrow$	Acc.@80% $\uparrow$	Acc.@50% $\uparrow$	AURC $\downarrow$
Text classification
AG News	0.885	0.077	0.942	0.953	0.056
IMDB	0.965	0.019	0.993	0.993	0.008
Rotten Tomatoes	0.933	0.038	0.975	0.981	0.024
SST-2	0.964	0.015	0.989	0.995	0.007
Emotion	0.585	0.279	0.642	0.733	0.268

Appendix

Table 8: Choice datasets: accuracy, ECE of the top-option probability, accuracy on the 80% and 50% most confident examples (ranked by Jev’s confidence), and area under the risk–coverage curve.

Explore similar work

Sep 29, 2026cs.CL

Chinese-Jev: Bringing System One Model to Chinese-Language Tasks

System One models such as Jev offer an efficient alternative to generative language models for tasks that require decisions rather than open-ended responses. However, existing Jev models exhibit limited Chinese-language decision accuracy, restricting their utility in both general and specialized settings. In this paper, we introduce Chinese-Jev, a System One model that addresses this gap through a unified data processing and training pipeline. Our data processing protocol converts heterogeneous Chinese-language annotations into probability targets over candidate options, enabling a shared training formulation across domains and question formats. To enable efficient inference, Chinese-Jev adopts a lightweight encoder-only backbone for text encoding and learns to score candidate answers through decision-oriented training. To address the misalignment between the pre-training distribution and downstream Chinese-language scenarios, we first train the model on a general-purpose corpus of 10 million examples, then fine-tune it separately for the medical, legal, and financial domains. To evaluate decision accuracy and calibration in both general and domain-specific Chinese-language settings, we introduce Chinese-Jev Bench (CJ-Bench). After first-stage pre-training, Chinese-Jev exceeds the accuracy of the closed-source Jev model by 1.24% on general-domain tasks while achieving a 20.3x speedup. Subsequent domain-specific fine-tuning yields a 4.0% accuracy improvement over Jev in medicine and achieves 92% of Jev's average accuracy across specialized domains, with a 17x speedup and an average latency of only 15 ms per example. We further demonstrate on-device deployment of an INT8-quantized model on mobile devices, achieving an inference latency of approximately 1.0 second per decision. The project is available at https://gulucaptain.github.io/Chinese-Jev/.

Zexiao Wang, Zihao Zhang, Xudong Wang +5

Fudan University · National University of Singapore · University of Chinese Academy of Sciences

Topic pending

Ask PDF

Sep 28, 2026cs.AI

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition

A

for which the exact probability

P(A)

is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers,

P(A)

and

P(\neg A)

are presented as if

P(A)+P(\neg A)=1

, while a term

P(U)\neq0

is missing in the sum. Recovering

P(U)

leads to an improvement of median soft accuracy in \texttt{Choice} answers from

0.771

0.978

, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.

Riccardo Porcedda

Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy

Probability Confidence Calibration Confidence Estimation

Ask PDF

Sep 24, 2026cs.CL

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

We ask whether Jev, a typed classifier that returns probabilities over permitted answers without generating text, can replace an LLM rubric judge. We compare it with three flash-tier LLM judges on nine panels drawn from seven benchmarks, giving every judge identical criterion texts. Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons, ahead mostly on binary criteria and behind only on graded ones, and most of the other comparisons are inconclusive. Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev and took 30 to 220 times as long. On graded criteria all four judges agree more with one another than with the labels and mostly assign lower levels than the raters. One of several observational accounts is that raters followed scale conventions our criterion texts omit. Jev's confidence ranks its own errors on most panels, which should make a cheap classifier the ideal first stage of a cascade that defers its uncertain verdicts to an LLM judge. Correlated errors undo that advantage. The LLM judges repeat nearly all of Jev's most confident errors, so a cascade replayed on the recorded verdicts lowers cost but gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 even with oracle thresholds.

Delip Rao, Chris Callison-Burch

University of Pennsylvania

Large Language Model Judges

Ask PDF