Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Riccardo Porcedda

cs.AISep 28, 2026

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Authors: Riccardo Porcedda

Organizations: Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy

Abstract

The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition $A$ for which the exact probability $P(A)$ is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers, $P(A)$ and $P(\neg A)$ are presented as if $P(A)+P(\neg A)=1$ , while a term $P(U)\neq0$ is missing in the sum. Recovering $P(U)$ leads to an improvement of median soft accuracy in \texttt{Choice} answers from $0.771$ to $0.978$ , suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.

Figures & tables

Figure 1: Probability transfer across all Sys1Cal-v1 families. Noul and Score expectation remain closer to the dashed identity line, while Choice is systematically shifted toward high True probabilities.

Output	Mean TV	Mean OVL
Jev- Choice	0.236	0.764
Jev- Noul	0.0817	0.918
Jev- Score	0.1139	0.886
SemIf- Choice	0.371	0.629

Table 1: Mean TV and OVL (soft accuracy) by model-primitive. Jev- Noul and Jev- Score are well calibrated, while Choice answers show poorer performances on both Jev and SemIf

Figure 2: Choice probability transfer for Jev and SemIf. Mean TV is reported in each panel.

Figure 3: Score expectations compared with ground truth, Noul , and Choice probabilities. Score expectations tracks Noul more tightly than it tracks Choice , which shows a peculiar shape that suggests the existence of a map that could fix the calibration.

Figure 4: Score -to- Choice mapping induced by the fitted latent uncertainty model. Introducing a latent uncertainty probability of the form $\mu_{S}^{\alpha}(1-\mu_{S})^{\beta}$ captures much of the systematic non-linearity between Score expectations and binary Choice probabilities.

Quantity	Estimate	95% CI	$p$ -value
Global $\hat{\lambda}$	0.978	[0.789, 1.000]	–
Latent $\lambda_{j}$	0.821	[0.760, 0.882]	$2.94{\times}10^{-44}$

Table 2: Tests for the scaled Score–Uncertainty model. The null hypothesis is

H_{0}:\lambda=0

. The global

\hat{\lambda}

row reports the fitted scale and a bootstrap confidence interval over latent problems. The

\lambda_{j}

tests whether the fitted uncertainty is positive across the latent problems.

Setting	Mean OVL	Med. OVL
Raw Choice	0.764	0.771
Calibrated Choice	0.880	0.903
T/U/F	0.931	0.978

Table 3: Choice OVL under three evaluation modes. The T/U/F score evaluates interval compatibility and should be interpreted together with the mean uncertainty width

\mathbb{E}[U]

Setting	Criterion	Action
Raw Choice	Bayes risk	Act True
Calibrated Choice	Bayes risk	Act False
T/U/F	Robust risk	Defer

Table 4: Decision induced by the three Choice-derived settings in a costly automation example. The same raw Choice output leads to three different actions: raw Choice commits to True , calibrated Choice commits to False , and T/U/F defers because the unresolved mass makes either committed action too risky.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Representation	Family	State	Proposition / gold
Direct	Explicit probability	sufficient_statistics: $p_{A}=0,\;p_{\neg A}=1$	“Event A is true.” $P(\mathrm{True})=0$
Ratio	Explicit probability	probability_ratio_A_to_not_A: 0:1	“Event A is true.” $P(\mathrm{True})=0$
Prose	Explicit probability	Natural-language rendering: “The state fully determines the target distribution. The probability of $A$ is 0, and the remaining probability belongs to the other outcome.”	“Event A is true.” $P(\mathrm{True})=0$
Distractor	Explicit probability	Same sufficient statistics as the direct form, plus irrelevant metadata	“Event A is true.” $P(\mathrm{True})=0$
Counts	Frequency	counts: zor $=800$ , nif $=200$ ; sampling is uniform over objects.	“The sampled object is a zor.” $P(\mathrm{True})=0.8$
Scaled counts	Frequency	Equivalent count representation with all counts scaled: zor $=80000$ , nif $=20000$ .	“The sampled object is a zor.” $P(\mathrm{True})=0.8$

Appendix

Table 5: Examples of representation types in Sys1Cal-v1. Each row shows one rendered state format used to express an exact latent probability problem. The proposition and gold distribution are shared by the three primitives Noul, Choice, and Score.

Output	Mean pairwise TV	Mean max TV
Choice	0.0998	0.181
Noul	0.0441	0.0787
Score exp.	0.0656	0.116
SemIf Choice	0.117	0.194

Appendix

Table 6: Representation sensitivity for Jev and SemIf. Mean pairwise TV is the average distance between equivalent renderings of the same latent problem.

Figure 5: Representation sensitivity by primitive. Choice and SemIf are more sensitive to equivalent renderings than Noul or Score expectation.

Explore similar work

Evaluating and Benchmarking the System One Model Jev
Sep 29, 2026Tobias Deußer, Lorenz Sparrenberg, Rafet SifaNatural Language Inference Rubric-Based Scoring
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany
Do System One Decisions Add Up? A Study of Probabilistic Coherence
Sep 27, 2026Saman Sarker JoyConfidence Calibration Decisions
Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia
Jev in Medicine: A Benchmark Evaluation
Sep 27, 2026Alfredo Madrid-García, Beatriz Merino-BarbanchoDiagnosis Preliminary Study
Independent researcher Madrid, Spain

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

Evaluating and Benchmarking the System One Model Jev

Tobias Deußer, Lorenz Sparrenberg, Rafet Sifa

University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany

Natural Language Inference Rubric-Based Scoring Artificial Intelligence Systems

Open paper AskPDF arXiv

Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities the vendor describes as calibrated. Such models target small decisions in information access pipelines, such as routing queries, checking grounding, moderating content, or rating against a rubric. We evaluate Jev (jev-1.13.0) zero-shot on 37 datasets spanning classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal clause analysis and rubric scoring, with one frozen template per dataset and full evaluation splits: 346,009 requests for under USD 10. For reference, we score Qwen3.8-27B and Gemma-4-E4B on identical requests via their exact next-token probabilities over the options. Jev reaches 95-99% accuracy on IMDB, SST-2, HellaSwag and ARC and 86.7% on Belebele across 122 languages. It beats Qwen on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals, and Gemma on all 37. All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments. Jev's choice probabilities are well calibrated and support selective prediction. Binary probabilities rank well but are poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder. Rotating the options leaves Jev's accuracy unchanged and withholding the question drops it to near chance, ruling out shallow memorization but not memorized question-answer pairs. We release the code, harness and all raw responses.

Abstract

Explore similar work

Evaluating and Benchmarking the System One Model Jev

Do System One Decisions Add Up? A Study of Probabilistic Coherence

Jev in Medicine: A Benchmark Evaluation