Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Sharath M Shankaranarayana; Davor Runje; Jan Jannink

cs.AIOct 1, 2026

Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Authors: Sharath M Shankaranarayana, Davor Runje, Jan Jannink

Abstract

Decision models return probabilities intended for routing, abstention and automated action. Calibration makes those probabilities useful on average, but does not establish whether low confidence reflects chance or missing knowledge, nor whether confidence falls when a model moves beyond what it knows. We audit this distinction in Jev, a decision model, with over 15 public datasets and 6 generated task families, with paired interventions that vary the information supplied for a fixed item. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge: with no answer-relevant information it assigns up to 0.80 to a salient option, and on news beyond an observed knowledge boundary it exceeds accuracy by 0.21--0.33, a gap that recalibration on earlier months does not close. Targeted yes/no questions give sharper readouts of the case: whether an outcome is settled (AUROC 1.00) and whether the evidence suffices (0.95, against 0.85 for confidence on the same items). Asking whether Jev knows the answer appears to flag fabricated entities and post-boundary news (0.91), but with realistic names or with dates removed it shows no advantage over answer uncertainty. Black-box knowledge audits therefore need explicit controls for surface cues. Code: https://github.com/Syntheme/beyond-answer-confidence.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. We change what Jev is told for a fixed item, state the chance of an outcome, or probe the knowledge boundary (dashed: observational; cue controls use realistic names and remove dates), and read two outputs from the same call. Confidence predicts correctness on familiar closed-choice tasks but does not show what Jev does not know; questions about the case detect gaps in it, while “do you know?” mostly reads surface cues ( table 1 , fig. 3 ).

Ground-truth contrast	Intended to measure	Validated interpretation	Answer uncertainty	Question	AUROC	Wordings
Fixed but unknown fact vs. future chance event	outcome settled	outcome settled, also inferred from tense alone (1.00)	0.96, only because Jev concentrates on the mode for chance events; ideal distributions identical	settled	1.00	0.99–1.00
Complete vs. incomplete evidence, same paragraph count	evidence sufficient	evidence sufficient, beyond length features (0.60 $\to$ 0.94)	0.85 (confidence, higher = complete)	enough	0.95	0.95–0.97
Pseudo-word made-up vs. obscure real person (same relations)	model’s knowledge	name form: text alone 1.00	0.78	known	0.88	0.87–0.97
Look-alike made-up vs. obscure real person	model’s knowledge	equivalent to answer uncertainty ( $\Delta=-0.02$ )	0.76	known	0.74	—
News past vs. before the boundary	model’s knowledge	explicit dates: date alone 0.997	0.385 (0.615 reversed)	known	0.91	—
same, answer option				“not known”	0.89	—

Table 1: Each contrast, what it was intended to measure, and what the controls show it measures. AUROC for the follow-up question and, on the same items, for answer uncertainty

1-p_{\max}

, or for confidence

p_{\max}

where that is the orientation that separates the contrast (as marked); “wordings” is the range over paraphrased, reversed and adversarial versions. The contrasts are different tasks, so their AUROCs are not directly comparable. Both news rows use the same 3,360 questions (1,680 per period); wordings for known were measured on all real subjects (AUROC 0.91 there). Fabricated-versus-real status and period are proxies for what the model knows, not direct labels of it. The cue-controlled rows (look-alike names, dates removed) are exploratory analyses added after the main studies.

	Banking77 ( $K$ =77)			CLINC150 ( $K$ =150)
Code carries	acc	$p_{\max}$	$\mathrm{smECE}$	acc	$p_{\max}$	$\mathrm{smECE}$
nothing	.010	.320	.310	.008	.363	.355
1 example	.681	.828	.170	.911	.904	.024
8 examples	.899	.927	.037	.979	.974	.016
name	.821	.893	.084	.926	.933	.025
name + 8	.909	.939	.039	.981	.978	.015

Table 2: Intent dose–response: accuracy, mean confidence and SmoothECE by what each intent code carries. With swapped examples, accuracy and SmoothECE are scored against the original labels; the near-zero accuracy shows that Jev follows the swapped mapping, not a calibration failure.

Setting	Ideal	$p_{\max}$
Opaque codes (Banking77/CLINC150)	.013/.007	.32/.36
same, open model	.013/.007	.017/.010
Fixed but unknown generated fact	.25	.49
Fabricated entities (PopQA templates)	.25 †	.52
Winner of a made-up future event	.25 §	.76
Fair die, all 720 option orders	.17	.80

Table 3: Top probability when nothing in the input identifies the answer. Where a hidden answer exists, the ideal is uniform over the options not ruled out. † No valid answer exists (false premise); the options are real people, so 0.25 is a uniform forced-choice reference, not a calibrated target. § The outcome is not yet decided; the contenders are interchangeable made-up names by construction, so 0.25 is the symmetric value.

Figure 2: Knowledge boundary. (a) PopQA: confidence tracks accuracy across popularity quintiles, but fabricated entities still get 0.52 on a top option, while $P(\text{known})$ drops to 0.20. (b) Daily Oracle yes/no questions by quarter: after the observed accuracy break (change point, around late 2024) accuracy falls to chance while confidence rises ; $P(\text{known})$ declines steadily, a signal carried mostly by the dates in the questions ( section 4.2 ).

Panel of Figure 3 — Figure 3: Controlling for surface cues. On the original contrasts (upper two rows) a text-only classifier that never queries Jev separates made-up from real subjects and later from earlier news almost perfectly, and known scores 0.88–0.91. With realistic names and with explicit temporal expressions removed (lower two rows), the text classifier weakens and known falls to the level of answer uncertainty ( $1-p_{\max}$ ). Names rows use the same 555 obscure real people as controls (text-only for pseudo-word names: against the least popular real subjects). Answer uncertainty below 0.5 for news with dates means confidence is higher after the boundary. Both news rows use the same 3,360 questions (1,680 per period); the date-only text baseline is from the full panel, and the masked-text baseline is month-grouped (0.83). 95% intervals; for masked news the text classifier still uses topic cues. Exploratory analyses, added after the main studies.

Figure 4: Daily Oracle yes/no questions around the change point (2024-11). (a) Accuracy at equal confidence (95% month-cluster intervals): lower and nearly flat after the change point, so a common map of confidence cannot match accuracy at each value on both sides. (b) Accuracy at matched coverage as the share of post-change questions varies: a “not known” option beats a confidence threshold when such questions are common and beats thresholding “do you know?” at every mix. The dotted line marks the share in our date panel, not a deployment prevalence.

Figure 5: Knowledge $\times$ chance scenarios (480 generated questions). (a) Stated chances of a future draw vs. Jev’s Choice probabilities: nearly all mass on the likeliest outcome, ties broken towards the outcome mentioned first. (b) A fixed fact, progressively revealed: right direction, too confident throughout. (c) The settled question separates fixed-but-unknown answers from future draws perfectly.

Stated	0	.1	.2	.3	.4	.7	MAE
Yes/no per outcome	.01	.09	.16	.25	.36	.68	.030
Score per outcome	.05	.13	.22	.34	.45	.74	.028
Instructed Choice	.00	.00	.00	.01	.99	1.0	.230

Table 4: Mean reading by stated chance (480 scenarios).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Set	$n$	mean of 3	single, 2.5–97.5%	estimate $<0.05$
Banking77, 8 examples	770	0.037	0.031–0.040	100%
CLINC150, 1 example	1,500	0.025	0.021–0.026	100%
PopQA	14,267	0.021	0.021–0.025	100%
Daily Oracle, after change point	1,680	0.305	0.304–0.310	0%
TriviaQA	9,960	0.031	0.030–0.032	100%
SimpleQA Verified	1,000	0.051	0.048–0.061	11%

Appendix

Table 5: SmoothECE of the mean of three calls and of one random call per item (500 draws).

Maps allowed	In-sample	Random months (descriptive), mean / 95th pct	Cross-fitted mean (range)
Value-wise (each value its own level)	0.066	0.034 / 0.038	0.170 (0.147–0.191)
Contiguous partitions (exact; lower bound for monotone maps)	0.047	0.005 / 0.013	0.057 (0.046–0.074)
Arbitrary groupings, 2–20 levels (search)	0.043–0.046	—	0.093–0.099

Appendix

Table 6: Period-conditional calibration error of confidence-only recalibration maps on dated news.

$p_{\max}$ band	$n$ before	acc. before	$n$ after	acc. after	Difference [95% CI]
0.5–0.6	988	0.526	206	0.466	0.060 [ $-0.014$ , 0.139]
0.6–0.7	943	0.621	204	0.456	0.166 [0.100, 0.228]
0.7–0.8	777	0.699	224	0.469	0.230 [0.152, 0.306]
0.8–0.9	821	0.764	370	0.484	0.280 [0.236, 0.327]
0.9–1.0	1,111	0.880	676	0.571	0.309 [0.260, 0.354]

Appendix

Table 7: Accuracy by answer-probability band before and after the change point (month-cluster intervals).

Contrast (items)	Text only	known	$1-p_{\max}$	Second look
Pseudo-word made-up vs. all real (1,600 / 14,267)	1.000	0.911	0.746	—
Pseudo-word made-up vs. real, second-look subset (1,600 / 2,000)	—	0.916	0.747	0.80
Pseudo-word made-up vs. least popular real	1.000	0.833	0.685	—
Pseudo-word made-up vs. obscure real persons (600 / 555)	—	0.884	0.784	—
Look-alike made-up vs. obscure real persons (600 / 555)	0.492	0.741	0.761	—
News after vs. before, full panel (1,680 / 4,640)	0.997	0.913	0.389	—

Appendix

Table 8: AUROC for the known contrasts with and without surface cues. Each row uses one set of items (made-up / real, or after / before).

Condition	Period	Coverage	Accuracy	Conf. errors (all)	Conf. errors (answered)
Base	before	100%	0.697	2.4%	2.4%
Base	after	100%	0.511	17.3%	17.3%
Today’s date given	after	100%	0.511	12.7%	12.7%
“Not known” allowed	before	64.3%	0.740	0.3%	0.5%
“Not known” allowed	after	6.6%	0.631	0.1%	0.9%
Date + “not known”	after	28.5%	0.491	0.1%	0.2%

Appendix

Table 9: The “not known” option on dated news: coverage, accuracy on answered questions and confident errors (wrong with

p_{\max}\geq 0.9

Set	$K$	$n$	Accuracy	Brier	Uniform	Log loss	Uniform
TriviaQA	4	9,960	0.955	0.067	0.750	0.132	1.386
PopQA (real subjects)	4	14,267	0.708	0.378	0.750	0.719	1.386
Daily Oracle yes/no	2	6,320	0.651	0.474	0.500	0.710	0.693
HotpotQA, non-supporting paragraphs	2	951	0.751	0.342	0.500	0.542	0.693
HotpotQA, both supporting paragraphs	2	951	0.958	0.071	0.500	0.193	0.693
MMLU-CF	4	10,000	0.776	0.377	0.750	1.075	1.386

Appendix

Table 10: Multiclass Brier score and log loss of the three-call mean distribution, with the values for a uniform answer over

K

options. Log loss uses

-\log\max(p,10^{-4})

without renormalisation, because Jev returns exact zeros; the floor therefore bounds, rather than measures, the loss on those items. With floors of

10^{-2}

10^{-6}

, MMLU-CF log loss ranges from 0.80 to 1.34 (5.8% of answers have probability zero on the correct option); TriviaQA (0.130–0.135) and dated news (0.709–0.711) barely change.

Figure 6: Intent dose–response reliability by what each intent code carries (about ten equal-count bins, with bins that would split identical confidence values merged; 95% intervals).

	Banking77				CLINC150
Code carries	acc	$p_{\max}$	$\tilde{H}$	$\mathrm{smECE}$	acc	$p_{\max}$	$\tilde{H}$	$\mathrm{smECE}$
nothing	0.010	0.320	0.774	0.310	0.008	0.363	0.592	0.355
1 example	0.681	0.828	0.109	0.170	0.911	0.904	0.057	0.024
2 examples	0.796	0.879	0.074	0.099	0.958	0.953	0.028	0.019
4 examples	0.861	0.914	0.056	0.066	0.969	0.964	0.021	0.015
8 examples	0.899	0.927	0.046	0.037	0.979	0.974	0.016	0.016

Appendix

Table 11: Intent dose–response, all conditions (mean of three calls).

Figure 7: Evidence in context. (a) HotpotQA: supporting paragraphs raise accuracy and confidence together; non-supporting paragraphs reduce accuracy without a corresponding confidence drop, and partial evidence remains overconfident; $P(\text{enough})$ rises only with complete evidence. (b) Quizbowl: confidence and accuracy rise together clue by clue.

Figure 8: Standard multiple choice. (a) Reliability: MMLU-Redux on the diagonal; MMLU-CF and ANLI below it. (b) CLINC150 with 5–150 options.

Setting	$n$	Accuracy	Confidence	Gap	SmoothECE
Fabricated entities (uniform reference 0.25)	1,600	—	0.518	—	—
Daily Oracle yes/no, after the change point	1,680	0.511	0.816	$+0.305$	0.305
MMLU-CF	10,000	0.776	0.914	$+0.138$	0.145
ANLI rounds 1–3	3,200	0.739	0.839	$+0.100$	0.105
HotpotQA, non-supporting paragraphs	951	0.751	0.830	$+0.079$	0.083
HotpotQA, one of two needed paragraphs	951	0.775	0.871	$+0.096$	0.104

Appendix

Table 12: Edge-of-knowledge settings. Gap = mean confidence

-

accuracy.

Experiment	Units	Replicates	Paid calls	Dates	Planning
Instrument checks	—	varied	402	26 Sep	planned
Intent study, development pilot	6,356	3	7,945	26 Sep	pilot before the freeze
Intent dose–response (test)	27,240	3	76,953	26–28 Sep	plan frozen in a git tag
Out-of-scope detection	3,570	3 formats	10,710	28 Sep	planned
Knowledge $\times$ chance scenarios	6,000	3	18,000	28 Sep	planned ∗
Knowledge boundary (PopQA, fabricated, news)	26,927	3	80,781	28 Sep	planned ∗

Appendix

Table 13: Items, replicates and paid calls per experiment. ∗ Planned before the run, plan committed afterwards (not externally verifiable).

Dataset	Source @ revision	Licence
Banking77 [ 9 ]	PolyAI task-specific-datasets @ 57ec275d	CC BY 4.0
CLINC150 [ 24 ]	clinc/clinc_oos @ 155b9c71	CC BY 3.0
PopQA [ 28 ]	akariasai/PopQA @ 098765c7	none stated
Daily Oracle [ 10 ]	agentic-learning-ai-lab/daily-oracle @ 455b35b2	CC BY 4.0
HotpotQA [ 48 ]	hotpotqa/hotpot_qa @ 1908d6af	CC BY-SA 4.0
Quizbowl [ 33 ]	community-datasets/qanta @ e3c56022	unknown

Appendix

Table 14: Public datasets, pinned revisions and licences.

Explore similar work

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration
Sep 28, 2026Riccardo PorceddaProbability Confidence Calibration
Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy
JevOut: Natural Context Can Flip Decision Models
Sep 24, 2026Zixiang XuDecisions Contextual Reasoning
University of Southern California
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1Large Language Model Judges Judges
Carnegie Mellon University

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Riccardo Porcedda

Department of Excellence L’EMbeDS, Sant’Anna School of Advanced Studies, Italy · Department of Computer Science, University of Pisa, Italy

Probability Confidence Calibration Confidence Estimation

Open paper AskPDF arXiv

The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition

A

for which the exact probability

P(A)

is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers,

P(A)

and

P(\neg A)

are presented as if

P(A)+P(\neg A)=1

, while a term

P(U)\neq0

is missing in the sum. Recovering

P(U)

leads to an improvement of median soft accuracy in \texttt{Choice} answers from

0.771

0.978

, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.

Abstract

Explore similar work

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

JevOut: Natural Context Can Flip Decision Models

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure