KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration

Hengyu Li

cs.MASep 23, 2026

KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration

Authors: Hengyu Li

Organizations: The University of Tokyo

Abstract

KITE queries a typed behavioral kernel once per unique state, then executes populations of any size from the table with event-keyed randomness and common random numbers. An expensive flagship model is reserved for sparse paired anchors that estimate intervention effects. Measured human-model discrepancy is propagated as shared error into every conclusion. Population-experiment cost thus scales with unique states and anchors, while uncertainty is governed by evidence about people rather than Monte Carlo noise. On Epstein experiments with 9,070 participants, anchors covering 1.7% of states reduced effect error by 41% (absolute MAE reduction 0.0125). On 37 held-out SocSci210 experiments, 0.5-1.5% anchor coverage raised captured decision gain from 0.27 to 0.39. The kernel passed content-fidelity criteria in all 15 new countries of a 16-country study. Shared discrepancy yielded retrospective coverage of 93% and 96% at nominal 80% and 90%, versus 29% and 36% from human sampling uncertainty alone. A million agents executed 20 tabulated steps in 0.9 seconds on a laptop. This architecture offers a route to screening candidate interventions before human trials, multi-country content audits, and uncertainty-aware policy comparison at the cost of a few thousand kernel calls with sparse flagship anchors. Property-specific evidence records connect each use to its validation scope, correction provenance, and uncertainty, making these applications auditable.

Figures & tables

Test part	Content/person gate	Prompt	Tips
US, even headlines	Pass	$-0.102$ (fail)	$+0.056$ (fail)
15 countries, odd	Pass (15/15)	$-0.034$ (fail)	$+0.005$ (fail)
15 countries, even	Pass (15/15)	$-0.060$ (fail)	$-0.003$ (fail)
Matched-subset comparison		Effect [95% CI]	Effect [95% CI]
Jev		$-0.025$ [ $-0.058$ , 0.011]	0.015 [ $-0.022$ , 0.053]
Astra		0.052 [0.010, 0.094]	0.045 [0.002, 0.088]

Table 1: Arechar test evidence. Upper panel: kernel gates, with prompt/tips effects in six-point units. Lower panel: models on the same 30-person-per-country-condition subset in the both-new part; intervals come from C3b’s paired analysis. Human references use all eligible respondents.

System	Raw effect MAE [95% CI]	Calibrated MAE	Model-only RMSE
Kernel	0.0305 [0.0242, 0.0412]	0.0376	0.033
Hybrid	0.0180 [0.0132, 0.0289]	0.0183	0.014
Flagship direct audit	0.0200 [0.0153, 0.0314]	0.0326	0.020

Table 2: Epstein’s declared secondary effect endpoint and subsequent robustness analyses. Raw MAE intervals resample participants within wave and arm. Calibrated MAE fits a slope on other waves and predicts the excluded wave; it is a post-run robustness analysis. Model-only RMSE subtracts estimated human sampling variance from observed squared error.

Engine	Agents	Seconds for 20 steps	Agent-steps/s
Vectorized	$10^{4}$	0.007	$2.7\times 10^{7}$
Vectorized	$10^{5}$	0.075	$2.7\times 10^{7}$
Vectorized	$10^{6}$	0.90	$2.2\times 10^{7}$
Scalar	$10^{4}$	0.95	$2.1\times 10^{5}$

Table 3: Execution from the fixed S1 table on a laptop. Each step evaluates both conditions; reported agent-steps follow the source record’s convention. These are timings for tabulated fixed-exposure decisions, not fresh inference or network simulation.

Property	Evidence and permitted interpretation
Content ranking	Kernel passes Arechar test gates; screening within the evaluated domain is supported.
Known intervention direction	Sparse flagship calibration improves Epstein effect error and SocSci210 decision gain; retain endpoint and domain qualifications.
Human effect intervals	D2 improves retrospective SocSci210 coverage; transport to the hybrid or another domain is not established.
Cost–quality comparison	Measured calls, tokens, table execution, and declared human metrics support comparison on these workloads.
Individual/country effects	Individual prediction and cross-country susceptibility comparisons are unsupported.
Repeated exposure/network	Conditional scenarios only; cascade sizes and endogenous network responses are unvalidated.

Table 4: Property-specific evidence record for a misinformation experiment. A supported property does not transfer automatically to another row.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Record	Frozen source	Main requirement and status
G1	frozen.yaml	p3/choice selected on development, commit 389b984 ; marginal and condition gates pass. Published-model comparisons require metric alignment.
G-A	decision_value.yaml	Sign: permutation $p<0.01$ and above null p95; gain: $p<0.01$ and bootstrap interval above zero. Both pass; stronger two-thirds-of-pilot criteria do not. Earlier test sensitivity was known.
C2/C3	arechar_split.yaml , arechar_criteria.yaml	Split fixed before download; criteria committed at d826501 after US/odd calibration. Content/person gates pass; kernel intervention gates fail.
C3b/A3b	flagship_baseline.yaml	Flagship comparison frozen at ffcb4a7 before predictions; existing criteria reused unchanged.
D1	epstein_criteria.yaml , epstein_screens.yaml	Commit 7b5a5a3 ; policies and prediction hashes locked before scoring. Positive policy-value difference with interval excluding zero required. Not shown. Effect MAE remains secondary.
D1 A2	epstein_headline_criteria.yaml	Commit cdc7864 , after arm aggregates but before headline outcomes were examined; positive within-arm correlation with interval above zero required. Not shown.

Appendix

Table 5: Criteria and their evidential status. Paths are relative to materials/configs/ . A freeze concerns a specified analysis or run, not every earlier use of the dataset.

Wave	Human best	Kernel choice (human value)	Hybrid choice (human value)	Audit choice
1	Control	Control (0.059)	Control (0.059)	Control
2	LE	LE (0.128)	LE (0.128)	LE
3	Tips	GN (0.057)	E (0.089)	GN
4	TN	TN (0.113)	TN (0.113)	TN
5	IN	IN (0.101)	I (0.083)	I

Appendix

Table 6: Locked within-wave choices and observed discernment of the chosen arm. E = evaluation; LE = long evaluation; GN = generic norms; TN = tips plus norms; I = importance; IN = importance plus norms. Wave 1 has no sharing intervention. Human choices are descriptive optima using the full sample. Source: D1 decision record and d1-policies-summary.json .

Wave	Intervention	Human	SE	95% interval	$n_{T}/n_{C}$	Kernel	Hybrid	Audit
2	Evaluation	$+0.0455$	0.0175	$[+0.0126,+0.0794]$	395/387	$-0.013$	0.023	0.008
2	Long evaluation	$+0.0624$	0.0172	$[+0.0291,+0.0966]$	410/387	$0.011$	0.025	0.013
3	Evaluation	$+0.0202$	0.0143	$[-0.0072,+0.0483]$	540/533	$-0.015$	0.011	0.015
3	Generic norms	$-0.0114$	0.0139	$[-0.0379,+0.0153]$	510/533	$0.032$	0.008	0.017
3	Tips	$+0.0265$	0.0149	$[-0.0019,+0.0557]$	498/533	$0.023$	0.006	0.016
4	Partisan norms	$+0.0058$	0.0128	$[-0.0198,+0.0302]$	949/487	$0.056$	0.024	0.027

Appendix

Table 7: Ten wave-relative discernment effects on the binary sharing scale. Human standard errors and 95% intervals use a participant bootstrap within wave and arm (4,000 resamples, seed 0);

n_{T}/n_{C}

gives treated/control participants. Controls are wave-specific. Human estimates come from results/d1/human_effect_intervals.json , as supplied in RESOLUTIONS.md ; model effects retain D1 rounding.

System	Policy value	Chance mean/p95	One-sided $p$	Same judge
Kernel	0.0915	0.0808/0.0944	0.105	0.0912
Hybrid	0.0942	0.0809/0.0944	0.062	0.0938
Direct audit	0.0878	0.0807/0.0942	0.197	0.0875
Human half-sample	—	—	—	0.0941
Hybrid minus kernel (primary)			0.0027 [ $-0.0118$ , 0.0169]
Hybrid minus direct audit			0.0063 [ $-0.0079$ , 0.0211]

Appendix

Table 8: D1 policy-value results. Intervals for differences use 2,000 participant bootstraps. Chance references use 5,000 within-wave permutations; same-judge values use 200 splits. No system exceeds its chance baseline at

p<0.05

Personas per wave	Kernel MAE	Hybrid MAE
10	$0.0307\pm 0.0025$	$0.0180\pm 0.0006$
20	$0.0308\pm 0.0011$	$0.0181\pm 0.0002$
30	$0.0307\pm 0.0014$	$0.0180\pm 0.0003$
50	$0.0308\pm 0.0015$	$0.0181\pm 0.0002$
100	$0.0307\pm 0.0006$	$0.0180\pm 0.0001$
200	0.0305	0.0180

Appendix

Table 9: Persona-count robustness with fixed anchors. Entries at fewer than 200 personas are means

\pm

standard deviations across 20 subsamples, not confidence intervals; the full-table row is deterministic conditional on recorded predictions. Source: robustness.json .

System	Within-task $r$	Sign	Gain	Raw MAE	Magnitude ratio	$r_{F}$
Kernel	0.397	0.671	0.268	0.052	0.68	0.608
$F_{5}$	0.489	0.671	0.336	0.078	1.64	1.000
$F_{3}$	0.461	0.697	0.339	0.086	1.87	0.927
$H_{1}$ (694 items)	0.433	0.717	0.382	0.086	1.85	0.701
$H_{2}$ (1,388 items)	0.417	0.742	0.386	0.084	1.80	0.721
$H_{3}$ (2,082 items)	0.423	0.709	0.386	0.083	1.75	0.709

Appendix

Table 10: D3 test results from report.json : 37 studies, 162 blocks, 532 effects, 304 reliable contrasts.

F_{k}

uses

k

unpaired flagship respondents per cell;

H_{k}

uses

k

paired anchors per condition plus the kernel table.

r_{F}

is correlation of effects with

F_{5}

. Calibration uses development slopes only.

System	Within-task $r$	Sign accuracy	Captured gain	Effect MAE	$r_{F}$
Kernel	[0.061, 0.613]	[0.495, 0.801]	[0.051, 0.446]	[0.041, 0.065]	[0.323, 0.780]
$F_{5}$	[0.188, 0.675]	[0.511, 0.788]	[0.117, 0.519]	[0.064, 0.094]	[1.000, 1.000]
$F_{3}$	[0.182, 0.639]	[0.556, 0.806]	[0.105, 0.532]	[0.073, 0.101]	[0.884, 0.958]
$H_{1}$	[0.201, 0.616]	[0.564, 0.833]	[0.170, 0.560]	[0.069, 0.109]	[0.540, 0.822]
$H_{2}$	[0.174, 0.611]	[0.607, 0.840]	[0.162, 0.573]	[0.065, 0.108]	[0.565, 0.838]
$H_{3}$	[0.189, 0.607]	[0.548, 0.823]	[0.164, 0.566]	[0.066, 0.103]	[0.550, 0.832]

Appendix

Table 11: D3 test 95% study-bootstrap intervals for each system. Magnitude-ratio intervals are not supplied.

F_{3}

and

F_{5}

share respondents, inflating their approximation correlation.

Comparison	Sign accuracy	Captured gain	Within-task $r$
$H_{3}-K$	0.038 [ $-0.044$ , 0.122]	0.119 [0.014, 0.224]	0.026 [ $-0.120$ , 0.213]
$H_{2}-K$	0.071 [ $-0.012$ , 0.155]	0.118 [0.006, 0.224]	0.020 [ $-0.120$ , 0.204]
$H_{1}-K$	0.046 [ $-0.028$ , 0.121]	0.114 [0.014, 0.215]	0.035 [ $-0.104$ , 0.217]
$H_{3}-F_{5}$	0.038 [ $-0.011$ , 0.092]	0.050 [ $-0.018$ , 0.117]	$-0.066$ [ $-0.129$ , 0.017]
$H_{2}-F_{5}$	0.071 [0.022, 0.128]	0.049 [ $-0.009$ , 0.115]	$-0.073$ [ $-0.128$ , 0.012]
$F_{5}-K$	0.000 [ $-0.062$ , 0.060]	0.069 [ $-0.037$ , 0.168]	0.092 [ $-0.048$ , 0.253]

Appendix

Table 12: Paired differences with study-bootstrap intervals. Comparisons with

F_{5}

use the declared 90% level; other rows use 95%. These are secondary endpoints, except the

H_{3}

approximation gain reported in the main text.

System	Within-task $r$	Sign	Gain	MAE	Magnitude ratio	$r_{F}$
Kernel	0.552	0.895	0.420	0.067	0.61	0.863
$F_{5}$	0.607	0.939	0.529	0.079	1.15	1.000
$F_{3}$	0.592	0.937	0.504	0.084	1.21	0.979
$H_{1}$	0.533	0.927	0.409	0.080	1.14	0.897
$H_{2}$	0.551	0.927	0.446	0.081	1.15	0.911
$H_{3}$	0.561	0.933	0.496	0.079	1.13	0.917

Appendix

Table 13: D3 development results, not independent confirmation: 91 seen studies, 236 blocks, 2,195 effects. Source: dev-20260923-022128.json .

Fit	$\beta$	$\tau$	$\sigma$
Jev, selected seen	0.925 [0.607, 1.173]	0.037 [0.021, 0.046]	0.080 [0.051, 0.128]
Astra, selected seen	0.579 [0.317, 0.660]	0.035 [0.020, 0.046]	0.074 [0.047, 0.125]
Jev, unselected dev	0.661	0.020	0.020

Appendix

Table 14: D2 parameter fits and bootstrap intervals. Selected seen studies use five model respondents per cell. The unselected fit uses 20 development studies with 50 respondents per cell. Source: discrepancy_model.json .

Evaluation	50%	80%	90%	Width (90%)	Score (90%)
Jev, split half	0.581	0.808	0.882	—	—
Astra, split half	0.599	0.839	0.902	—	—
Jev, test	0.669	0.932	0.959	0.324	0.422
Astra, test	0.625	0.859	0.937	0.295	0.394
Jev, no discrepancy	0.140	0.288	0.362	0.064	0.746
Astra, no discrepancy	0.204	0.334	0.397	0.064	0.754

Appendix

Table 15: Study-weighted coverage and 90% interval diagnostics. Test discrepancy intervals include parameter uncertainty; split-half results summarize the seen-study calibration. No-discrepancy intervals include human sampling uncertainty only. Interval score penalizes width and misses.

Respondents per cell	5	10	20	50	100	200
Sign accuracy	0.655	0.674	0.632	0.645	0.661	0.671
Captured gain	0.272	0.358	0.264	0.255	0.278	0.268
Cost (USD)	0.0007	0.0015	0.0029	0.0074	0.0147	0.0295

Appendix

Table 16: G-A’s test decision-value curve. Entries reuse nested subsets of the same recorded kernel run and are not independent experiments. Costs are per simulated experiment at the recorded average of 4.3 conditions.

Comparable-block $r$ : Jev 0.347; Luna 0.495; Astra 0.489.
Model	$W_{1}$	Condition $r$	Magnitude ratio	Sign	Gain
Jev p3	0.1360	0.323	0.89	0.645	0.272
GPT-5.6 Luna (low)	0.1320	0.506	1.61	0.697	0.305
GPT-6 Astra (high)	0.1013	0.489	1.58	0.671	0.336
Astra–Jev $r$ difference: 0.142 [0.014, 0.289] (95%).
Astra–Jev sign difference: 0.026 [ $-0.031$ , 0.083] (95%).

Appendix

Table 17: Three-model comparison (A3/A3b), from three_models.json . The intended common sample has 3,615 items; the historical Jev five-per-cell file lacks two. Upper-panel

r

and magnitude ratio use the marginal scorer. Lower-panel

r

uses comparable decision blocks; its paired differences therefore do not equal differences between all upper-panel entries.

Study	Independent magnitude	Corrected	Independent structure	Corrected	Distant/human
m52pd	0.175	0.289	0.018	0.518	0.274
3xy9j	0.548	0.645	0.005	0.263	0.611
gx6hp	0.456	0.786	0.001	0.514	0.791
ef6my	0.305	0.345	0.089	0.410	0.326
5mt6r	0.338	0.523	0.132	0.711	0.471
ac9jm	0.122	0.281	$-0.067$	0.718	0.258

Appendix

Table 18: B2 memory replication. Magnitude is mean absolute inter-item Spearman correlation relative to humans; structure is correlation between the simulated and human correlation patterns. “Distant/human” compares distant-pair coherence after correction with people’s. Source: B2 decision record.

Explore similar work

Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model
Oct 1, 2026Sharath M Shankaranarayana, Davor Runje, Jan JanninkModel Auditing Self-Knowledge
Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets
Sep 30, 2026Priyanath Maji, Spandan Ghose ChowdhuryThompson Sampling Token Budget Allocation
Georgia Institute of Technology Atlanta, GA 30332
Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents
Sep 30, 2026Shuyao Xiao, Shengling Wang, Xuan Chen +8Deep Research Adaptive Sampling
Beijing Normal University · Ke Holdings

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Sharath M Shankaranarayana, Davor Runje, Jan Jannink

Model Auditing Self-Knowledge Uncertainty

Open paper AskPDF arXiv

Decision models return probabilities intended for routing, abstention and automated action. Calibration makes those probabilities useful on average, but does not establish whether low confidence reflects chance or missing knowledge, nor whether confidence falls when a model moves beyond what it knows. We audit this distinction in Jev, a decision model, with over 15 public datasets and 6 generated task families, with paired interventions that vary the information supplied for a fixed item. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge: with no answer-relevant information it assigns up to 0.80 to a salient option, and on news beyond an observed knowledge boundary it exceeds accuracy by 0.21--0.33, a gap that recalibration on earlier months does not close. Targeted yes/no questions give sharper readouts of the case: whether an outcome is settled (AUROC 1.00) and whether the evidence suffices (0.95, against 0.85 for confidence on the same items). Asking whether Jev knows the answer appears to flag fabricated entities and post-boundary news (0.91), but with realistic names or with dates removed it shows no advantage over answer uncertainty. Black-box knowledge audits therefore need explicit controls for surface cues. Code: https://github.com/Syntheme/beyond-answer-confidence.

Abstract

Explore similar work

Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets

Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents