What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

Seyed Mosayeb Alam

cs.CLSep 5, 2026

What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

Authors: Seyed Mosayeb Alam

Organizations: KTH Royal Institute of Technology

Abstract

Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown. The audit flags 36 items and separates two failures a single flag would conflate: evidence genuinely absent from the window and answers that must be computed from numbers the window does supply. Flagged items change their answers far more often, wobbling at 0.255 against 0.087 on the 427 clean items, and excluding them cuts apparent cross-model agreement by about a fifth. Before testing whether the missing evidence explains the instability, we registered a prediction: re-cut each window to hold its evidence, and instability should fall below a set threshold. It failed: the repair moved wobble by 0.058, with an interval containing zero. We report the association as correlational. Almost all measurements sit where instability cannot show, which bounds what a corpus built for accuracy can say about stability. We release the corpus, all 112,800 raw responses, and the audit as a runnable check for any document benchmark.

Figures & tables

Model	Serving	Items	Unstable	Acc.
gemma4-31b	routed	$426$	$6$	$0.965$
haiku-4.5	direct	$425$	$6$	$0.962$
llama3.3-70b	routed	$426$	$8$	$0.953$
gemini3-flash	routed	$426$	$9$	$0.965$
mistral-large	routed	$427$	$12$	$0.944$
gpt5-mini	routed	$427$	$20$	$0.965$

Table 1: Twelve configurations on the 427 items that pass the provenance audit (52 of 60 tasks retain at least one item), 20 runs per item at temperature 0.7. Unstable counts items whose answer is not identical across all 20 runs; accuracy is majority-vote correctness over the same runs. Wobble, the ratio of the two counts, is Figure 1 . The rule marks the order-of-magnitude gap; the two 1B rows are quantizations of one base model, so the twelve configurations span eleven distinct models.

Figure 1: Wobble by configuration on the audited corpus, ascending, with Wilson 95% intervals (

n=427

items). The axis is broken: the ten frontier configurations occupy

0

0.145

and the two 1B ones resume at

0.25

. A logarithmic axis would fit them on one scale and compress the order-of-magnitude difference that is the panel’s point. The break is drawn instead, and never where a frontier interval would cross it. Hatching marks the 1B group so the grouping survives a greyscale print. Item counts and accuracy are in Table 1 .

Figure 2: The flagged-against-clean contrast and the pre-registered repair. Both panels are on the wobble scale. (a) Wobble by provenance group with Wilson 95% intervals, item counts on each bar, ordered by what the window supplies rather than by effect size; the dotted rule is the clean rate (

n=427

). Item-bootstrap intervals for the same groups are in Table 2 . The pooled flagged-against-clean contrast is 0.167, [0.128, 0.211] by Newcombe’s method. (b) The pre-registered repair on the 10 repairable items, 20 runs each under twelve configurations: original window against a re-cut window of equal length. Both bars carry the flagged colour because re-cutting does not make an item clean. The interval on the re-cut bar is the change , [

-

0.049, 0.165], drawn against the heavy rule at the original rate, the no-change position, which it crosses. A positive change is a fall in wobble. The dotted rule is the threshold registered before the runs.

	Stratum	Cfg.	f/c	Flagged	Clean	$\Delta$ (Newc.)	$\Delta$ (boot)
wobble	pooled	12	36/427	0.255 [.185,.329]	0.087 [.077,.098]	0.167 [.128,.211]	[.097,.243]
wobble	numeric	12	30/118	0.283 [.205,.378]	0.093 [.073,.115]	0.190 [.143,.241]	[.109,.282]
wobble	non-numeric	12	6/309	0.111 [.042,.181]	0.085 [.074,.097]	0.026 [ $-$ .029,.119]	[ $-$ .046,.103]
wobble	pooled	10	36/427	0.144 [.072,.231]	0.036 [.027,.045]	0.109 [.075,.149]	[.036,.198]
wobble	numeric	10	30/118	0.163 [.070,.270]	0.043 [.026,.063]	0.121 [.081,.168]	[.026,.229]
wobble	non-numeric	10	6/309	0.050 [.017,.083]	0.033 [.023,.044]	0.017 [ $-$ .017,.104]	[ $-$ .022,.056]

Table 2: The flagged-against-clean contrast, stratified. Brackets on the Flagged and Clean cells are 95% intervals from a bootstrap over items (2,000 draws, resampling flagged and clean items independently; narrower than Wilson where items are homogeneous, degenerate at

n=2

), the resampling unit because each item contributes one pair per configuration and those pairs are not independent; the difference column carries the Newcombe interval on the two Wilson intervals beside the item-bootstrap interval for the same difference. Cfg. 12 is every configuration, 10 drops the two 1B configurations. The three failure-type rows split the flagged items by cause (Section 3 ), each against the whole clean set, pooled configurations only. Newcombe’s method applies to proportions; dispersion is a mean, so its difference carries the bootstrap interval only. Dispersion is the share of runs disagreeing with the modal answer, averaged over the same model-item pairs as wobble. The numeric stratum holds 83.3% of the flags; the non-numeric flagged group has 6 items and the label-untraceable group 2, reported for completeness, not weight.

Figure 3: (a) Distribution of the per-item correct rate

p

over the 5,116 measured pairs of one configuration with one audited item. The shaded region is the observable band,

0.3\leq p\leq 0.7

. The labelled bar is the bin

p\geq 0.9

; the 86.0% quoted in Section 4 is the narrower set with

p=1

. (b) Largest observed and largest structurally reachable wobble per accuracy decile, the structural maximum being the most wobble an item pool with the observed

p

values could produce. The axis starts at 0.6 because no task is answered correctly less often than that, and the four deciles shown are every one containing a task.

$k$	Mean	Chance	Ratio	$p$	95% CI	Trunc.
$3$	0.144	$0.014$	$10.23$	$<$ 0.00050	[.042,.261]	0/90
$4$	0.161	$0.019$	$9.01$	$<$ 0.00050	[.066,.266]	18/90
$5$	0.187	$0.024$	$8.91$	$<$ 0.00050	[.102,.278]	36/90

Table 3: Cross-model transfer of instability, frontier configurations only (10 models, 90 ordered pairs, 425 clean items), at each of the 3 values of

k

the corpus supports. Each pair asks what share of configuration

A

’s

k

least-stable items fall inside

B

’s

2k

least-stable. Chance is the expected share under the effective

k

after truncation, Ratio is observed hits over expected, and the permutation

p

is over 2,000 shuffles of each configuration’s own values. The interval is a bootstrap over the ten configurations , not the 90 pairs, which are not independent. Trunc. counts pairs where a configuration had fewer than

k

unstable items and the set was truncated rather than padded.

k=5

is the pre-specified value. The claims this benchmark could not support are Table 4 .

Claim tested	Point estimate	Under the conservative reading	Status
Re-windowing lowers wobble to the registered level	0.208 after, 0.267 before	change 0.058 [ $-0.049$ , $0.165$ ] contains zero, and 0.208 exceeds the registered 0.161	not supported
Worst category runs $N\times$ the mean	median 2.64 $\times$	0.60 $\times$ at the Wilson bounds; 0/12 above $2\times$	not supported
The gap grows as models improve	slope 3.43	intercept [ $-0.082$ , $0.049$ ] contains zero ( $R^{2}$ 0.680, $n$ = 10)	not supported
Instability is not merely difficulty	$\rho=-0.487$	residual $\rho=0.189$ after the structural term	conceded
Correct-but-unstable answers do not occur	0 tasks observed	ceiling 0.115 $<$ 0.30 threshold	unreachable by construction

Table 4: Claims tested against this benchmark and not supported by it, our own pre-registered prediction first. Point estimate is the number a reader of the aggregate would quote; under the conservative reading is what survives attacking it. For the worst-category claim that means the largest Wilson lower bound over estimable categories against the Wilson upper bound of the mean, which avoids selecting a category on the same data used to score it. Conceded marks a claim we do not make; unreachable by construction marks one this corpus cannot test at all, for the reason given in Section 4 .

Figure 4: Mean transfer at

k=5

against its permutation null, on 425 clean items. The histogram is the null itself, 2,000 permutations that shuffle each configuration’s dispersion values among its own item positions, preserving its value multiset and therefore its effective

k

. The observed 0.187 sits outside every draw; the reported

p<0.0005

is the resolution of 2,000 permutations rather than a computed tail. The horizontal span is the 95% bootstrap interval, [0.102, 0.278], resampled over the ten configurations rather than the 90 ordered pairs, which are ten models crossed with themselves and are not independent.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	cap table (1)	convertibles (12)	exit waterfall (1)	founder equity (5)
gemma4-31b	0.000 [.000,.324]	0.000 [.000,.039]	0.000 [.000,.658]	0.000 [.000,.077]
haiku-4.5	0.000 [.000,.324]	0.000 [.000,.040]	0.000 [.000,.658]	0.000 [.000,.077]
llama3.3-70b	0.000 [.000,.324]	0.000 [.000,.039]	0.000 [.000,.658]	0.022 [.004,.113]
gemini3-flash	0.000 [.000,.324]	0.000 [.000,.039]	0.000 [.000,.658]	0.000 [.000,.077]
mistral-large	0.000 [.000,.324]	0.000 [.000,.039]	0.000 [.000,.658]	0.065 [.022,.175]
gpt5-mini	0.000 [.000,.324]	0.000 [.000,.039]	0.000 [.000,.658]	0.043 [.012,.145]

Appendix

Table 5: Wobble by configuration and clause category on the audited corpus, each rate followed by its Wilson 95% interval with leading zeros dropped. The eight categories are split across two panels; every panel lists all twelve configurations, ordered by aggregate wobble. Parenthesised numbers are the tasks each category retains after the provenance exclusion. cap table and exit waterfall fall below the three-task estimability floor and are shown for completeness only: no worst-category claim in this paper uses them. Intervals overlap almost everywhere, which is why no ordering between categories is asserted in the text.

Quantity	Incl. flagged	Audited
Items	463	427
Tasks	60	52
Model-item pairs	5548	5116
Tasks with non-zero wobble	57	50
Split-half reliability	0.904	0.903
Share at $p=1$	0.846	0.860

Appendix

Table 6: Every headline quantity under both readings. The provenance audit flags 36 of 470 items. Both columns drop the 7 items removed at author adjudication: incl. flagged keeps the 36 flagged items and audited excludes them, so their item rows read 463 and 427. Pooled over all twelve configurations, wobble on flagged items is 0.255 [0.216, 0.298] against 0.087 [0.080, 0.095] on the rest, a difference of 0.167 [0.128, 0.211], computed by Newcombe’s method on the two Wilson intervals. The interval excludes zero, so the audited reading is the paper’s headline and this table is the record of what that choice moved.

Task	Items	Cells	Original window	Repaired window	Change
convert vs. preference	2	24	0.000 [0.000, 0.138]	0.083 [0.023, 0.258]	$-$ 0.083 [ $-0.258$ , $0.067$ ]
multi-round dilution	5	60	0.167 [0.093, 0.280]	0.200 [0.118, 0.318]	$-$ 0.033 [ $-0.172$ , $0.107$ ]
option pool shuffle	3	36	0.611 [0.449, 0.752]	0.306 [0.180, 0.469]	0.306 [ $0.075$ , $0.494$ ]
Pooled	10	120	0.267 [0.196, 0.352]	0.208 [0.145, 0.289]	0.058 [ $-0.049$ , $0.165$ ]

Appendix

Table 7: The re-windowing repair, 10 items each measured in both conditions by all twelve configurations (120 paired cells). The repaired window is cut to the same length as the original and centred on the validating quote, so the only difference is whether the evidence is inside it. Registered before the experiment ran: the repaired rate would fall below 0.161. It is 0.208 [0.145, 0.289], above that threshold, so the prediction fails. The pooled change is 0.058 [

-0.049

0.165

] and contains zero. Change is the original rate minus the repaired one, so a positive value is a fall in wobble and a negative value a rise; 2 of the 3 tasks are negative. One task of three moves; the per-task split is descriptive and was not the registered test.

Unit	$k$	Units	Trunc.	Mean	Chance	Ratio	Perm. $p$
items	$3$	$425$	0/90	$0.144$	$0.014$	$10.23$	$<$ 0.00050
	$4$	$425$	18/90	$0.161$	$0.019$	$9.01$	$<$ 0.00050
	$5$	$425$	36/90	$0.187$	$0.024$	$8.91$	$<$ 0.00050
tasks	$3$	$52$	18/90	$0.315$	$0.115$	$2.87$	$<$ 0.00050
	$4$	$52$	27/90	$0.331$	$0.154$	$2.42$	$<$ 0.00050
	$5$	$52$	42/90	$0.369$	$0.192$	$2.28$	$<$ 0.00050

Appendix

Table 8: Transfer at every

k

under both ranking units, over the same 425 clean items grouped into 52 tasks and the same ten frontier configurations, 90 ordered pairs each. A task’s score is the share of its items that flip. Trunc. counts pairs where one side had fewer than

k

non-zero units, so its top-

k

was cut short rather than padded. Chance is recomputed per pair from the effective

k

after truncation, which is why the coarser unit carries a chance line an order of magnitude higher and a correspondingly smaller ratio at the same permutation

p

p

is over 2,000 permutations that shuffle each configuration’s values among its own positions.

Pairing	Pairs	Mean	Median	Cross-fam.	Mean
both directly served	$6$	$0.567$	$0.600$	$4$	$0.550$
one of each	$42$	$0.167$	$0.200$	$42$	$0.167$
both routed	$42$	$0.152$	$0.000$	$42$	$0.152$

Appendix

Table 9: Transfer at

k=5

by how each side of a pair is served, on the clean population. The two rightmost columns repeat the split after the two same-family configurations are removed, which is the check that the directly served result is not one model agreeing with its own sibling. Six ordered pairs from three configurations is the narrowest result in the paper and is reported as underpowered wherever it appears.

Configuration dropped	Left	Mean	Hits	Ratio
full lineup	$10$	$0.187$	$84$	$8.91$
gemini3-flash	$9$	$0.197$	$71$	$9.43$
haiku-4.5	$9$	$0.169$	$61$	$7.81$
gemma4-31b	$9$	$0.186$	$67$	$8.58$
mistral-large	$9$	$0.211$	$76$	$10.22$
llama3.3-70b	$9$	$0.197$	$71$	$9.31$

Appendix

Table 10: Transfer at

k=5

with each frontier configuration dropped in turn, on the clean population. Left is how many configurations remain, Hits the total shared items over the ordered pairs among them, and Ratio those hits against the chance expectation recomputed for that lineup. The first row above the rule is the full lineup, so every row below it is read against that one.

Task	Oracle	Models say	Runs	Floor	Unan.	Adjudication
preference_seniority	pari-passu	stacked	235/237	$17$	yes	excluded
preference_seniority	pari-passu	stacked	224/226	$5$	yes	excluded
participation_type	participating	capped	228/237	$13$	yes	excluded
liquidation_preference_multiple	1x	other	220/229	$11$	yes	excluded
flag_uncapped_participation	yes	no	220/234	$14$	no	excluded
flag_uncapped_participation	yes	no	220/236	$16$	no	excluded

Appendix

Table 11: Items where the configurations’ modal answers agree with each other and contradict the stored oracle label, ranked by run support. Of the 7, 7 have been adjudicated against the source filing. Runs is the number of the responses across all twelve configurations backing the models’ answer. Floor is the weakest single configuration’s own support out of 20 runs, carried as a column rather than applied as a filter: requiring every configuration to hold its answer on at least 18 of 20 admits none of these, and a criterion that empties the table is a result about the criterion. Unan. marks the items where all twelve configurations produced the same normalised answer. A flag is a candidate for human review and never a correction; no label changes until the author has read the filing and upheld it.

Figure 5: Split-half reliability by configuration on the audited corpus, over the 49 tasks with non-zero wobble in at least one configuration. Circles are the observed correlation between the odd-run and even-run task vectors; diamonds are the Spearman-Brown correction to full length. The dotted rule marks the median corrected value. Two configurations sit at exactly 1.000 on both measures, so their circle and diamond coincide. They have four and five tasks with non-zero wobble out of the 49, so both half-vectors are almost entirely ties and the rank correlation is correspondingly easy to saturate; the reported statistic is the median, which does not depend on them.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 802 papers

Sep 14, 2026

Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced accuracy, increased forced declarations, increased tool calls, and increased cost per correct answer. Confidence and benchmark calibration did not fully capture wrong answers; a documented production incident shows fabricated structural claims can be mixed with accurate numeric tables. Agentic evaluations need claim-level receipts (statement-level provenance, not answer-level scores), condition-aware scoring, and human-adversarial verification - an auditing discipline, not a leaderboard. The setting we measure is financial due diligence; the setting we are building toward next is defense staff work, where the same buried-evidence shape appears. In both, the model is not a party to the consequences; the person who signs is. In plain terms: in the documented cases we examine, agents can pair accurate numbers with confident fabricated explanations, and the burden of proof must therefore move from the model to the evidence trail.

Luis M. Sánchez

Toryx Inc.

Data Provenance AI Agent Auditing Factual Consistency Evaluation Agent Evaluation Agentic AI Language Model Auditing

Ask PDF

Sep 27, 2025

WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

Technical-domain benchmarks constructed from arXiv papers can overlap the same public text used in LLM pretraining. Auditing this risk at training-corpus scale requires searching billions of corpus n-grams while retaining per-item evidence that users can inspect and recompute. We contribute a reverse-probe audit at a fixed 13-gram threshold: it indexes benchmark prompts, streams public pretraining corpora, and emits per-problem prompt-surface lexical-overlap metadata with memory that scales with the benchmark. We instantiate the protocol in WirelessMathBench-XL, a 4,027-problem wireless mathematical-reasoning benchmark built from 836 retained arXiv papers across 20 subfields. Against 12.9B streamed 13-grams from RedPajama-arXiv, the audit identifies a strict zero-hit view S0 covering 3,853 problems (95.7%). Filtering to S0 changes accuracy by less than 1 pp for every evaluated model; frontier calibration rows form one high-accuracy cluster between 86.5% and 91.3%, not a resolved rank order. Only 30/800 test items carry detected overlap. Under an all-flagged-correct counterfactual, their largest possible positive score inflation is 0.31-0.51 pp for the frontier rows, so full-versus-S0 is a bounded, structurally underpowered stability summary rather than a contamination-effect test or cleanliness claim. The audit channel does not cover paraphrase, target-answer, post-training, or closed-corpus exposure. The release includes source-paper identifiers, verifier-facing ground truths, audit and threshold metadata, filtered views, a paper-disjoint sensitivity view, evaluation traces, paired-bootstrap scripts, training recipes, Croissant metadata, and a Datasheet for Datasets.

Xin Li, Mengbing Liu, Yiyang Zhu +4

Nanyang Technological University

Mathematical Reasoning Benchmarks Benchmark Auditing LLM Auditing Benchmark Design Benchmark Contamination Benchmark Validity

Ask PDF

Feb 25, 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1

Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Benchmark Validity Benchmark Design

Ask PDF

Abstract

LLM Reliability

Explore similar work

Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity