Language model benchmarks assign each question a single outcome from one fixed wording, even though the same underlying request can be expressed in many equivalent ways. We argue that the quantity of interest should therefore not be correctness on one wording, but the accessibility of an inquiry, which we define as the probability that a model succeeds across meaning-preserving formulations. The distribution of accessibility across inquiries reveals a structure that accuracy and existing best-, worst-, or average-case prompt scores collapse: how broadly a model's capabilities extend across inquiries and how dependably those capabilities can be elicited. Two models can have the same average performance while one succeeds across more inquiries unreliably and the other succeeds across fewer inquiries but more consistently, leading to opposite preferences depending on whether the task allows multiple retries until one success or when repeated consistent success is required. Across six benchmarks and more than twenty models, this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model looks better depends on whether one success across five formulations is enough or all five must succeed, quantities we call Access@5 and Reliable@5, respectively. For some pairs the two measures diverge significantly, by up to 10 points in opposite directions. Accessibility also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries in ways aggregate accuracy obscures, sometimes broadening the set of inquiries a model can succeed on while making success on them less repeatable. Accessibility turns formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes succeed on from what users can actually depend on.
Figures & tables
Figure 1: Benchmark accuracy is one point inside a capability envelope. Inquiries per model are reliable ( A^q=1 ), brittle ( 0<A^q<1 ), or absent ( A^q=0 ). Black ticks mark benchmark accuracy and blue diamonds the latent share P(A^q>0) ; models are ordered by accuracy. All non-thinking models on a shared inquiry set per benchmark; SimpleQA, MATH-500, and MMLU in Figure 7 .
Figure 2: Similar accuracy can conceal different reliability profiles. Reliability curves R(τ)=P(Aq≥τ) for near-tied models on GSM8K (left; accuracy within 2.8 pp) and BFCL (right; within 2.0 pp). The left end is the latent share, the right end (dot) the reliable share.
Figure 3: Changing the wording changes behavior more than resampling it. Rows are model–benchmark settings; T=0.7 . (a) Probability that two answers differ in correctness, for two samples of the same wording (blue) and for two different equivalent wordings (orange). (b) With five attempts: Access@5 and Reliable@5 over five different wordings minus the same over five samples of one wording. Only paraphrases count as different wordings. Bars: 95% bootstrap CIs over inquiries. ⋆ : the 95% CI of the difference excludes zero.
Figure 4: Canonical labels reveal only part of what changes under an intervention. Per row, the upper bar shows the share of inquiries whose canonical outcome improves, stays the same, or worsens; the lower bar the share whose accessibility Aq increases, stays the same, or decreases (means over the model changes in the row, count in parentheses). Rows are the benchmarks on which every intervention was run; all benchmarks in Appendix H . Right: change in accuracy and in mean accessibility, Δμ = mean gain − mean loss.
Figure 5: The same change in accuracy can correspond to very different changes in reliable capability. Each point is one model change; x: change in benchmark accuracy, y: change in reliable share R=P(A^q=1) . Points above the dashed line gain reliable capability faster than accuracy (or lose it more slowly). Counts give the changes on each side of the diagonal and exact ties.
Figure 6: Accessibility profiles before and after a model change. Each pair of bars shows the model before (upper) and after (lower) the change, with the same encoding as Figure 1 . Right: change in accuracy and in reliable share (points), and the share of inquiries whose accessibility increases or decreases.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7Figure 8Figure 9
Benchmark
Pairs
Agreement
SimpleQA
460
96.9%
TruthfulQA
232
98.0%
GSM8K
380
97.7%
Appendix
Table 1: Human validation of formulation equivalence. We report the number of annotated formulation pairs and agreement between human judgments and the automatic semantic-verification pipeline.
Table 3: Per-benchmark summary (medians over models, %). R=P(A^=1) is the reliable share and L=P(A^>0) the latent share; accuracy always lies in [R,L] , and the brittle share L−R is the width of that interval. Majority reversal is the share of inquiries whose canonical outcome disagrees with the majority of formulations. Maxima over models in parentheses.
Figure 7: Capability envelopes for all six benchmarks, including SimpleQA and MATH-500 (as in Figure 1 ).
Figure 8: Majority reversals as a share of all inquiries; one dot per model, with the median (value above) and interquartile range.
Figure 9: Majority reversals conditional on the canonical outcome, P(A^<0.5∣Yorig=1) and P(A^>0.5∣Yorig=0) . A model is omitted from a group with fewer than 20 canonical successes (failures); counts below the axis.
Figure 10: All model pairs within 3 points of accuracy: accuracy gap (x) versus reliable-share gap (y). Filled markers: paired bootstrap 95% CI of the reliable-share difference excludes zero. The dashed line marks equal gaps.
Model
Formulations flipped
Inquiries moved (up / down)
Canonical flips
Qwen-3 8B
2.2%
11.6% (6.3 / 5.3)
2.0%
Qwen-3 8B (thinking)
1.7%
8.5% (3.3 / 5.2)
0.9%
Gemma-3 4B
2.3%
11.1% (5.3 / 5.8)
2.4%
GPT-4.1-Mini
1.2%
5.5% (3.0 / 2.6)
1.0%
Appendix
Table 4: Greedy rerun of the same model on the full GSM8K inquiry set. “Formulations flipped”: paraphrases whose outcome differs between the two runs. “Inquiries moved”: share whose estimated accessibility differs (up / down).
Figure 11: RQ2 including greedy decoding. (a) Greedy: repeated runs of the same wording (blue; collected for three models on the canonical wording of a separate sample of 200 inquiries per benchmark; for GSM8K with a 256-token limit, which may inflate repeat disagreement) versus different wordings (orange). (b) T=0.7 samples. (c) Budget of five attempts. ⋆ : the 95% CI of the difference excludes zero ((b) and (c); the repeats in (a) come from a separate sample, so (a) has no paired CI).
Figure 12: Access@k (top) and Reliable@k (bottom) for k≤5 : solid lines distribute the k attempts over k different wordings, dashed lines draw k samples of one wording ( T=0.7 ).
Figure 13: All intervention–benchmark rows (as in Figure 4 ), including smallest-to-largest scaling and GEPA and MIPROv2 separately.
Change
Benchmark
n
Acc
μ
R
L
Access@5
Reliable@5
Aq↑
Aq↓
Scaling
TruthfulQA
10
41.7 → 50.9
40.8 → 48.9
15.5 → 22.5
71.1 → 76.9
65.2 → 72.0
19.3 → 26.8
42%
21%
Scaling
GSM8K
10
88.0 → 94.6
85.7 → 92.6
67.7 → 81.0
95.7 → 97.8
95.1 → 97.6
71.3 → 83.5
24%
5%
Scaling
MATH-500
10
82.5 → 90.7
80.7 → 89.1
64.8 → 77.3
91.8 → 95.9
90.6 → 95.4
67.8 → 79.8
26%
6%
Scaling
MMLU
9
67.2 → 75.3
67.7 → 75.9
47.0 → 56.1
85.8 → 90.4
83.5 → 88.7
50.1 → 59.5
35%
19%
Scaling
BFCL
9
69.7 → 76.3
68.3 → 74.8
50.5 → 60.6
80.6 → 84.1
78.2 → 82.3
55.9 → 65.1
27%
12%
Thinking
TruthfulQA
6
39.5 → 54.0
39.3 → 52.2
13.7 → 23.9
69.7 → 80.0
63.8 → 75.4
17.7 → 28.8
47%
16%
Appendix
Table 5: Model changes by family and benchmark (means over the n changes, in %; before → after). Aq↑ / Aq↓ : share of inquiries whose accessibility increases / decreases.
Figure 14: Paired profiles for consecutive scaling steps.
Figure 15: Paired profiles for enabling native thinking in Qwen3.
Figure 16: Paired profiles for prompted chain-of-thought on TruthfulQA.
Figure 17: Paired profiles for prompt optimization with GEPA and MIPROv2.
Benchmark
K
n
Acc.
μ
Access@K
Reliable@K
SimpleQA
2
309
1.00
1.00
1.00
1.00
3
206
0.95
0.95
0.95
1.00
TruthfulQA
2
235
1.00
1.00
0.48
0.52
3
157
0.00
0.29
0.00
0.29
GSM8K
2
451
1.00
1.00
1.00
1.00
3
300
1.00
1.00
1.00
0.90
Appendix
Table 6: Share of models whose estimate is within 5 points of the full-data value in at least 95% of equal-cost subsamples ( n=N/K inquiries, K formulations each). Acc.: accuracy on the same n inquiries.
Figure 18: Breadth–repeatability decomposition of model changes. Each point is one model change; ΔlogB (x) and ΔlogD (y) sum to Δlogμ , so the dotted line marks unchanged mean accessibility. Corner labels give the number of changes in each quadrant.