Multiple-choice question (MCQ) benchmarks have been a standard evaluation practice for measuring LLMs' ability to reason and answer knowledge-based questions. Through a synthetic NonsenseQA benchmark, we observe that different LLMs exhibit varying degrees of label-position-few-shot-prompt bias, where the model either uses the answer position, the label in front of the answer, the distributions of correct answers present in the few-shot prompt, or a combination of all to answer each MCQ question. We propose a simple bias-reduced evaluation protocol that replaces the labels of each question with uniform, unordered labels and prompts the LLM to use the whole answer presented. With a simple sentence similarity model, we demonstrate improved robustness and lower standard deviation between different permutations of answers with a minimal drop in LLM's performance, exposing the LLM's capabilities under reduced evaluation artifacts, without any help from the prompt examples or the option labels. Across multiple benchmarks and models, this protocol substantially improves the robustness to answer permutations, reducing mean accuracy variance 3× with only a minimal decrease in the mean model's performance. Through ablation studies on various embedding models and similarity functions, we show that the method is more robust than the standard ones.
Figures & tables
Figure 1: An example of the wrong conclusion reached by the Gemma-3-12 B-it model on one of the questions from the ARC [ 8 ] dataset with our proposed method. By changing the presentations from standard evaluation (with different labels and predicting only the answer label) to matched (with uniform labels and predicting a whole answer, shown on the left), we eliminate the label bias present, and the model arrives at a correct conclusion that aligns with its reasoning.
Figure 2: An example question from NonsenseQA with random answers and a golden answer chosen at random as ”D. Arms”.
Figure 3: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters (A/B/C/D) as labels (S&L) on NonsenseQA with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 4: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on CSQA [ 29 ] with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Model name
σRCSQA2
σRARC2
σRGPQA2
σRINCLUDE2
σRMMLU−PRO2
xGEOM(σR2)
Deepseek-r1-0528-qwen3
0.005
0.089
0.226
0.033
0.945
0.08
Deepseek-r1-distill-qwen
0.038
0.135
0.184
0.118
0.041
0.09
Llama-3.1-8b-instruct
1.688
0.104
0.645
3.479
2.121
0.96
Ministral-3-8b-base
0.346
0.813
0.658
0.534
0.577
0.56
Nvidia-nemotron-3-nano
0.008
0.066
0.073
0.059
0.620
0.07
Nemotron-cascade-8b
0.105
0.351
1.540
0.154
1.610
0.43
Table 1: We report the variance ratio σR2 for each model, where σR2<1 indicates lower variance under the M&D evaluation protocol, whereas σR2>1 indicates lower variance under the S&L evaluation protocol. To summarize overall model-level trends, we compute the geometric mean of σR2 across all datasets, denoted xGEOM(σR2) . Across all datasets and models, we observe a geometric mean of variance ratio of 0.33 , demonstrating a 3× reduction in models’ accuracy variance under our protocol.
Figure 5: Cross-benchmark Spearman and Kendall Tau rank correlation agreement
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: We identify three categories of models: those that explicitly exploit biases present in the few-shot prompt (e.g., GPT-OSS [ 2 ] ), those that implicitly rely on such biases (e.g., Qwen3-8B [ 35 ] ), and those that fail to reliably leverage few-shot prompt bias (e.g., Gemma-3-27b [ 31 ] ). For each category, we provide representative examples and report their performance across all benchmarks. Model categorization is determined using performance on NonsenseQA, shown in the last column, where the expected performance should be close to chance-level (25%).
Figure 7: Ablation study on the effect of the similarity model and the similarity function. For the ablation study, we compare standard cloze-style evaluation (”Cloze”) with the letter prediction and matching (”Letter (standard)”) to various full-sentence prediction outputs. We compare the original setting of the Qwen3-Embedding-0.6B [ 36 ] model tested under varying conditions (”Original” experiments) with different model types and sizes.
Figure 8: Ablation study on the effect of different labels used in the proposed matching protocol. For the ablation study, we compare standard letter prediction and matching (”Baseline”) to various label settings for the matching model. We compare using homogeneous per-option labels ( like dash (”Matched-dash”), a ” ” (”Matched-emoji”), and ”X” (”Matched-x”) ) to using heterogeneous labels (e.g, ( / / / / ) (”Different-emoji”), (A/B/C/D/E), and (1/2/3/4/5) ).
Figure 9: The performance of Standard-And-Letter (S&L) on our NonsenseQA dataset with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 10: The performance of Standard-And-Dash (S&D) on our NonsenseQA dataset with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 11: The performance of Matched-And-Letter (M&L) on our NonsenseQA dataset with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 12: The performance of Matched-And-Dash (M&D) on our NonsenseQA dataset with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 13: The performance comparison under all the possible evaluation protocols (S&L, S&D, M&L, and M&D) on our NonsenseQA dataset with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric. Each color is consistent with the previous Figures 9 , 10 , 11 and 12 .
S&L
M&D
Generation (s)
Extraction (s)
Total (s)
Extraction cost (%)
Generation (s)
Extraction (s)
Total (s)
Extraction cost (%)
Ratio
Gemma-3-12b-it
36.31(±1.47)
-
36.31(±1.47)
-
34.65(±1.42)
2.70(±0.04)
37.35(±1.43)
7%
1.03
Appendix
Table 2: The computational cost of answer generation and extraction for the Gemma-3-12b-it model on CommonsenseQA under the S&L and M&D evaluation protocols across all six answer permutations. For each setting, we report the average runtime in seconds, together with the corresponding standard deviation. The results show that the additional extraction step required by M&D accounts for only a small fraction of the total runtime, indicating that the proposed protocol introduces minimal computational overhead and remains practical for large-scale evaluation.
Frequency, %
Model Name
Extraction #1
Extraction #2
Extraction #3
Extraction #4
Failed
DeepSeek-R1-0528-Qwen3-8B
72.75
1.92
8.24
17.09
0.00
DeepSeek-R1-Distill-Qwen-7B
83.75
2.73
3.74
9.78
0.00
Llama-3.1-8B-Instruct
95.81
0.00
0.88
3.30
0.00
Ministral-3-8B-Base-2512
97.34
0.04
1.26
1.36
0.00
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
95.99
0.70
2.08
1.23
0.00
Appendix
Table 3: Extraction frequency of each proposed regular expression on MMLU-Pro [ 34 ] across models and all permutations evaluated. We observe that, for most models, approximately 98% of outputs adhere to the intended answer format and are successfully extracted by the primary matching rules, indicating that models generally follow the instructed output format reliably.
Parameter
Temperature
Top K
Top P
Min P
Value
0.6
20
0.95
0
Appendix
Table 4: Default set of parameters used when the generation configuration is not provided on Huggingface.
Figure 14: The proposed prompt modification with all changes in red. We find that by changing ”X”to”OPTION” and adding the additional constraint forces the model to switch from letter prediction to full-text option prediction without mentioning any letters in their answers.
Figure 15: An example few-shot prompt with a question from the ARC-Easy [ 8 ] dataset within our evaluation protocol. As with the MMLU-Pro [ 34 ] prompt, we add ”Let’s think step by step.” to induce Chain-of-thought reasoning.
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
72.80
76.00
4.27
15.74
66.20
76.00
9.80
0.39
28.82
30.00
1.57
9.01
25.80
33.20
7.40
0.59
0.57
-0.195
Deepseek-r1-distill-qwen
69.17
75.30
8.17
53.22
57.00
75.30
18.30
0.42
27.28
27.20
0.10
1.82
26.10
29.50
3.40
0.58
0.03
-0.154
Llama-3.1-8b-instruct
53.88
56.10
2.97
58.26
46.40
65.40
19.00
0.38
38.38
53.10
19.63
74.07
31.30
53.10
21.80
0.43
1.27
-0.052
Ministral-3-8b-base
40.27
42.90
3.50
49.58
30.10
49.50
19.40
0.34
30.40
40.40
13.33
36.80
24.70
40.40
15.70
0.44
0.74
-0.106
Nvidia-nemotron-3-nano
89.12
96.40
9.70
26.41
83.80
96.40
12.60
0.58
37.08
56.90
26.43
149.54
23.60
56.90
33.30
0.53
5.66
0.050
Appendix
Table 5: Detailed results on NonsenseQA .
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
64.18
42.92
25.51
94.43
42.92
70.84
27.92
0.46
52.32
52.66
0.41
0.48
51.19
53.40
2.21
0.46
0.01
0.002
Deepseek-r1-distill-qwen
61.08
53.15
9.52
33.58
53.15
68.88
15.73
0.51
52.50
50.29
2.65
1.28
50.29
53.97
3.68
0.54
0.04
-0.025
Llama-3.1-8b-instruct
76.93
75.59
1.61
3.01
74.77
80.02
5.25
0.78
75.54
75.51
0.03
5.07
71.99
79.69
7.70
0.78
1.69
-0.001
Ministral-3-8b-base
67.53
64.86
3.20
21.28
63.47
77.07
13.60
0.61
64.30
62.90
1.69
7.36
59.95
68.30
8.35
0.63
0.35
-0.018
Nvidia-nemotron-3-nano
84.77
63.88
25.06
98.58
63.88
94.35
30.47
0.77
73.68
72.65
1.24
0.79
72.65
75.18
2.53
0.75
0.01
0.015
Appendix
Table 6: Detailed results on CommonsenseQA [ 29 ] .
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
74.67
64.68
12.48
25.60
64.68
78.44
13.76
0.57
67.75
66.26
1.86
2.27
65.90
69.76
3.86
0.56
0.09
0.012
Deepseek-r1-distill-qwen
74.28
67.78
8.12
11.05
67.78
76.89
9.11
0.59
70.10
68.74
1.70
1.49
68.74
72.01
3.27
0.64
0.14
-0.051
Llama-3.1-8b-instruct
85.47
87.57
2.63
10.05
81.99
90.61
8.62
0.89
88.21
88.19
0.02
1.05
86.56
89.63
3.07
0.88
0.10
0.009
Ministral-3-8b-base
82.96
81.57
1.74
2.78
81.12
85.79
4.67
0.75
77.72
76.94
0.98
2.26
76.47
80.64
4.17
0.72
0.81
0.024
Nvidia-nemotron-3-nano
91.93
90.33
2.00
3.47
89.40
94.31
4.91
0.87
89.57
88.92
0.81
0.23
88.92
90.39
1.47
0.84
0.07
0.029
Appendix
Table 7: Detailed results on ARC [ 8 ] .
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
25.31
27.23
2.40
15.53
18.97
29.24
10.27
0.39
23.35
23.44
0.11
3.51
20.54
25.67
5.13
0.44
0.23
-0.043
Deepseek-r1-distill-qwen
31.56
33.93
2.96
16.11
24.55
36.61
12.06
0.43
28.03
28.57
0.67
2.97
25.22
30.36
5.14
0.45
0.18
-0.019
Llama-3.1-8b-instruct
27.81
29.46
2.06
12.79
24.33
33.93
9.60
0.42
30.09
30.58
0.61
8.25
27.68
35.49
7.81
0.45
0.65
-0.028
Ministral-3-8b-base
30.71
33.71
3.75
7.47
27.23
34.15
6.92
0.30
27.55
24.78
3.46
4.91
24.78
31.03
6.25
0.39
0.66
-0.095
Nvidia-nemotron-3-nano
36.65
39.51
3.57
18.45
30.58
42.86
12.28
0.41
34.82
37.05
2.79
1.35
33.93
37.05
3.12
0.46
0.07
-0.052
Appendix
Table 8: Detailed results on GPQA [ 23 ] .
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
31.57
35.30
4.11
4.07
27.66
35.30
7.64
0.33
31.29
36.12
5.31
3.85
28.95
36.12
7.17
0.29
0.94
0.047
Deepseek-r1-distill-qwen
31.56
31.33
0.25
33.83
21.53
39.74
18.21
0.34
27.71
28.47
0.83
1.39
26.10
29.72
3.62
0.30
0.04
0.036
Llama-3.1-8b-instruct
40.79
43.33
2.79
6.24
36.77
44.27
7.50
0.55
40.07
42.67
2.86
13.23
35.52
47.78
12.26
0.51
2.12
0.041
Ministral-3-8b-base
24.48
42.03
19.30
45.36
18.28
42.03
23.75
0.28
27.60
41.29
15.06
26.15
22.74
41.29
18.55
0.32
0.58
-0.043
Nvidia-nemotron-3-nano
55.31
61.61
6.93
9.62
49.20
61.61
12.41
0.54
47.77
53.31
6.10
5.96
44.92
53.31
8.39
0.45
0.62
0.094
Appendix
Table 9: Detailed results on MMLU-Pro [ 34 ] .
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
52.72
41.24
14.35
39.16
41.24
58.88
17.64
0.38
44.06
43.90
0.20
1.29
41.97
45.29
3.32
0.43
0.03
-0.052
Deepseek-r1-distill-qwen
39.61
29.77
12.30
27.08
29.77
45.35
15.58
0.38
31.32
29.35
2.46
3.19
28.93
32.91
3.98
0.41
0.12
-0.032
Llama-3.1-8b-instruct
60.14
59.24
1.12
0.25
59.24
60.81
1.57
0.67
56.86
57.49
0.79
0.89
55.07
57.73
2.66
0.64
3.48
0.030
Ministral-3-8b-base
56.65
54.83
2.28
4.35
54.35
59.96
5.61
0.54
52.63
51.99
0.80
2.33
51.39
55.62
4.23
0.55
0.53
-0.008
Nvidia-nemotron-3-nano
72.68
69.69
3.74
10.94
69.69
78.80
9.11
0.75
72.77
72.40
0.46
0.65
71.92
74.28
2.36
0.78
0.06
-0.032
Appendix
Table 10: Detailed results on the subset of INCLUDE [ 27 ] consisting of French, German, Italian and Spanish languages.
S&L
M&D
Model name
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
ACCMean
ACCORG
∣ΔACC∣
σ2
ACCmin
ACCmax
Δmax−min
SCORE
σRatio2
ΔSCORE
Deepseek-r1-0528-qwen3
44.72
44.96
0.29
64.14
36.12
61.10
24.98
0.32
54.18
54.55
0.45
1.60
52.99
56.27
3.28
0.47
0.02
-0.153
Deepseek-r1-distill-qwen
53.04
53.32
0.33
15.07
44.64
56.51
11.87
0.49
52.65
52.91
0.31
1.68
50.61
54.55
3.94
0.56
0.11
-0.068
Llama-3.1-8b-instruct
74.24
75.59
1.62
2.70
70.93
75.68
4.75
0.78
74.23
75.35
1.35
1.29
72.40
75.51
3.11
0.78
0.48
0.002
Ministral-3-8b-base
61.68
64.86
3.81
14.12
58.23
68.55
10.32
0.59
62.84
63.39
0.65
1.43
60.20
63.72
3.52
0.63
0.10
-0.042
Nvidia-nemotron-3-nano
74.11
74.04
0.08
4.34
70.76
77.89
7.13
0.76
73.76
72.89
1.05
0.61
72.73
74.77
2.04
0.77
0.14
-0.010
Appendix
Table 11: Detailed results on CommonsenseQA [ 29 ] without inducing malicious permutations on the few-shot prompt.
CommonsenseQA (prompt modified)
CommonsenseQA (prompt unmodified)
Model name
SCORES&L
SCOREM&D
ΔSCORE
SCORES&L
SCOREM&D
ΔSCORE
Deepseek-r1-0528-qwen3
0.458
0.455
0.002
0.317
0.471
-0.153
Deepseek-r1-distill-qwen
0.511
0.537
-0.025
0.489
0.557
-0.068
Llama-3.1-8b-instruct
0.783
0.785
-0.001
0.781
0.780
0.002
Ministral-3-8b-base
0.612
0.630
-0.018
0.586
0.628
-0.042
Nvidia-nemotron-3-nano
0.768
0.754
0.015
0.757
0.767
-0.010
Appendix
Table 12: SCORE comparison between results on CommonsenseQA [ 29 ] with and without few-shot distribution modification.
Figure 16: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on CommonsenseQA [ 29 ] with a 5-shot prompt that preserves the original few-shot answers distribution . The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 17: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on ARC [ 8 ] with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 18: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on GPQA [ 23 ] with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 19: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on MMLU-Pro [ 34 ] with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Figure 20: Comparison of the matched prediction with dashes as labels (M&D; our method) with standard letter prediction with letters as labels (S&L) on the subset of common languages on the INCLUDE benchmark [ 27 ] with a 5-shot prompt. The boxes illustrate the model performance under all possibilities of ”answer-moving attacks”, where the whiskers indicate the minimum and maximum accuracy for each model. Each dot represents the performance of the original permutations. Additionally, each star symbolizes a SCORE [ 19 ] robustness metric.
Multiple-choice question (MCQ) benchmarks are widely used to evaluate large language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A 2×2 ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.
LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.
Harshita Katoch, David Antony Selby, Gerrit Großmann +1
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany
Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity. I introduce inspect_permute, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals. I apply the tool across four vendors (gpt-4o-mini, claude-haiku-4-5, gemini-2.5-flash, grok-3) on five MMLU subjects, 24,000 API calls under temperature-0 generation, with falsifier predictions pre-registered via a public SHA-256 hash before half the data was observed. Position bias turns out to be statistically detectable only within a roughly 60-95% base-accuracy Goldilocks zone. Below it, processing-load dominance swamps subject-specific signal; above it, ceiling effects compress the variance below the chi-squared test resolution. Detectable cells separate into two mechanism types: monotone A-to-D decrease (processing_load, in low-tier models) and non-monotone D-drop (content_ambiguity, in a narrow capability band). Standard MMLU places every frontier-tier model above the detection band, so absence of signal there should be read as not measurable, not unbiased. Together with the ceiling-effect characterisation in arXiv:2606.26185, this work brackets the detectable region of position-bias measurement and makes the field central question askable in a verifiable form. Package, data, preregistration under MIT.