One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Hyunjong Ok; Minjae Park; Jaeho Lee

cs.CLOct 14, 2026

One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Authors: Hyunjong Ok, Minjae Park, Jaeho Lee

Organizations: Pohang University of Science and Technology (POSTECH)

Abstract

Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in answers across option orderings). We connect these perspectives through the pooled selection-rate standard deviation (pooled SD), which measures how unevenly option letters are selected across questions and orderings. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two label-free factors: the flip rate and the positional skew. The flip rate is the fraction of questions whose chosen option changes across orderings; the positional skew is the standard deviation of option-letter selection rates among those questions. Across 41 checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while the positional skew tracks the estimated option-symbol prior. Prior correction for selection bias mainly reduces the positional skew. Aggregating each ordering with its reversal, a mitigation for order sensitivity, mainly reduces the flip rate. Both reduce the range of accuracy across correct-answer letters by similar amounts, but aggregation leaves fewer model-rank reversals. Applying prior correction before aggregation reduces both factors and further narrows this accuracy range. However, aggregation lowers the flip rate partly by settling questions on the same wrong option. We therefore recommend reporting both factors to distinguish mitigation effects, alongside accuracy to distinguish stability from correctness.

Figures & tables

Panel of Figure 1 — Figure 1: The same pooled SD can come from different behaviours, and mitigation methods lower it through different factors. (a) An answer that stays on one option lands once on every letter; one that stays on letter A changes option each time, a flip. (b) Rows are questions; bars give each letter’s share of flipped and of all answers from the dashed $1/4$ . Two constructed models reach the same pooled SD: Model X flips often with a mild lean to A , Model Y rarely but strongly. (c) Each method’s measured mean direction; the dashed curve joins equal pooled SDs.

Figure 2: An item that does not flip lands once on every position, so only flipping items can make the pooled shares uneven. Left: rotation moves every content through every position. Middle: a stable-wrong item’s answers spread evenly; a flipping item’s answers pile onto early letters. Right: pooling weights the even share by

1-\mathrm{FL}

and the flipping items’ share

g

\mathrm{FL}

. Stars mark the gold option, dots the choice and dashes

1/K

; the histograms are illustrative.

Figure 3: The pooled SD is the flip rate times the skew; the flip rate tracks accuracy, the skew only weakly. Points are cells, coloured by option count

K

;

r_{s}

is Spearman’s correlation with accuracy. Cells A (Qwen2.5-72B-Instruct, PIQA) and B (Llama-3.2-3B, LogiQA) reach nearly the same pooled SD from opposite factor combinations.

Statistic	Value
Order sensitivity
Flip rate	$\mathrm{FL}$
Two-order disagreement	${\leq}\,\mathrm{FL}$
Selection bias
Pooled SD	$\mathrm{FL}\cdot\operatorname{std}(g)$
Balanced RStd	$\mathrm{FL}\cdot\operatorname{std}(h)$

Table 1: Consistency statistics count changing answers; the pooled SD and balanced RStd factor through the flip rate, and the forward KL is bounded through it. Values under balanced orderings (

\leq

: bound for two evaluated orders);

g

is where flipping answers land and

h_{j}

their accuracy with the correct option at

j

. Only the pooled SD splits into two label-free factors.

Panel of Figure 6 — Figure 6: The reversal product removes an added letter preference only where mirrored letters differ, while PriDe largely removes each tested shape. Change in placement range after adding $\alpha d_{i}$ to the logit of letter $i$ in every stored query of the MMLU cells, where $d$ is the panel’s shape and a negative $\alpha$ reverses it; priors are re-estimated after the addition, and shading gives 95% intervals. Under mirroring ( A with D , B with C ) the first two shapes cancel, the third passes through and the fourth has both parts.

Method	$\Delta\log\mathrm{FL}$	$\Delta\log\mathrm{Skew}$	Placement range (pp)	Rank reversals (% of pairs)	$\Delta$ Accuracy (pp)	$\Delta$ Stable-wrong (pp)
Baseline	0.00	0.00	16.7	15.4	0.0	0.0
PriDe ( Zheng et al., 2024 )	$-0.00$	$-0.38$	9.6	15.4	0.5	0.3
Batch calibration, variant ( Zhou et al., 2024a )	$-0.03$	$-0.50$	8.2	15.4	0.5	0.5
Reversal product ( Wei et al., 2024 )	$-0.54$	$-0.28$	8.2	10.6	2.8	6.2
PriDe then reversal	$-0.54$	$-0.47$	5.3	10.6	2.9	6.6
Batch then reversal	$-0.56$	$-0.62$	4.6	10.5	2.9	6.8

Table 2: PriDe and the reversal product narrow placement range by similar amounts, but the product leaves fewer flips and rank reversals; a correction followed by the product lowers both factors and narrows the placement range most. Cells with at least three options, on shared held-out items. Top: mean log-factor changes, placement range and rank reversals as levels, and changes in accuracy and stable-wrong share. Bottom: cellwise paired differences, the reversal product minus each correction and each composition minus each of its components.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Params (B)	Model	Params (B)
Llama-3.x		Qwen3
Llama-3.2-1B	1	Qwen3-0.6B	0.6
Llama-3.2-1B-Instruct	1	Qwen3-1.7B	1.7
Llama-3.2-3B	3	Qwen3-4B	4
Llama-3.2-3B-Instruct	3	Qwen3-8B	8
Llama-3.1-8B	8	Gemma-2

Appendix

Table 3: The checkpoints span seven families. Parameter counts are the released sizes. The Qwen3 releases carry no -Base suffix and are post-trained hybrid-thinking checkpoints.

Benchmark	$K$	Items	Models
PIQA	2	1,838	41
WinoGrande	2	1,267	41
ARC-Challenge	4	1,165	41
ARC-Easy	4	2,365	41
BBH-MC	4	732	36
HellaSwag	4	10,042	41

Appendix

Table 4: Every statistic is computed on the items of these benchmarks that are usable under all

K

orderings. “Items” is the modal usable item count under the first-token sweep; “Models” counts checkpoints with a first-token cell. MMLU is a five-subject subset.

Canonical-order AUC, flipping against stable items
Quantity	Estimate [95% CI]
Top-two gap, all cells	$0.848\,[0.828,0.863]$
Top-two gap, least accurate third	$0.781\,[0.755,0.806]$
Top-two gap, middle accurate third	$0.832\,[0.813,0.853]$
Top-two gap, most accurate third	$0.908\,[0.898,0.925]$
Entropy, all cells	$0.848\,[0.825,0.862]$

Appendix

Table 5: The canonical top-two gap separates flipping items in every accuracy third, and flipping answers point the way the estimated prior does. Median AUC (a smaller gap or larger entropy scores as flipping), the partial Spearman between the skew and the standard deviation of the estimated prior after adjusting both for accuracy rank, and the median cosine between

g

and the prior, each minus uniform; the prior is estimated on the held-out 5% of each cell’s items. Each AUC summarises only cells containing both compared item states. For the letter-permuted prior each cell takes the median cosine over its non-identity letter permutations, which places it near zero.

Strength $\alpha=0.5$
Quantity	Estimate [95% CI]
$\Delta\log$ skew, estimated	$-0.316\,[-0.397,-0.238]$
$\Delta\log$ skew, permuted	$0.132\,[0.103,0.165]$
$\Delta\log$ skew, difference	$-0.449\,[-0.528,-0.373]$
$\Delta\log$ flip rate, estimated	$-0.025\,[-0.032,-0.018]$
$\Delta\log$ flip rate, permuted	$0.027\,[0.021,0.033]$

Appendix

Table 6: Dividing out the estimated prior lowers the skew and raises accuracy; the same prior with permuted letters does the opposite. Mean change on the items not used to estimate the prior, after dividing by the prior raised to

\alpha

; permuted rows average over letter permutations, and differences are paired within each draw of the 95% checkpoint-block intervals. For each factor and strength, log-change rows use the common cells where that factor is positive before correction and after correction by the estimated prior and every included letter permutation. The flip rate moves little either way.

Figure 7: Items flip where their two leading options are close. Share of stable-correct, flipping and stable-wrong items by the top-two letter-probability gap under the canonical order on all first-token cells, cells weighted equally.

Flip outcome	Median AUC [95% CI]
All $K$ answers	$0.865\,[0.844,0.883]$
Canonical answer left out	$0.819\,[0.790,0.838]$
Left out; canonical answer correct	$0.837\,[0.814,0.858]$
Left out; canonical answer wrong	$0.633\,[0.617,0.645]$
All $K$ answers; items wrong under every ordering	$0.760\,[0.750,0.771]$
Left out; stratified by own and peer accuracy	$0.752\,[0.738,0.772]$

Appendix

Table 7: The canonical top-two gap predicts flips on the other orderings and among items wrong everywhere. Median AUC over first-token cells with at least three options and both outcome classes for the stated comparison. Left out counts an item as flipping only when its other

K-1

answers disagree, so the answer the gap is read from is not in the outcome. The last row compares items only within strata of their own canonical correctness and of five bins of the other checkpoints’ accuracy on the item, keeping items seen by at least three of them.

	All cells	$K{=}2$	$K{=}4$	$K{=}10$
Share of the variance of the log pooled selection-rate SD
Flip rate, $\log\mathrm{FL}$	$0.41$ $[0.35,0.48]$	$0.22$ $[0.14,0.31]$	$0.45$ $[0.39,0.52]$	$0.07$ $[0.04,0.14]$
Skew factor, $\log\operatorname{std}(g)$	$0.35$ $[0.30,0.40]$	$0.62$ $[0.48,0.79]$	$0.23$ $[0.20,0.28]$	$0.64$ $[0.53,0.74]$
Twice their covariance	$0.24$ $[0.18,0.29]$	$0.16$ $[0.02,0.28]$	$0.31$ $[0.26,0.36]$	$0.29$ $[0.19,0.36]$

Appendix

Table 8: The flip rate and the skew carry comparable shares of the variance of the log pooled SD. Top: over the first-token cells, shares carried by the log flip rate, the log skew and twice their covariance, which sum to one. Bottom: the skew’s share and its Spearman correlation with accuracy for each readout, on its own pool and on the cells run under the first-token, chain-of-thought and direct-answer readouts, with checkpoint-block intervals. The five-shot row uses the subset of the common pool with a five-shot readout. The skew is negatively associated with accuracy in every readout and pool.

	Spearman with accuracy	Split-half Spearman	Spearman–Brown
Flip rate $\mathrm{FL}$	$-0.947$ $[-0.957,-0.934]$	$0.993$	$0.996$
Positional skew, $\operatorname{std}(g)$	$-0.333$ $[-0.403,-0.254]$	$0.913$	$0.954$
Positional skew, median within a benchmark	$-0.389$ $[-0.526,-0.281]$	$0.947$	$0.973$

Appendix

Table 9: The flip rate and the skew are reliable across random item halves. Over the first-token cells: Spearman correlations with accuracy, with 95% cell-resampling intervals, and split-half correlations, uncorrected and Spearman–Brown corrected; the last row gives the median over benchmarks of each within-benchmark value, its interval resampling cells within each benchmark.

Comparison	Cells	Estimate	95% interval
Prior and skew, Spearman	All	$0.745$	$[0.680,0.799]$
Prior and skew, given accuracy	All	$0.728$	$[0.664,0.782]$
Prior and skew, given accuracy	$K=2$	$0.763$	$[0.632,0.848]$
Prior and skew, given accuracy	$K=4$	$0.677$	$[0.601,0.742]$
Prior and skew, given accuracy	$K=10$	$0.623$	$[0.092,0.838]$

Appendix

Table 10: The mean option-symbol profile tracks the skew on disjoint items. Spearman correlations across first-token cells between the profile’s nonuniformity and the skew; given accuracy rows are partial rank correlations, overall and by option count.

Figure 8: The flip rate’s association with accuracy exceeds the cap null where the cap does not bind. First-token cells by option count and cap stratum: dots are observed Spearman correlations with accuracy, grey bars the central 95% of a null that permutes the flip rate divided by its cap

\min(1,K(1-A))

, and ticks its median.

Cap null	Observed $r_{s}$	Null median	Null 95% range	$P$
$K{=}2$ , all	$-0.743$	$-0.764$	$[-0.830,-0.685]$	$0.71$
binding	$-0.744$	$-0.760$	$[-0.830,-0.682]$	$0.67$
cap equal to one	$1.000$	$-1.000$	$[-1.000,1.000]$	$1.00$
$K{=}4$ , all	$-0.968$	$-0.623$	$[-0.675,-0.569]$	$<0.001$
binding	$-0.943$	$-0.951$	$[-0.960,-0.941]$	$0.95$
cap equal to one	$-0.881$	$0.000$	$[-0.138,0.135]$	$<0.001$

Appendix

Table 11: The cap null is compatible with the flip rate’s association with accuracy where the cap binds, but not on four-option cells where the cap equals one. The scaled flip rate is

\mathrm{FL}/\min(1,K(1-A))

, and the null permutes it within each option count, and also within each cap subset for the binding and cap-one rows; the cap binds where

\min(1,K(1-A))<1

, and the two-option row where it equals one holds only two cells, where any Spearman correlation is

\pm 1

. Null ranges are 2.5th–97.5th percentiles, and

P

is the share of permutations, with a plus-one correction, whose

|r_{s}|

reaches the observed one; lower-block intervals resample checkpoints (1.25th–98.75th).

Statistic	Item-disjoint	Benchmark-disjoint	Ordering-disjoint
Flip rate $\mathrm{FL}$	$-0.941$	$-0.892$	$-0.980$
	$[-0.949,-0.930]$	$[-0.931,-0.811]$	$[-0.993,-0.935]$
Per-item KL to uniform	$-0.927$	$-0.865$	$-0.977$
	$[-0.940,-0.909]$	$[-0.917,-0.764]$	$[-0.992,-0.931]$
Per-item selection-rate SD	$-0.855$	$-0.873$	$-0.977$
	$[-0.879,-0.816]$	$[-0.921,-0.777]$	$[-0.992,-0.931]$

Appendix

Table 12: The flip rate stays negatively associated with accuracy measured on disjoint items or orderings. Signed Spearman correlations and magnitude gaps (per-item minus pooled; flip rate minus pooled for benchmark-disjoint), on first-token cells scored on held-out item folds, six benchmarks each scored against the other five, and all-ordering cells scored on held-out orderings; benchmark-disjoint estimates average within-benchmark correlations, and gap intervals are simultaneous over each column’s two gaps. The other count-based statistics are the fluctuation rate, the mean pairwise disagreement between orderings and the mean entropy of each item’s chosen content, given as the range of their correlations. In the all-ordering cells the per-item gaps are smaller and their intervals reach zero; scoring those cells on their own cyclic orderings instead leaves the gaps unchanged.

Model	Accuracy	$\mathrm{FL}$	$\operatorname{std}(g)$	Pooled SD	Repeat disagreement	Reorder disagreement
gpt-6.1-sol	$0.890$	$0.036$	$0.019$	$0.0007$	$0.017$	$0.016$
gpt-6-luna	$0.736$	$0.342$	$0.011$	$0.0038$	$0.168$	$0.210$
gpt-5.6-terra	$0.867$	$0.116$	$0.026$	$0.0030$	$0.062$	$0.067$

Appendix

Table 13: Reordering changes a closed model’s answer about as often as asking the same ordering again. MMLU, four cyclic orderings, one-letter reply. Both disagreement columns compare two calls on each item: repeat disagreement pairs ordering 0 with a second call on it, and reorder disagreement pairs ordering 0 with each other ordering and averages the three. Only for gpt-6-luna does reordering change answers more often than repeating, by an amount whose item-bootstrap interval excludes zero.

Operation	Stop flipping (%)	Start flipping (%)	Resolved flips settling wrong (%)	Former flips on favoured letter (%)
PriDe	$3.5\,[2.9,4.0]$	$2.7\,[2.3,3.1]$	$38.1\,[35.1,40.9]$	$22.8\,[21.6,23.8]$
Batch	$2.6\,[2.2,3.1]$	$1.3\,[1.1,1.6]$	$39.1\,[36.3,41.9]$	$27.5\,[26.5,28.5]$
PriDe then batch	$3.7\,[3.1,4.2]$	$2.5\,[2.3,2.8]$	$39.4\,[36.8,42.0]$	$22.8\,[21.8,23.6]$
Reversal product	$15.9\,[14.7,17.0]$	$0.9\,[0.8,1.0]$	$38.8\,[36.3,41.3]$	$26.4\,[25.5,27.3]$
PriDe then reversal	$16.7\,[15.3,18.1]$	$1.0\,[0.9,1.2]$	$39.0\,[36.5,41.5]$	$22.5\,[21.9,23.0]$
Batch then reversal	$16.9\,[15.6,18.3]$	$0.9\,[0.8,1.0]$	$39.2\,[36.6,41.7]$	$24.4\,[23.9,24.9]$

Appendix

Table 14: Corrections move flips both ways, while the reversal product mostly resolves them, about two in five onto a wrong option. Shares of evaluated items that stop or start flipping, the share of resolved flips that become stable-wrong, and the share of previously flipping items’ answers on the PriDe prior’s top letter (

34.9\%

before any operation), on the cells with at least three options.

	Pooled SD opposite (%)	Per-item KL opposite (%)	Opposite if resolved (%)
Post-training pairs	30.1	6.8	15.9
Checkpoint pairs	17.3	4.6	9.6
Scale trends	5.8	0.0	–
PriDe cells	20.0	17.7	10.5

Appendix

Table 16: A comparison read off the pooled SD can point opposite to the flip rate, most often between a base checkpoint and its post-trained release. Share of comparisons across first-token cells in which the pooled SD, or the per-item KL, moves opposite to the flip rate; scale trends compare rank correlations with parameter count within a family, training status and benchmark. The last column keeps comparisons with both changes resolved.

Number of orderings: cyclic against all $K!$
Condition	Measurement
$\mathrm{FL}_{\mathrm{cyc}}\leq\mathrm{FL}_{\mathrm{full}}$ holds	$100\%$ of cells
Relative undercount, mean over cells	$25.7\%$
$r_{s}(\mathrm{acc},\mathrm{FL})$ , cyclic vs. all $K!$	$-0.980$ vs. $-0.974$
Which orderings: mean flip rate, and $r_{s}$ of the per-cell flip rate with the cyclic one
Cyclic, $K$ orderings	$0.377$

Appendix

Table 17: The cyclic protocol undercounts flips in every cell but barely changes the ranking of cells. Cells run under all 24 orderings of four options; random subsets average draws of four orderings, and the exact expectation averages over every four-ordering subset.

	Flip rate (%)		Placement range (pp)		Cyclic-stable items flipping elsewhere (%)
	Cyclic set	Other sets	Cyclic set	Other sets
All cells, uncorrected answers
Level	$37.7$	$38.2$ ( $37.5$ , $38.7$ )	$14.8$	$14.8$ ( $14.6$ , $15.2$ )	$10.5\,[6.9,14.7]$
Cells with unrounded option probabilities
Uncorrected answers, level	$45.2$	$45.4$ ( $44.7$ , $45.9$ )	$26.1$	$26.3$ ( $25.8$ , $26.6$ )	$14.6$
PriDe, change	$-0.1$	$0.1$ ( $-0.3$ , $0.4$ )	$-18.3$	$-16.6$ ( $-17.5$ , $-15.7$ )	$15.8$

Appendix

Table 18: The cyclic set is a typical balanced set of four orderings, and on the cells with unrounded probabilities each operation’s effect is consistent across them. Each set shows every content once at every position and holds no ordering with its reversal. Other-set columns give the mean and range over the other such sets; the last column is the share of cyclic-stable items that flip on another set, averaged over the other sets. The top row covers all cells run under all orderings, with a 95% checkpoint-block interval for that share; the lower rows cover the all-ordering cells with unrounded option probabilities, using only items with unambiguous answers for each operation at the probability precision used and comparing each operation with the uncorrected answers on those same items.

		Pooling identity		Spearman with accuracy
Ordering design	Position balanced	Relative residual, median	Relative residual, largest	Flip rate	Positional skew	Pooled SD	Per-item minus pooled SD magnitude
Cyclic	yes	$4\times 10^{-16}$	$6\times 10^{-15}$	$-0.980$ $[-0.99,-0.93]$	$-0.580$ $[-0.84,-0.36]$	$-0.935$ $[-0.98,-0.86]$	$0.042$ $[0.00,0.08]$
All 24 orderings	yes	$4\times 10^{-15}$	$6\times 10^{-14}$	$-0.974$ $[-0.99,-0.92]$	$-0.621$ $[-0.86,-0.38]$	$-0.932$ $[-0.99,-0.86]$	$0.043$ $[-0.01,0.08]$
Latin square	yes	$5\times 10^{-16}$	$3\times 10^{-15}$	$-0.979$	$-0.584$	$-0.924$	$0.052$
Gold at each position	gold only	$0.070$	$0.186$	$-0.977$	$-0.577$	$-0.922$	$0.064$
Random set per item	no	$0.157$	$0.674$	$-0.978$	$-0.579$	$-0.885$	$0.072$

Appendix

Table 19: The pooling identity is exact on every position-balanced ordering design and departs from it on unbalanced ones. Rows are ways of choosing each item’s orderings in the cells run under all 24 orderings; the gold row puts the correct option once at each position. The relative residual is

|\operatorname{std}(f)-\mathrm{FL}\cdot\operatorname{std}(g)|/\operatorname{std}(f)

; intervals resample checkpoints, and Latin-square and random rows are medians over sets and draws. The two residual columns take each cell’s median and give its median and maximum over cells. The last column is the magnitude of the per-item SD’s Spearman correlation with accuracy minus that of the pooled SD.

Design	Largest corrected absolute residual	Median residual / bound	Pooled SD Spearman	Corrected SD Spearman
Cyclic	$5.55\times 10^{-17}$	–	$-0.935$	$-0.935$
Random set of 4 orderings shared by all items	$1.39\times 10^{-16}$	$0.611$	$-0.872$ $[-0.922,-0.791]$	$-0.924$ $[-0.947,-0.884]$
Random 4 orderings per item	$1.11\times 10^{-16}$	$0.734$	$-0.885$ $[-0.926,-0.816]$	$-0.922$ $[-0.940,-0.889]$

Appendix

Table 20: Removing the design term restores the identity and, in the median draw, strengthens the accuracy association under random ordering subsets. The corrected residual is

|\operatorname{std}(f-b_{S})-\mathrm{FL}\operatorname{std}(g)|

, the largest over cells and draws; the bound

\operatorname{std}(b_{S})

is zero for the cyclic design, and the ratio column divides the uncorrected residual by it. Spearman correlations are with same-design accuracy; random designs show the median and 2.5–97.5% range over draws.

Statistic	Definition	Related measure
Per-item consistency
Flip rate $\mathrm{FL}$	$\frac{1}{N}\bigl\|\{q:z_{q}\text{ non-constant on }S_{\mathrm{cyc}}\}\bigr\|$	Analogue of $1-{}$ ReCon ( Cocchieri et al., 2026 )
Fluctuation rate	$\frac{1}{N}\sum_{q}\mathbf{1}\bigl[z_{q}(\pi_{0})\neq z_{q}(\pi_{\lfloor K/2\rfloor})\bigr]$	adapted ( Wei et al., 2024 )
Pooled position statistics
$D_{\mathrm{KL}}$ to uniform	$\sum_{r}f(r)\log_{2}\bigl(Kf(r)\bigr)$	CKLD ( Choi et al., 2025 )
Selection-rate SD	$\bigl(\frac{1}{K}\sum_{r}(f(r)-1/K)^{2}\bigr)^{1/2}$	RStd ( Zheng et al., 2024 )

Appendix

Table 21: The flip rate and the fluctuation rate count content changes; position statistics measure positional imbalance, pooled or within each item. All use the cyclic predictions:

z_{q}

is the chosen content,

f(r)

the share of all answers at position

r

and

f_{q}(r)

the share of item

q

’s answers there, and the fluctuation rate compares two cyclic orderings. The per-item statistics apply the pooled ones to each item’s own answers and average over items. Larger values mean more disagreement or more imbalance.

Input Order Sensitivity in Language Models

See all 14 papers

Aug 12, 2026

A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation

Multiple-choice question (MCQ) benchmarks are widely used to evaluate large language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A

2\times2

ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.

Karl Hanna, Chen Feng

Queen’s University Belfast

LLM Evaluation Multiple-Choice Question Answering Language Model Bias Evaluation Input Order Sensitivity in Language Models

Ask PDF

Apr 16, 2026

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options

Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by shortcut strategies that obscure true competence. Therefore, we propose a massive option evaluation protocol that scales the candidate set to one hundred options and sharply reduces the impact of chance performance. We apply this framework to a Korean orthography error detection task where models must pick the single incorrect sentence from a large candidate set. With fixed targets and repeated resampling and shuffling, we obtain stable estimates while separating content driven failures from positional artifacts. Across experiments, results indicate that strong performance in low option settings can overstate model competence. This apparent advantage often weakens under dense interference at high

N

, revealing gaps that conventional benchmarks tend to obscure. We identify two failure modes, semantic confusion and position bias toward early options under uncertainty. To isolate the effect of context length, we run padding controlled and length matched tests, which suggest that the main bottleneck is candidate ranking rather than context length. Together, these findings support massive option evaluation as a general framework for stress testing model reliability under extreme distractor density, beyond what low option benchmarks can reveal.

Nahyun Lee, Guijin Son

Chung-Ang University · Seoul National University

LLM Evaluation Multiple-Choice Question Answering LLM Reliability Language Model Robustness

Ask PDF

Oct 13, 2026

Beyond Bias Scores: Answerability Confounds in Activation Steering with Gradient-Constrained Direction Design (GCD)

Bias scores in language-model evaluations often conflate what a model prefers with whether it answers at all. We identify this confound in BBQ, whose ambiguous score factorizes into answerability and signed preference. Across 11 categories, 28,}824 items, and five models, contrastive steering lowers the signed ambiguous score in 11/11 categories but lowers the signed disambiguated preference score in only 8/11. Across 99 configurations, the answerability channel exceeds half of the observed score change in 17% and accounts for the entire change in 11%. In a six-category disambiguated evaluation, 20.8% of 1,500 items flip to the unknown option, where abstention is incorrect; StereoSet shows no analogous answer-rate shift. The effect reflects selection and near-orthogonality between the contrastive and preference-gradient directions. We introduce Gradient-Constrained Direction Design (GCD), an option-gradient construction with a first-order unknown-logit constraint. Under a matched Qwen-7B protocol, GCD reduces answer-rate distortion 14-fold and yields a 4-fold larger preference change, while ablations show similar gains for alternative and unconstrained option-gradient directions; nulling the unknown gradient alone is not responsible for the gain. We recommend reporting answerability alongside published scores.

Weitong Li, Donglin Song, Junkai Lin +1

Beijing Normal-Hong Kong Baptist University · The Chinese University of Hong Kong · University of Eastern Finland +1

Activation Steering Language Model Bias Evaluation LLM Evaluation

Ask PDF

Abstract

Input Order Sensitivity in Language Models

Explore similar work

A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options

Beyond Bias Scores: Answerability Confounds in Activation Steering with Gradient-Constrained Direction Design (GCD)