cs.CLAug 9, 2026

Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation

Authors: Yu Wang, Zhe Zhou, Menglin Liu, Ge Shi

Abstract

Large language models are increasingly used for ordinal classification, yet semantically equivalent changes to prompt organization can alter their predictions. We conduct systematic experiments to characterize positional bias from label order, demonstration order, and demonstration placement. First, we apply the three probes to ten frontier LLMs on a common ordinal-classification task; every model is sensitive to all three positional sources, showing that the problem is pervasive. Second, we vary eight prompt-, task-, and model-level factors across five datasets; accuracy and stability are often misaligned, and only lower scale cardinality consistently improves both. Third, we compare pointwise, pairwise, and listwise inference, alternative aggregation and debiasing methods, and joint configurations; the tested corrections do not provide a reliable remedy, while a comparison-based listwise formulation offers the best balance but transfers unevenly across models and bias sources. These findings show that positional robustness depends on the full system configuration rather than the model alone. Ordinal-classification systems should therefore be selected jointly for predictive performance and stability.

Explore similar work

CardsList
Aug 12, 2026cs.CL

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.
Karl Hanna, Chen Feng
Order MattersOrderClass ImbalanceInfluence
AskPDF
Oct 1, 2025cs.SD

Hearing the Order: Investigating Position Bias in Large Audio-Language Models

Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a form of position bias and undermine their reliability. In this paper, we identify and analyze this problem in LALMs. We demonstrate that no model is immune to this bias through extensive experiments on six LALMs across three widely used benchmarks and their spoken counterparts. Shuffling the order of answer options can cause performance fluctuations of up to 24% and even change model rankings, raising concerns about the reliability of current evaluation practices. We also study permutation-based strategies and show that they can mitigate bias in most cases. Our work represents the first systematic investigation of this issue in LALMs, and we hope it raises awareness and motivates further research in this direction.
Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei +3
Large Audio Language ModelsBias Detection and MitigationOrderRanking
AskPDF
Jul 26, 2026cs.IR

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

Large language models (LLMs) used as listwise rerankers in recommendation systems suffer from position bias when serializing candidate sets into prompts. We show this order sensitivity creates an exploitable attack surface: an attacker can promote a label-0 target into the top-kk solely by reordering candidates, without changing item content, labels, or model parameters. We introduce promo@k\mathrm{promo}@k to quantify this vulnerability, measuring the fraction of label-0 targets that can be elevated into top-kk rankings via permutation. Evaluating across three domains (MovieLens, Amazon Books, and Amazon Fashion), promo@5\mathrm{promo}@5 reaches up to 0.57 at an attack budget of RR = 50 orderings. Furthermore, ordinary permutation stability predicts vulnerability without running the attack. While a bidirectional T5 encoder scorer reduces exposure, permutation-consistency regularization and architectural invariance effectively mitigate it. Pointwise scoring avoids the bias issue but degrades ranking quality. These results demonstrate that input candidate order in listwise LLM reranking is a security-relevant attack vector. Code and data are available at https://github.com/geoz-lab/position_bias_attack.
Ge Zhang, Jingru Cheng, Huiyuan Chen
Language Model Evasion AttacksOrder MattersAttack SurfaceAdversarial Attack StrategiesPosition
AskPDF