Users increasingly rely on large language models (LLMs) to analyse collections of independent inputs, such as product reviews or social media posts, and return a single aggregated answer. This setting requires multi-instance processing (MIP), where a model combines information from multiple instances into one prediction. We study a task in which the model must infer a binary label for each instance, and identify the most frequent label among the full instance set. The correct answer should be invariant to instance order. By evaluating fifteen LLMs across five datasets and seven instance order patterns, with each instance set held fixed across patterns, we find substantial order sensitivity. Accuracy can differ by about 90 percentage points between the best and worst order patterns, with models generally more accurate when majority-label instances appear at the start than when they appear in the middle or at the end. For subsets of models, available reasoning traces and gradient-based attribution reveal uneven emphasis across positions. Traces disproportionately mention early instances, while attribution scores are generally higher near the beginning and end than in the middle. These results show that instance order can systematically bias aggregated predictions, highlighting the need to evaluate MIP across different input arrangements.
Figures & tables
Figure 1: An illustrative example of our MIP setting, with 20 textual instances and two pre-defined labels, arranged in four order patterns. In this example, the model correctly predicts the majority label when instances are randomly shuffled or majority-label instances are grouped at the beginning, but not when they are grouped in the middle or at the end. Note that while all instances are color-coded for informative purposes, LLMs receive instances without label information.
Name
Target
Label A : Label B
Math
Answer
odd_number : even_number
NER
Entity
PER : LOC
Occurrence
“ the ”
exactly_once : more_than_once
Sentiment
Polarity
positive : negative
WSD
“ java ”
programming_language : place
Table 1: Overview of our binary-labelled tasks. Each instance’s target is assigned either Label A or Label B, and the MIP objective is to identify the majority label across all instances.
Figure 2: Per-model accuracy on four representative order patterns in the core setting. Error bars indicate standard deviations over three instance-selection seeds.
Model
Acc
Δbest−worst
Invalid
DeepSeek-V4-Flash
79.8 ± 6.9
64.0 ± 16.0
40.0 ± 5.8
DeepSeek-V4-Pro
78.6 ± 25.8
50.0 †
88.9 ± 4.4
Gemini 3.1 Flash-Lite
63.0 ± 4.1
67.4 ± 12.2
10.7 ± 3.4
Gemini 3.1 Pro
81.7 ± 4.2
40.0 ± 0.0
1.1 ± 1.1
Gemma 4 26B A4B
64.0 ± 6.9
91.1 ± 7.8
28.1 ± 2.3
Gemma 4 31B
68.2 ± 0.3
51.3 ± 12.0
10.4 ± 6.3
Table 2: Accuracy (Acc, %), best-worst order-pattern gap ( Δ , pp), and invalid rate (%) in the core setting. Values are means ± standard deviations over three instance-selection seeds unless marked. Gaps marked † use one paired seed (no standard deviation), and ‡ use two.
Figure 3: Accuracy over different instance counts and label distributions , for the four representative patterns. Standard deviation is computed over three order seeds for random only.
Figure 4: Accuracy (top) and order-pattern gap Δ (bottom) under the standard prompt and three prompting variants for three evaluated models. Error bars indicate standard deviations over three instance-selection seeds.
Model
<25%
<75%
<100%
=100%
DeepSeek-V4-Flash (valid rate: 60.0%)
#
132
27
1
2
Acc.
78.0
85.2
100.0
100.0
gpt-oss-120b (valid rate: 78.1%)
#
201
7
2
1
Acc.
63.2
57.1
50.0
100.0
Qwen3.6-35B-A3B (valid rate: 66.3%)
#
132
28
1
18
Acc.
71.2
82.1
0.0
100.0
Table 3: Number of valid experiments (#, out of 270 per model) and the corresponding accuracy (Acc., %) across estimated instance-coverage bins in the 2,000-instance core setting across all seven order patterns. For example, the <75% bin denotes the runs whose traces explicitly specify [25%,75%) of the 2,000 instance IDs.
Figure 5: Per-ID mention rate across valid core -setting experiments . Each bin represents one instance ID, with red markers highlighting IDs that are multiples of 50.
Figure 6: IxG position profiles pooled over valid replays with available attribution from NER , Sentiment , and the six structured order patterns. Each point averages ten consecutive instances. The dotted line at 1 denotes the uniform reference, and the y -axis is logarithmic. For reference, the inset reports accuracy (%) for each model under three representative patterns.
Figure 7: IxG position profiles for Qwen3-4B-Instruct , separated by NER (top) and Sentiment (bottom), for three structured order patterns. Dotted horizontal lines denote the uniform reference at 1 , and shaded backgrounds mark majority-label blocks. Each point averages ten consecutive instances, and the y -axis is logarithmic. Panel labels report accuracy (%) over the same outputs used for attribution.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Both Correct (%)
Qwen3.6-35B-A3B Only (%)
Gemini 3.1 Flash-Lite Only (%)
Both Incorrect (%)
Math
95.52
1.80
0.80
1.88
NER
90.44
3.50
1.26
4.80
Occurrence
99.16
0.70
0.08
0.06
Sentiment
92.10
1.43
2.52
3.95
WSD
99.60
0.06
0.04
0.30
Appendix
Table 4: Agreement between the two filtering LLMs across tasks. “Correct” and “Incorrect” denote the proportion of instances correctly and incorrectly classified by both Qwen3.6-35B-A3B and Gemini 3.1 Flash-Lite .
Figure 8: A toy example with 20 instances illustrating how the order patterns arrange instances with majority (Label A) and minority (Label B) labels.
Figure 9: Outcomes for Qwen3.6-35B-A3B under 20K and 100K output-token budgets across all seven order patterns in the core setting. Bars show the percentages of all experiments with correct, incorrect or invalid outputs. Values above bars give mean accuracy across three instance-selection seeds.
Figure 10: Accuracy (%) of Qwen3.6-35B-A3B under 20K and 100K output-token budgets across instance counts and label distributions. Results pool all five tasks, both majority labels and all seven order patterns.
Figure 11: Invalid rates (%) of Qwen3.6-35B-A3B under 20K and 100K output-token budgets across instance counts and label distributions. Results pool all five tasks, both majority labels and all seven order patterns.
Variant
Budget
Accuracy (%)
Δ Order Pattern ( pp )
Invalid [0.3ex]Rate (%)
All-Experiment
Valid-Only
All-Experiment
Valid-Only
Standard
20K
50.0 ± 4.0
75.5 ± 3.1
56.7 ± 5.8
61.7 ± 12.6
33.7 ± 5.3
100K
73.3 ± 1.9
74.2 ± 2.1
43.3 ± 11.5
50.0 ± 10.0
1.1 ± 1.1
Hint
20K
54.1 ± 4.6
75.6 ± 2.0
50.0 ± 17.3
63.9 ± 12.7
28.5 ± 4.5
100K
73.7 ± 3.4
73.7 ± 3.4
53.3 ± 15.3
53.3 ± 15.3
0.0 ± 0.0
Suggested Strategy
20K
51.9 ± 7.1
75.2 ± 6.7
43.3 ± 11.5
62.7 ± 11.3
31.1 ± 5.6
Appendix
Table 5: Accuracy, order-pattern gaps and invalid rates for Qwen3.6-35B-A3B under four prompts and two output-token budgets in the core setting. Values are means ± standard deviations over three instance-selection seeds. Best and worst patterns are selected separately for each prompt, budget and accuracy definition.
Figure 12: Per-ID mention rates for Qwen3.6-35B-A3B across valid core -setting experiments under 20K and 100K output-token budgets. Colours distinguish budgets. Filled and open markers highlight IDs at multiples of 100 and other multiples of 50, respectively.
Figure 13: Mean estimated explicit instance coverage for Qwen3.6-35B-A3B with a 100K output-token budget across instance counts and label distributions, using valid experiments only. Bold values give coverage percentages, with mean numbers of mentioned IDs in parentheses. Corresponding 20K results appear in Figure 23(c) .
Figure 14: Model error distribution by error type. Standard deviation is computed over three instance-selection seeds.
Figure 15: Invalid experiments for each model, task, and order pattern. Each cell reports the number of invalid experiments out of six runs, comprising two majority labels and three instance-selection seeds.
Model
Accuracy (%)
Δ Order Pattern ( pp )
Invalid [0.3ex]Rate (%)
All-Experiment
Valid-Only
All-Experiment
Valid-Only
DeepSeek-V4-Flash
47.8 ± 4.8
79.8 ± 6.9
50.0 ± 17.3
64.0 ± 16.0
40.0 ± 5.8
DeepSeek-V4-Pro
9.3 ± 5.3
78.6 ± 25.8
20.0 ± 10.0
50.0 †
88.9 ± 4.4
Gemini 3.1 Flash-Lite
56.3 ± 5.1
63.0 ± 4.1
63.3 ± 15.3
67.4 ± 12.2
10.7 ± 3.4
Gemini 3.1 Pro
80.7 ± 4.2
81.7 ± 4.2
40.0 ± 0.0
40.0 ± 0.0
1.1 ± 1.1
Gemma 4 26B A4B
45.9 ± 4.6
64.0 ± 6.9
63.3 ± 15.3
91.1 ± 7.8
28.1 ± 2.3
Appendix
Table 6: All-experiment and valid-only accuracy, best–worst order-pattern gaps, and invalid rates in the core setting. Results use a 20K output-token budget unless marked 100K. Values are means ± standard deviations over three instance-selection seeds unless marked. Valid-only gaps marked † use one paired seed (no standard deviation), and ‡ use two. The 20K valid-only metrics and invalid rates are repeated from Table 2 for comparison.
Figure 16: Model accuracy on each of the five tasks and overall. Standard deviation is reported over three instance-selection seeds.
Figure 17: Model accuracy on all seven order patterns, with each counterpart pair overlaid. Error bars indicate standard deviations over three instance-selection seeds.
Figure 18: Task accuracy on all seven order patterns, with each counterpart pair overlaid. Error bars indicate standard deviations over three instance-selection seeds.
Figure 19: Accuracy over different instance counts and label distributions , for all seven representative patterns. Standard deviation is computed over three random order seeds for random only.
Figure 20: Per-model accuracy changes (variant minus standard, in pp) across all seven order patterns. Mean changes and their sample standard deviations are computed over three instance-selection seeds.
Figure 21: Per-ID mention rate for each order pattern across valid core -setting experiments for models with available reasoning traces. Each narrow bin represents one instance ID, with red markers highlighting IDs that are multiples of 50 and black markers indicating block boundaries . Panel annotations give the number of valid experiments.
Figure 22: Mean estimated explicit instance coverage across instance counts and label distributions , pooled over valid experiments from the four reasoning models . Bold values give coverage percentages, and values in parentheses give the corresponding mean numbers of instance IDs mentioned per trace.
Figure 23: Average estimated explicit instance coverage for each model across instance counts and label distributions. Bold values give coverage percentages, and values in parentheses give the corresponding mean numbers of instance IDs mentioned per trace.
Variant
Budget
Invalid
<25%
25 – <75%
75 – <100%
=100%
Count
Rate (%)
Count
Acc. (%)
Count
Acc. (%)
Count
Acc. (%)
Count
Acc. (%)
Standard
20K
91
33.7
132
71.2
28
82.1
1
0.0
18
100.0
100K
3
1.1
101
70.3
27
70.4
22
77.3
117
77.8
Hint
20K
77
28.5
136
74.3
35
80.0
5
100.0
17
70.6
100K
0
0.0
96
71.9
40
77.5
22
81.8
112
72.3
Suggested Strategy
20K
84
31.1
130
70.8
28
78.6
3
100.0
25
92.0
Appendix
Table 7: Estimated explicit coverage and accuracy for Qwen3.6-35B-A3B under four prompts and two output-token budgets. Each row contains 270 experiments pooled over three instance-selection seeds. Coverage-bin counts include valid experiments only, and Acc. gives accuracy within each bin. The invalid rate uses all 270 experiments. Bold rows indicate the default 20K budget.
Figure 24: IxG position profiles for Gemma 4 26B A4B on NER , separated by order pattern.
Figure 25: IxG position profiles for Gemma 4 26B A4B on Sentiment , separated by order pattern.
Figure 26: IxG position profiles for Gemma 4 31B on NER , separated by order pattern.
Figure 27: IxG position profiles for Gemma 4 31B on Sentiment , separated by order pattern.
Figure 28: IxG position profiles for Qwen3-4B-Instruct on Math , separated by order pattern.
Figure 29: IxG position profiles for Qwen3-4B-Instruct on NER , separated by order pattern.
Figure 30: IxG position profiles for Qwen3-4B-Instruct on Occurrence , separated by order pattern.
Figure 31: IxG position profiles for Qwen3-4B-Instruct on Sentiment , separated by order pattern.
Figure 32: IxG position profiles for Qwen3-4B-Instruct on WSD , separated by order pattern.
Figure 33: IxG position profiles by instance count, pooled over valid replays from NER , Sentiment , all seven order patterns, and four label distributions. Each point averages a 5% position bin. The inset reports accuracy (%) on the attribution subset for each count and distribution. The dotted line at 1 is the uniform reference, and the y -axis is logarithmic.
Figure 34: IxG position profiles by label distribution, pooled over valid replays from NER , Sentiment , all seven order patterns, and five instance counts. The inset reports accuracy (%) on the attribution subset for each distribution and count. Binning, axes and normalisation follow Figure 33 .
Figure 35: IxG position profiles for Qwen3-4B-Instruct , pooled over valid replays from NER , Sentiment and all seven order patterns, for the standard prompt and three variants. The inset reports accuracy (%) on the same outputs for four representative patterns. Binning, axes and normalisation follow Figure 33 .
Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
School of Computer Science and Informatics, Cardiff University
Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.
Large language models are increasingly used for ordinal classification, yet semantically equivalent changes to prompt organization can alter their predictions. We conduct systematic experiments to characterize positional bias from label order, demonstration order, and demonstration placement. First, we apply the three probes to ten frontier LLMs on a common ordinal-classification task; every model is sensitive to all three positional sources, showing that the problem is pervasive. Second, we vary eight prompt-, task-, and model-level factors across five datasets; accuracy and stability are often misaligned, and only lower scale cardinality consistently improves both. Third, we compare pointwise, pairwise, and listwise inference, alternative aggregation and debiasing methods, and joint configurations; the tested corrections do not provide a reliable remedy, while a comparison-based listwise formulation offers the best balance but transfers unevenly across models and bias sources. These findings show that positional robustness depends on the full system configuration rather than the model alone. Ordinal-classification systems should therefore be selected jointly for predictive performance and stability.
Yu Wang, Zhe Zhou, Menglin Liu +1
Cornell University · University of Washington · The Chinese University of Hong Kong, Shenzhen +1