Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.
Figures & tables
Figure 1: A toy example of SIP and MIP settings for sentiment analysis, where an LLM succeeds under SIP but fails under MIP given the same instances.
Name
Task
Arithmetic
Solve arithmetic problems & Sum of answers
Category
Classify news category & Aggregate class counts
Language
Identify language & Aggregate class counts
NER
Count “person” entities & Aggregate total counts
Parity
Detect odd or even number & Aggregate counts
Sentiment
Detect sentiment polarity & Aggregate counts
Table 1: Overview of the selected tasks and their aggregation logic in the MIP setting. More details can be found in Appendix A .
Figure 2: Model success rates (averaged across all tasks) as a function of the number of instances. Error bars indicate standard deviation across five random seeds. LLMs from the same company share the same colour family, while markers denote categories: ∙ (open-weight, ≥ 37B active parameters), ■ (open-weight, ≤ 22B active parameters), ▲ (frontier closed-source), and ⧫ (lightweight closed-source).
Figure 3: Task success rates (averaged across all LLMs) as a function of the number of instances. Error bars indicate standard deviation across five random seeds.
Model
Size /
Success
Invalid
Cost
(%)
(%)
DeepSeek R1
A37B
67.5 ± 2.6
2.9 ± 0.6
DeepSeek V3
A37B
39.0 ± 3.6
2.9 ± 0.6
gpt-oss-120b
A5.1B
68.3 ± 2.8
3.6 ± 1.1
gpt-oss-20b
A3.6B
60.8 ± 2.5
4.9 ± 0.6
Llama 3.3
70B
39.0 ± 3.8
2.9 ± 0.6
Table 2: Model success rate and invalid rate (mean ± std), averaged across all tasks and instance counts. Standard deviation is computed over five random seeds. The top nine models are open-weight LLMs, for which we report model size, with “A” denoting active parameters for mixture-of-experts models (total sizes 21B–671B). The bottom seven are closed-source LLMs with undisclosed sizes, for which we report average cost per task across five runs.
Figure 4: Success rates as a function of the number of instances for the original instance order and two shuffled variants constructed from the same instance sets.
Figure 5: Breakdown of failure types. Key mistakes, aggregation mistakes, individual mistakes, and combined mistakes (Agg . +Indi.) are categorised as wrong answers (blue), while parsing errors and overlong input errors are categorised as invalid outputs (orange).
Model
Wrong Exp.
Wrong Indv. Per Exp.
DeepSeek R1
21.0 ± 3.7
49.5 ± 9.9
DeepSeek V3
29.6 ± 2.7
34.8 ± 13.1
gpt-oss-120b
16.2 ± 2.9
37.5 ± 17.4
gpt-oss-20b
15.8 ± 2.4
11.8 ± 5.0
Llama 3.3
35.6 ± 2.2
53.8 ± 26.7
Llama 4 Maverick
33.6 ± 4.2
38.4 ± 4.6
Table 3: Percentage of experiments with at least one incorrect instance-level prediction (Wrong Exp.) and the average number of incorrect instance-level predictions per failed experiment (mean ± std). Standard deviation is computed over five random seeds.
Figure 6: Success rate (lines) and total prompt token length (bars) in the artificial length setting as the number of instances increases. Error bars indicate standard deviation across five random seeds.
Figure 7: Success rates as a function of the number of instances for different injected-noise positions constructed from the same instance sets.
Number of Instances
Success rate and Context length
Correlation
P-Value
2
0.043
0.498
5
0.044
0.491
10
0.044
0.490
20
-0.042
0.510
50
-0.033
0.602
Table 4: Correlation between success rate and prompt token length when the number of instances is fixed.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Agreement (%)
BShift
69.3
SubjNum
82.5
Voice
78.3
Appendix
Table 5: Task filtering across tasks that have been removed.
Task
Agreement (%)
Max (%)
Min (%)
Arithmetic
89.0
99.5
93.7
Category
94.8
99.2
97.2
Language
98.8
99.6
99.2
NER
87.6
97.3
93.6
Parity
100.0
100.0
100.0
Sentiment
96.4
99.7
97.8
Appendix
Table 7: Task filtering results showing agreement (i.e., the percentage of instances for which all LLMs produce correct SIP predictions), as well as the maximum and minimum SIP success rates across all LLMs (the actual agreement and minimum rate for Parity is 99.96%).
Figure 8: Success rate of models.
Figure 9: Success rate of models.
Figure 10: Success rate of models.
Figure 11: Success rate of models.
Figure 12: Model success rate for tasks.
Figure 13: Model success rate for tasks.
Figure 14: Failure breakdown for models.
Figure 15: Failure breakdown for models.
Figure 16: Failure breakdown for models.
Figure 17: Failure breakdown for tasks.
Figure 18: Failure breakdown for tasks.
Figure 19: Success rate (lines) and the number of instances (bars) in the artificial length setting as total prompt token length increases. Error bars indicate standard deviation across five random seeds.