Organizations: The Hong Kong University of Science and Technology(Guangzhou) · China University of Petroleum (East China) · Northeastern University at Qinhuangdao · Lingnan University, Hong Kong.
Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however'' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.
Figures & tables
Granularity
Task
Avg Len
Metric
#Data
Word
Single-QA
23,887
Acc.
420
Multi-QA
29,048
Acc.
420
Sentence
Cloze
30,838
Acc.
420
Paragraph
Filling
33,687
Rouge-L
420
Document
Summarization
28,189
Rouge-L
420
Translation
33,294
BLEU
420
Table 1: Overview of MGAL task design across four levels of granularity in six UN official languages.
Benchmark
Max Len
En
Zh
Es
Fr
Ru
Ar
Pos-Par.
Pos-Doc.
Gran.
LongBench ( Bai et al., 2024 )
∼ 10K
✓
✓
×
×
×
×
×
✓
×
ZeroSCROLLS ( Shaham et al., 2023 )
∼ 10K
✓
×
×
×
×
×
×
✓
×
NeedleBench ( Li et al., 2024a )
∼ 10K
✓
✓
×
×
×
×
✓
×
×
RULER ( Hsieh et al., 2024a )
∼ 10K
✓
×
×
×
×
×
✓
×
×
LaRA ( Li et al., 2025 )
∼ 10K
✓
×
×
✓
×
×
×
✓
×
M 4 LE ( Kwan et al., 2024 )
∼ 10K
✓
✓
×
×
×
×
✓
×
×
Table 2: Comparison to existing long-context benchmarks. “En”, “Zh”, “Es”, “Fr”, “Ru”, and “Ar” refer to tasks in English, Chinese, Spanish, French, Russian, and Arabic,“Pos-Par.” and “Pos-Doc.” indicate positional evaluation at the paragraph and document level. “Gran.” refers to Granularity.
Figure 1: Overview of MGAL. From (a) aligned multilingual UN long-context documents, we construct (b) a multi-task benchmark covering QA, cloze, paragraph filling, summarization, and translation. These tasks are organized by (c) linguistic granularity and evaluated through (d) diagnostic analyses over granularity, position, and language.
Figure 2: MGAL dataset instantiation at each granularity.
Figure 3: MGAL data generation and human verification pipeline. For each granularity level, candidate instances are generated from aligned UN reports and then checked by human annotators for grounding, semantic consistency, positional correctness, and cross-lingual alignment.
Word
Sentence
Paragraph
Document
Model
Single-QA
Multi-QA
Cloze
Filling
Summarization
Translation
GPT-5
79.79
74.99
30.62
14.47
15.47
35.30
Claude Sonnet 4
72.75
73.76
21.53
12.81
22.09
14.96
Gemini-2.5-flash
77.01
75.49
26.00
15.97
23.89
36.17
Grok 4
78.49
72.64
21.58
19.90
18.93
31.05
Doubao-Seed-1.6
76.63
73.63
19.47
15.22
17.08
6.16
Table 3: Results on word, sentence, paragraph and document tasks. Open-source and proprietary models are separated with a divider, and the top-performing LLM is highlighted in bold, second best are underlined.
Figure 4: Performance of our evaluated models on MGAL. (a) shows LLM performance across different tasks, (b) presents multilingual performance, and (c) reports task performance across the six official UN languages. GPT-5 achieves the top average across granularity tasks, whereas Gemini-2.5-Flash excels across languages. And models achieve strong results on word-level tasks but exhibit weaker performance at coarser granularities.
Figure 5: Position Performance on MGAL. Performance dips in the middle for word and paragraph tasks but peaks there for sentence-level tasks.
Figure 6: Ablation studies: (a) Accuracy rises as instruction moves nearby to the blank. (b) Accuracy drops when high-confusion distractors are positioned earlier.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Prompt template for the Numerical type question generation.
Figure 8: Prompt template for the Reference type question generation.
Figure 9: Prompt template for the Classification type question generation.
Figure 10: Prompt template for the Synthesis type question generation.
Figure 11: Prompt template for the Retrieval type question generation.
Figure 12: Prompt template for the Comparision type question generation.
Figure 13: Numerical Type Sample Question.
Figure 14: Classification Type Sample Question.
Figure 15: Reference Type Sample Question.
Figure 16: Comparision Type Sample Question.
Figure 17: Synthesis Type Sample Question.
Figure 18: Retrieval Type Sample Question.
Figure 19: The prompt for LLM-as-a-judge.
Figure 20: An example prompt for the QA task.
Figure 21: An example prompt for the Cloze task.
Figure 22: An example prompt for the Paragraph filling task.
Figure 23: An example prompt for the Translation task.
Figure 24: An example prompt for the Summarization task.
Figure 25: Blank omissions heatmap by position across models.
Figure 26: Position choices heatmap with the ground truth and model predictions.
Figure 27: Repetition letter distribution.
Figure 28: Sentence analysis results heatmap.
Case 1: Regional human rights initiatives
Paragraph with blank. The extent of cooperation among government departments, as well as among Governments, academia, national human rights institutions and NGOs in any given country appears particularly significant. [#] Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community.
Gold sentence. Similarly, there seem to be an increasing number of regional initiatives, which States can both contribute to and benefit from, thereby fostering a valuable exchange of experiences and cross-fertilization of ideas.
Majority-selected wrong sentence. Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community.
Error analysis. The wrong sentence is locally plausible because it lists concrete regional organizations related to the same human-rights cooperation topic. However, the blank requires a general transition from national cooperation to regional initiatives before the examples are introduced. Models therefore confuse a concrete example sentence with the higher-level framing sentence required by the paragraph.
Case 2: Gender dimension in HIV policies
Paragraph with blank. [#] Member States have continued to use two approaches to address the gender dimension in HIV policies and action plans. The UNDP/World Bank/UNAIDS HIV Mainstreaming Programme supported Namibia, Botswana, Lesotho and Swaziland in strengthening their development planning processes to better integrate and implement HIV and gender priorities.
Appendix
Table 4: Qualitative examples of sentence selection errors in long-context modeling.
Model
Correct
Omission
Position Error
Distractor
GPT-5
52.89
17.43
21.51
8.17
Gemini-2.5-Flash
52.87
14.66
22.39
10.08
Doubao-Seed-1.6
47.25
24.06
21.72
6.96
Qwen3-235B-A22B
43.92
6.54
40.37
9.17
Mistral-Small-3.2-24B
40.57
4.32
40.34
14.77
Qwen3-30B-A3B
37.76
7.93
44.95
9.36
Appendix
Table 5: Error breakdown for the sentence-cloze task. Omission denotes failing to fill the blank with any correct sentence; Position Error denotes selecting a correct sentence from the wrong document position; Distractor denotes selecting an incorrect distractor option.
Task
Model
English
Chinese
Spanish
French
Russian
Arabic
Single-QA
GPT5
81.25
78.33
80.83
82.29
77.50
78.54
Claude Sonnet 4
72.75
78.95
70.00
74.38
66.88
71.88
Gemini-2.5-flash
79.16
75.42
78.54
78.54
73.75
76.67
Grok 4
78.75
76.25
78.75
79.17
62.5
77.29
Doubao-Seed-1.6
78.54
75.83
78.75
81.46
68.75
76.46
Qwen3-235B-A22B
82.29
53.75
77.71
77.71
66.04
73.54
Appendix
Table 6: Model Performance Across All Tasks and Languages
Task
Model
Begin
Middle
End
Single-QA
GPT5
80.42
78.85
80.21
Claude Sonnet 4
75.44
77.43
71.75
Gemini-2.5-flash
77.4
75.94
77.71
Grok 4
77.19
78.75
80.31
Doubao-Seed-1.6
76.88
76.62
76.88
Qwen3-235B-A22B
72.4
69.27
71.35
Appendix
Table 7: Model Performance on Multi-QA Task at Document-Level.Rows indicate the position of the first paragraph and columns indicate the position of the second, with each cell representing the performance for the corresponding positional pair (e.g., ’Begin-Middle’).
Task
Position
Model
Begin
Middle
End
Multi-QA
Begin
GPT5
76.54
78.79
84.18
Claude Sonnet 4
77.88
72.86
79.52
Gemini-2.5-flash
75.12
77.8
70.69
Grok 4
72.79
66.08
69.71
Doubao-Seed-1.6
66.93
57.44
74.54
Qwen3-235B-A22B
70.21
72.15
75.02
Appendix
Table 8: Model Performance in Single-QA Task at Document-Level.Accuracy on the Single-QA task, categorized by the position (Begin, Middle, End) of the single answer-bearing paragraph.
Task
Document-level
Model
Paragraph-level
Begin
Middle
End
Cloze
Begin
GPT5
30.17
21.34
25.79
Claude Sonnet 4
27.15
19.72
17.50
Gemini-2.5-flash
26.74
24.54
21.73
Grok 4
28.52
22.83
20.92
Doubao-Seed-1.6
30.31
21.32
14.77
Appendix
Table 9: Model performance on the Cloze task, evaluated by hierarchical position. We measure performance at the beginning (Begin), middle (Middle), and end (End) of the document, and for each of these sections, we further evaluate at the beginning, middle, and end of the target paragraph. Scores are reported in accuracy.Best results are marked inbold, second best results are underlined.
Task
Model
Begin
Middle
End
Rouge-L
LLM
Rouge-L
LLM
Rouge-L
LLM
Filling
GPT5
14.55
43.64
13.15
44.64
14.13
46.43
Claude Sonnet 4
13.77
43.57
12.87
50.39
15.49
49.34
Gemini-2.5-flash
16.04
46.26
14.66
52.96
15.26
52.18
Grok 4
19.58
44.27
19.28
51.34
19.34
56.14
Doubao-Seed-1.6
15.21
44.47
14.35
48.91
14.26
48.69
Appendix
Table 10: Model performance on the Filling task, evaluated by position. We measure performance at the beginning (Begin), middle (Middle), and end (End) of the document. Scores are reported in Rouge-L and via an LLM-based judge (“LLM”).Best results are marked inbold, second best results are underlined.
Single-QA
Multi-QA
Granularity-Tasks
Numerical
Classification
Reference
Comparison
Retrieval
Synthesis
GPT-5
73.44
80.73
85.31
62.66
76.52
87.01
Claude Sonnet 4
62.89
80.64
81.05
66.77
73.84
82.71
Gemini-2.5-flash
67.29
81.56
82.19
63.9
68.72
84.64
Grok-4
72.5
80.00
83.75
48.23
79.66
67.77
Doubao-Seed-1.6
69.12
78.75
82.5
55.91
65.73
74.77
Appendix
Table 11: Performance of various models on different types of word-level question-answering tasks.On word-level question-answering tasks, GPT-5 achieves the highest scores in Numerical, Reference, and Synthesis. Gemini-2.5-flash leads in Classification, Qwen3-30B-A3B excels in Comparison, and Kimi-K2 delivers the best performance in Retrieval.Best results are marked inbold, second best results are underlined.
Model
Topic Fid.
Local Coh.
Entity Cons.
Instr. Foll.
Format Comp.
Overall
GPT-5
4.58
10.51
4.39
5.36
19.96
44.81
Claude Sonnet 4
4.83
12.03
5.24
5.74
19.81
47.65
Gemini-2.5-flash
5.90
11.27
6.37
6.91
19.93
50.38
Grok-4
5.94
11.83
5.89
6.70
19.86
50.22
Doubao-Seed-1.6
5.08
11.29
5.11
5.90
19.92
47.29
Qwen3-235B-A22B
4.67
12.07
4.91
5.74
20.00
47.39
Appendix
Table 12: Model performance across different metrics based on llm-as-a-judge in six UN languages. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, Gemini-2.5-flash achieves the highest overall score, also leading in Entity Consistency and Instruction Following. Grok-4 delivers the best Topic Fidelity, while Qwen3-235B-A22B is the strongest in Local Coherence.Best results are marked inbold, second best results are underlined.
Model
Topic Fid.
Local Coh.
Entity Cons.
Instr. Foll.
Format Comp.
Overall
GPT-5
5.32
12.73
6.78
6.51
19.88
51.22
Claude Sonnet 4
5.26
12.42
6.54
6.63
19.84
50.69
Gemini-2.5-flash
5.93
11.89
6.32
6.28
19.85
50.27
Grok-4
5.14
11.96
5.87
5.74
20.00
48.71
Doubao-Seed-1.6
5.07
11.78
5.21
5.47
19.93
47.46
Qwen3-235B-A22B
6.15
12.86
6.78
5.83
19.83
51.54
Appendix
Table 13: Model performance across different metrics based on human evaluation in Chinese and English. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, DeepSeek V3.1 achieves the highest overall score, also leading in Instruction Following. Qwen3-235B-A22B delivers the best Topic Fidelity, Local Coherence and Entity Consistency, while GPT-5 also lead in Entity Consistency and Qwen3-30B-A3B also delivers the best Local Coherence. Grok-4 and Mistral-Small-3.2-24B are the strongest in Local Coherence. Best results are marked inbold, second best results are underlined.
Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.
Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.
Shiwen Qin, Cameron Barker, Henry Gouk +4
University of Edinburgh, Edinburgh, United Kingdom