MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Chunhan Li; Chenglin Xu; Zongyang Zhang; Jiale Liu; Zhuoxi Rao; Xudong Jia; Junxiu He; Menglin Yang

cs.AIAug 21, 2026

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Authors: Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong Jia, Junxiu He, Menglin Yang,

Organizations: The Hong Kong University of Science and Technology(Guangzhou) · China University of Petroleum (East China) · Northeastern University at Qinhuangdao · Lingnan University, Hong Kong.

Abstract

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however'' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

Figures & tables

Granularity	Task	Avg Len	Metric	#Data
Word	Single-QA	23,887	Acc.	420
Word	Multi-QA	29,048	Acc.	420
Sentence	Cloze	30,838	Acc.	420
Paragraph	Filling	33,687	Rouge-L	420
Document	Summarization	28,189	Rouge-L	420
Document	Translation	33,294	BLEU	420

Table 1: Overview of MGAL task design across four levels of granularity in six UN official languages.

Benchmark	Max Len	En	Zh	Es	Fr	Ru	Ar	Pos-Par.	Pos-Doc.	Gran.
LongBench ( Bai et al., 2024 )	$\sim$ 10K	✓	✓	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$
ZeroSCROLLS ( Shaham et al., 2023 )	$\sim$ 10K	✓	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$
NeedleBench ( Li et al., 2024a )	$\sim$ 10K	✓	✓	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$
RULER ( Hsieh et al., 2024a )	$\sim$ 10K	✓	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$
LaRA ( Li et al., 2025 )	$\sim$ 10K	✓	$\times$	$\times$	✓	$\times$	$\times$	$\times$	✓	$\times$
M 4 LE ( Kwan et al., 2024 )	$\sim$ 10K	✓	✓	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$

Table 2: Comparison to existing long-context benchmarks. “En”, “Zh”, “Es”, “Fr”, “Ru”, and “Ar” refer to tasks in English, Chinese, Spanish, French, Russian, and Arabic,“Pos-Par.” and “Pos-Doc.” indicate positional evaluation at the paragraph and document level. “Gran.” refers to Granularity.

Figure 1: Overview of MGAL. From (a) aligned multilingual UN long-context documents, we construct (b) a multi-task benchmark covering QA, cloze, paragraph filling, summarization, and translation. These tasks are organized by (c) linguistic granularity and evaluated through (d) diagnostic analyses over granularity, position, and language.

Figure 2: MGAL dataset instantiation at each granularity.

Figure 3: MGAL data generation and human verification pipeline. For each granularity level, candidate instances are generated from aligned UN reports and then checked by human annotators for grounding, semantic consistency, positional correctness, and cross-lingual alignment.

	Word		Sentence	Paragraph	Document
Model	Single-QA	Multi-QA	Cloze	Filling	Summarization	Translation
GPT-5	79.79	74.99	30.62	14.47	15.47	35.30
Claude Sonnet 4	72.75	73.76	21.53	12.81	22.09	14.96
Gemini-2.5-flash	77.01	75.49	26.00	15.97	23.89	36.17
Grok 4	78.49	72.64	21.58	19.90	18.93	31.05
Doubao-Seed-1.6	76.63	73.63	19.47	15.22	17.08	6.16

Table 3: Results on word, sentence, paragraph and document tasks. Open-source and proprietary models are separated with a divider, and the top-performing LLM is highlighted in bold, second best are underlined.

Figure 4: Performance of our evaluated models on MGAL. (a) shows LLM performance across different tasks, (b) presents multilingual performance, and (c) reports task performance across the six official UN languages. GPT-5 achieves the top average across granularity tasks, whereas Gemini-2.5-Flash excels across languages. And models achieve strong results on word-level tasks but exhibit weaker performance at coarser granularities.

Figure 5: Position Performance on MGAL. Performance dips in the middle for word and paragraph tasks but peaks there for sentence-level tasks.

Figure 6: Ablation studies: (a) Accuracy rises as instruction moves nearby to the blank. (b) Accuracy drops when high-confusion distractors are positioned earlier.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Prompt template for the Numerical type question generation.

Figure 8: Prompt template for the Reference type question generation.

Figure 9: Prompt template for the Classification type question generation.

Figure 10: Prompt template for the Synthesis type question generation.

Figure 11: Prompt template for the Retrieval type question generation.

Figure 12: Prompt template for the Comparision type question generation.

Figure 13: Numerical Type Sample Question.

Figure 14: Classification Type Sample Question.

Figure 15: Reference Type Sample Question.

Figure 16: Comparision Type Sample Question.

Figure 17: Synthesis Type Sample Question.

Figure 18: Retrieval Type Sample Question.

Figure 19: The prompt for LLM-as-a-judge.

Figure 20: An example prompt for the QA task.

Figure 21: An example prompt for the Cloze task.

Figure 22: An example prompt for the Paragraph filling task.

Figure 23: An example prompt for the Translation task.

Figure 24: An example prompt for the Summarization task.

Figure 25: Blank omissions heatmap by position across models.

Figure 26: Position choices heatmap with the ground truth and model predictions.

Figure 27: Repetition letter distribution.

Figure 28: Sentence analysis results heatmap.

Case 1: Regional human rights initiatives
	Paragraph with blank. The extent of cooperation among government departments, as well as among Governments, academia, national human rights institutions and NGOs in any given country appears particularly significant. [#] Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community.
	Gold sentence. Similarly, there seem to be an increasing number of regional initiatives, which States can both contribute to and benefit from, thereby fostering a valuable exchange of experiences and cross-fertilization of ideas.
	Majority-selected wrong sentence. Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community.
	Error analysis. The wrong sentence is locally plausible because it lists concrete regional organizations related to the same human-rights cooperation topic. However, the blank requires a general transition from national cooperation to regional initiatives before the examples are introduced. Models therefore confuse a concrete example sentence with the higher-level framing sentence required by the paragraph.
Case 2: Gender dimension in HIV policies
	Paragraph with blank. [#] Member States have continued to use two approaches to address the gender dimension in HIV policies and action plans. The UNDP/World Bank/UNAIDS HIV Mainstreaming Programme supported Namibia, Botswana, Lesotho and Swaziland in strengthening their development planning processes to better integrate and implement HIV and gender priorities.

Appendix

Table 4: Qualitative examples of sentence selection errors in long-context modeling.

Model	Correct	Omission	Position Error	Distractor
GPT-5	52.89	17.43	21.51	8.17
Gemini-2.5-Flash	52.87	14.66	22.39	10.08
Doubao-Seed-1.6	47.25	24.06	21.72	6.96
Qwen3-235B-A22B	43.92	6.54	40.37	9.17
Mistral-Small-3.2-24B	40.57	4.32	40.34	14.77
Qwen3-30B-A3B	37.76	7.93	44.95	9.36

Appendix

Table 5: Error breakdown for the sentence-cloze task. Omission denotes failing to fill the blank with any correct sentence; Position Error denotes selecting a correct sentence from the wrong document position; Distractor denotes selecting an incorrect distractor option.

Task	Model	English	Chinese	Spanish	French	Russian	Arabic
Single-QA	GPT5	81.25	78.33	80.83	82.29	77.50	78.54
	Claude Sonnet 4	72.75	78.95	70.00	74.38	66.88	71.88
	Gemini-2.5-flash	79.16	75.42	78.54	78.54	73.75	76.67
	Grok 4	78.75	76.25	78.75	79.17	62.5	77.29
	Doubao-Seed-1.6	78.54	75.83	78.75	81.46	68.75	76.46
	Qwen3-235B-A22B	82.29	53.75	77.71	77.71	66.04	73.54

Appendix

Table 6: Model Performance Across All Tasks and Languages

Task	Model	Begin	Middle	End
Single-QA	GPT5	80.42	78.85	80.21
	Claude Sonnet 4	75.44	77.43	71.75
	Gemini-2.5-flash	77.4	75.94	77.71
	Grok 4	77.19	78.75	80.31
	Doubao-Seed-1.6	76.88	76.62	76.88
	Qwen3-235B-A22B	72.4	69.27	71.35

Appendix

Table 7: Model Performance on Multi-QA Task at Document-Level.Rows indicate the position of the first paragraph and columns indicate the position of the second, with each cell representing the performance for the corresponding positional pair (e.g., ’Begin-Middle’).

Task	Position	Model	Begin	Middle	End
Multi-QA	Begin	GPT5	76.54	78.79	84.18
		Claude Sonnet 4	77.88	72.86	79.52
		Gemini-2.5-flash	75.12	77.8	70.69
		Grok 4	72.79	66.08	69.71
		Doubao-Seed-1.6	66.93	57.44	74.54
		Qwen3-235B-A22B	70.21	72.15	75.02

Appendix

Table 8: Model Performance in Single-QA Task at Document-Level.Accuracy on the Single-QA task, categorized by the position (Begin, Middle, End) of the single answer-bearing paragraph.

Task	Document-level	Model	Paragraph-level
Task	Document-level	Model	Begin	Middle	End
Cloze	Begin	GPT5	30.17	21.34	25.79
		Claude Sonnet 4	27.15	19.72	17.50
		Gemini-2.5-flash	26.74	24.54	21.73
		Grok 4	28.52	22.83	20.92
		Doubao-Seed-1.6	30.31	21.32	14.77

Appendix

Table 9: Model performance on the Cloze task, evaluated by hierarchical position. We measure performance at the beginning (Begin), middle (Middle), and end (End) of the document, and for each of these sections, we further evaluate at the beginning, middle, and end of the target paragraph. Scores are reported in accuracy.Best results are marked inbold, second best results are underlined.

Task	Model	Begin		Middle		End
Task	Model	Rouge-L	LLM	Rouge-L	LLM	Rouge-L	LLM
Filling	GPT5	14.55	43.64	13.15	44.64	14.13	46.43
	Claude Sonnet 4	13.77	43.57	12.87	50.39	15.49	49.34
	Gemini-2.5-flash	16.04	46.26	14.66	52.96	15.26	52.18
	Grok 4	19.58	44.27	19.28	51.34	19.34	56.14
	Doubao-Seed-1.6	15.21	44.47	14.35	48.91	14.26	48.69

Appendix

Table 10: Model performance on the Filling task, evaluated by position. We measure performance at the beginning (Begin), middle (Middle), and end (End) of the document. Scores are reported in Rouge-L and via an LLM-based judge (“LLM”).Best results are marked inbold, second best results are underlined.

	Single-QA			Multi-QA
Granularity-Tasks	Numerical	Classification	Reference	Comparison	Retrieval	Synthesis
GPT-5	73.44	80.73	85.31	62.66	76.52	87.01
Claude Sonnet 4	62.89	80.64	81.05	66.77	73.84	82.71
Gemini-2.5-flash	67.29	81.56	82.19	63.9	68.72	84.64
Grok-4	72.5	80.00	83.75	48.23	79.66	67.77
Doubao-Seed-1.6	69.12	78.75	82.5	55.91	65.73	74.77

Appendix

Table 11: Performance of various models on different types of word-level question-answering tasks.On word-level question-answering tasks, GPT-5 achieves the highest scores in Numerical, Reference, and Synthesis. Gemini-2.5-flash leads in Classification, Qwen3-30B-A3B excels in Comparison, and Kimi-K2 delivers the best performance in Retrieval.Best results are marked inbold, second best results are underlined.

Model	Topic Fid.	Local Coh.	Entity Cons.	Instr. Foll.	Format Comp.	Overall
GPT-5	4.58	10.51	4.39	5.36	19.96	44.81
Claude Sonnet 4	4.83	12.03	5.24	5.74	19.81	47.65
Gemini-2.5-flash	5.90	11.27	6.37	6.91	19.93	50.38
Grok-4	5.94	11.83	5.89	6.70	19.86	50.22
Doubao-Seed-1.6	5.08	11.29	5.11	5.90	19.92	47.29
Qwen3-235B-A22B	4.67	12.07	4.91	5.74	20.00	47.39

Appendix

Table 12: Model performance across different metrics based on llm-as-a-judge in six UN languages. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, Gemini-2.5-flash achieves the highest overall score, also leading in Entity Consistency and Instruction Following. Grok-4 delivers the best Topic Fidelity, while Qwen3-235B-A22B is the strongest in Local Coherence.Best results are marked inbold, second best results are underlined.

Model	Topic Fid.	Local Coh.	Entity Cons.	Instr. Foll.	Format Comp.	Overall
GPT-5	5.32	12.73	6.78	6.51	19.88	51.22
Claude Sonnet 4	5.26	12.42	6.54	6.63	19.84	50.69
Gemini-2.5-flash	5.93	11.89	6.32	6.28	19.85	50.27
Grok-4	5.14	11.96	5.87	5.74	20.00	48.71
Doubao-Seed-1.6	5.07	11.78	5.21	5.47	19.93	47.46
Qwen3-235B-A22B	6.15	12.86	6.78	5.83	19.83	51.54

Appendix

Table 13: Model performance across different metrics based on human evaluation in Chinese and English. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, DeepSeek V3.1 achieves the highest overall score, also leading in Instruction Following. Qwen3-235B-A22B delivers the best Topic Fidelity, Local Coherence and Entity Consistency, while GPT-5 also lead in Entity Consistency and Qwen3-30B-A3B also delivers the best Local Coherence. Grok-4 and Mistral-Small-3.2-24B are the strongest in Local Coherence. Best results are marked inbold, second best results are underlined.

Table 42

Table 43

Table 44

Table 45

Table 46

Table 47

Long-Context Language Model Evaluation

Long-context language model evaluation measures how reliably models find, reason over, and use information across extended inputs.

See all 68 papers

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model Evaluation Long-Context Language Model Inference
University of Alberta · Alberta Machine Intelligence Institute (Amii)
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Jul 9, 2026Siddhartha Jain, Ameya VelingkerLLM Evaluation Long-Context Language Model Evaluation
NVIDIA
Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Oct 15, 2026Shiwen Qin, Cameron Barker, Henry Gouk +4Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
University of Edinburgh, Edinburgh, United Kingdom

Sep 29, 2026J/K move · Enter open · S save

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye

University of Alberta · Alberta Machine Intelligence Institute (Amii)

Long-Context Language Model Evaluation Long-Context Language Model Inference Efficient Language Model Reasoning Long-Context Inference Long-Context Reasoning in Language Models

Open paper AskPDF arXiv

Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.

Abstract

Long-Context Language Model Evaluation

Explore similar work

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning