Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.
Figures & tables
Model
Access Provider
Type
GPT-5.6 Luna
OpenAI
Closed-source
GPT-4o Mini
OpenAI
Closed-source
Gemini 3.1 Flash Lite
Google AI Studio
Closed-source
Gemini 2.5 Flash
Google AI Studio
Closed-source
LLaMA 3.3 70B
Groq
Open-weight
DeepSeek V4 Flash
OpenRouter
Open-weight
Table 1: Overview of the eight evaluated LLMs and their access providers.
Lang.
Model
R-1
R-2
R-L
BLEU
chrF
BERTScore
SE
BSE
English
GPT-5.6 Luna
24.12
5.87
15.96
2.06
31.51
86.75
58.05
—
Gemini 3.1 Flash Lite
22.94
5.89
15.32
2.14
30.39
87.01
52.28
—
Gemini 2.5 Flash
22.13
5.43
14.72
1.90
29.62
86.57
59.44
—
Qwen 3.5 Flash
21.99
4.93
14.48
1.66
29.46
86.69
58.52
—
Qwen 3.6 27B
21.47
4.85
13.99
1.68
28.96
86.57
56.58
—
DeepSeek V4 Flash
21.46
5.28
14.13
1.82
29.36
86.30
63.45
—
Table 2: English and Bengali XL-Sum benchmark results. R-1, R-2, and R-L denote ROUGE-1, ROUGE-2, and ROUGE-L, respectively. BERTScore, SE, and BSE denote BERTScore-F1, SummEval F1, and BanglaSummEval F1, respectively. All metrics are reported on a 0–100 scale. A dash (—) indicates that the corresponding language-specific factuality score is not applicable. Bold values indicate the best performance within each language group.
Comparison
Mean ± SD
95% CI
t
Paired t -test p / Bootstrap p
LLaMA 3.3 70B
16.23 ± 9.05
[15.69, 16.79]
2.713
0.0067 / 0.0070
GPT-4o Mini
15.63 ± 8.27
[15.12, 16.13]
LLaMA 3.3 70B
16.23 ± 9.05
[15.69, 16.79]
5.364
<0.0001 / <0.001
Gemini 2.5 Flash
15.05 ± 7.86
[14.55, 15.52]
Table 3: Statistical comparison of selected model pairs on Bengali ROUGE-1 ( N=1,012 ). Bootstrap confidence intervals and bootstrap significance estimates are based on 5,000 paired resamples.
Lang.
Model
Rel
Comp
Faith
Coh
Conc
Flue
English
GPT-5.6 Luna
3.16
3.54
1.24
4.76
3.36
5.00
Gemini 3.1 Flash Lite
3.04
3.48
1.34
4.82
3.14
5.00
DeepSeek V4 Flash
2.92
3.38
1.30
4.78
3.06
5.00
Qwen 3.5 Flash
2.96
3.26
1.28
4.70
3.16
5.00
LLaMA 3.3 70B
3.02
3.36
1.24
4.78
2.80
5.00
Qwen 3.6 27B
2.86
3.26
1.24
4.76
3.10
5.00
Table 4: LLM-as-a-Judge ( gpt-5.6-terra ) evaluation using a 1–5 Likert scale. Rel: Relevance; Comp: Completeness; Faith: Faithfulness; Coh: Coherence; Conc: Conciseness; Flue: Fluency. Bold values indicate the best performance within each language group.