Text summarization is an important task for advancing natural language processing (NLP) research and applications in Indic languages. Despite progress in extractive and abstractive summarization, both monolingual summarization (MS) and cross-lingual summarization (CLS) remain relatively underexplored for these languages. Recent advances in Transformer architectures, pretrained language models (PLMs), and dataset availability offer new opportunities to address this gap. In this work, we construct cross-lingual training data from the monolingual Marathi MahaSum corpus and combine it with the smaller cross-lingual PMIndiaSum dataset to fine-tune IndicTrans2, an Indic neural machine translation model. We evaluate the resulting models using ROUGE, chrF and chrF++ metrics, focusing primarily on Marathi-to-English summarization due to computational constraints. We also discuss the applicability of our approach to the English-to-Marathi direction. Our experiments demonstrate that leveraging a pretrained translation model yields substantially better summarization performance than the IndicBART and mBART-50 based baselines reported in previous work on cross-lingual summarization.
Figures & tables
Metric
Mean
Median
Std. Dev.
chrF
56.34
56.88
11.70
chrF++
49.97
50.22
11.41
Character-length ratio
0.967
0.966
0.122
Word-length ratio
0.994
1.000
0.348
Table 1: Round-trip summary translation fidelity on the MahaSum validation split ( N=1,667 ). Scores compare original Marathi summaries with their Marathi back-translations.
Metric
Mean
Median
Std. Dev.
chrF
63.01
63.61
8.17
chrF++
58.69
59.15
7.71
Character-length ratio
0.981
0.909
0.713
Word-length ratio
0.971
0.916
0.718
Table 2: Round-trip text translation fidelity on the MahaSum validation split ( N=1,667 ). Scores compare original Marathi documents with their Marathi back-translations.
Dataset
Split
N
Text length
Summary length
Compression
Mean
Median
P95
Mean
Median
P95
Mean
Median
P95
MahaSum
Train
22000
316.99
300
511.00
23.32
22
42.0
16.67
13.75
36.94
MahaSum
Val
1667
320.70
302
528.10
23.10
22
42.0
17.08
14.20
37.64
MahaSum
Test
1667
91.03
71
187.00
18.48
18
32.0
6.21
4.47
17.25
XL-Sum
Train
10903
653.28
592
1424.90
32.57
30
60.0
23.34
18.76
58.15
XL-Sum
Val
1362
519.98
521
877.00
29.87
29
51.0
19.58
16.94
40.29
Table 3: Marathi-English dataset statistics for the training, validation, and test splits. Lengths are measured in words, and compression ratio is defined as source-text length divided by summary length.
Figure 1: Distributional comparison of PMIndiaSum, MahaSum, XL-Sum and CrossSum. Panels (a)–(c) show empirical cumulative distribution functions (ECDFs) of source-text length, summary length, and compression ratio, respectively; panel (d) shows source-text versus summary length. Colours denote PMIndiaSum (blue), MahaSum (orange), XL-Sum (red) and CrossSum (green). Solid and dashed lines denote training and validation splits, respectively.
Direction
Dataset
Original document
Original summary
Translated field
Data type
Role
MR → EN
MahaSum
MR
MR
SUMM
Synthetic
Training
MR → EN
XL-Sum-MR
MR
MR
Summary
Synthetic
Training
MR → EN
PMIndiaSum
EN
MR
None
Native
Evaluation
MR → EN
CrossSum
EN
MR
None
Native
Evaluation
EN → MR
MahaSum
MR
MR
TEXT
Synthetic
Training
EN → MR
XL-Sum-EN
EN
EN
Summary
Synthetic
Training
Table 4: Construction and use of datasets for the two cross-lingual summarization directions. “Translated field” identifies the field generated using IndicTrans2.
Hyperparameter
Value / Range
Description
Learning rate
1e-5 – 5e-5
Range explored; lower values can reduce catastrophic forgetting on smaller datasets
Batch size
2 – 8
Batch-size range considered; gradient accumulation may be used to increase the effective batch size
Optimizer
Adam ( β1=0.9 , β2=0.98 , ϵ=10−8 )
Standard optimizer for transformer models
Learning rate scheduler
Inverse square root / linear decay
Schedulers considered during fine-tuning
Dropout
0.1 – 0.3
Regularisation range explored during fine-tuning
Maximum sequence length
256 – 2048
Maximum number of tokens in each source–target pair
Table 5: Hyperparameters considered for fine-tuning the IndicTrans2 model
Training data
ROUGE-1
ROUGE-2
ROUGE-L
BERTScore-F1
PMIndiaSum
72.0151
53.2036
69.7104
99.0504
MahaSum
40.4998
22.3857
36.7866
98.5998
PMIndiaSum + MahaSum
72.4968
53.3091
69.8116
99.0485
PMIndiaSum + XL-Sum Marathi
72.2468
52.7246
69.4549
99.0442
XL-Sum Marathi
37.5122
20.8242
34.0955
98.5330
Table 9: MR → EN cross-lingual summarization results for the IndicTrans2 1B model under different training-data configurations.
Figure 2: Validation-metric trajectories during Marathi-to-English fine-tuning of indictrans2-200M model. Each line corresponds to training on PMIndiaSum, MahaSum, XL-Sum and their blended training set, as identified in the in-panel legends. The panels are enlarged to make the training dynamics and dataset labels legible; final held-out test-set scores should be reported separately in a results table.
Figure 3: Validation-metric trajectories during Marathi-to-English fine-tuning of rotary-indictrans2-200M model. Each line corresponds to training on PMIndiaSum, MahaSum, XL-Sum and their blended training set, as identified in the in-panel legends. The panels are enlarged to make the training dynamics and dataset labels legible; final held-out test-set scores should be reported separately in a results table.
Figure 4: Validation-metric trajectories during Marathi-to-English fine-tuning of 1B parameter models.
Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored. To address this gap, we introduce multi-target cross-lingual element-aware (MEA), a new MTXLS benchmark covering 24 target languages. We benchmark end-to-end and pipeline approaches across various LLMs and show that MTXLS performance still substantially lags behind English monolingual summarization. To better understand MTXLS in LLMs, we propose a layer-wise analysis framework for investigating how LLMs internally perform MTXLS. Our analyses suggest that translation and summarization behaviors emerge jointly within later layers rather than as distinctly decomposed stages. Most task-relevant processing occurs within these layers, and errors also tend to arise at similar depths. Motivated by these findings, we introduce an inference-time activation steering method that leverages hidden representations from English summarization to guide MTXLS generation. Experiments show that our method consistently improves MTXLS quality across target languages.
Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.
Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.