Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization

Chaimae Chellaf; Salima Mdhaffar; Yannick Estève; Stéphane Huet

cs.CLMar 9, 2026

Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization

Authors: Chaimae Chellaf, Salima Mdhaffar, Yannick Estève, Stéphane Huet

Organizations: LIA - Avignon Université, France · Lundi Matin, France

Abstract

Abstractive summarization aims to generate concise summaries by creating new sentences, allowing for flexible rephrasing. However, this approach can be vulnerable to inaccuracies, particularly `hallucinations' where the model introduces non-existent information. In this paper, we leverage the use of multimodal and multilingual sentence embeddings derived from pretrained models such as LaBSE, SONAR, and BGE-M3, and feed them into a modified BART-based French model. A Named Entity Injection mechanism that appends tokenized named entities to the decoder input is introduced, in order to improve the factual consistency of the generated summary. Our novel framework, SBARThez, is applicable to both text and speech inputs and supports cross-lingual summarization; it shows competitive performance relative to token-level baselines, especially for low-resource languages, while generating more concise and abstract summaries.

Figures & tables

Figure 1: The sentence-based summarization model architecture

Dataset	Train	Dev	Test
MLSUM	392,876	16,059	15,828
OrangeSum	21,401	1,500	1,500
DECODA	1,390	396	200
WikiLingua	43,423	6,193	12,405
CrossSum	8,648	1,083	1,083

Table 1: Number of documents per dataset

Model	R-L $\uparrow$	BertS $\uparrow$	NEHR $\downarrow$
Ground Truth	100	100	34.01
BARThez	22.57	27.95	26.88
SBARThez-BGE	19.24	24.61	58.52
SBARThez-LaBSE	17.53	22.83	71.79
SBARThez-SONAR	19.10	24.58	59.23

Table 2: Results on the OrangeSum test-set. Best results are bold , and second-best results are underlined . BertS refers to BertScore.

Figure 2: The sentence-based summarization model architecture with the NEI mechanism

Model	R-L $\uparrow$	BertS $\uparrow$	NEHR $\downarrow$
Ground Truth	100	100	34.01
BARThez	22.57	27.95	26.88
SBARThez-BGE	20.22	26.05	34.16
SBARThez-LaBSE	18.56	24.24	38.41
SBARThez-SONAR	19.40	25.37	36.31

Table 3: Results on the OrangeSum test-set, with SBARThez models using the NEI module.

Model	Ground Truth		Source Document			Length
Model	R-L $\uparrow$	BertS $\uparrow$	P-R1 $\downarrow$	EFC $\downarrow$	EFD $\downarrow$	Length
Token-Encoding
BARThez	22.57	27.95	91.65	89.11	6.66	24.71
flanT5-base	22.61	27.54	94.00	92.37	8.60	27.87
mT5-small	20.22	24.06	95.22	95.39	9.99	25.88
mBART-50	22.05	26.76	93.13	90.96	8.63	27.72

Table 4: Comparison of summarization methods on OrangeSum in terms of abstractiveness, accuracy, and summary length (average words excluding punctuation).

	SBARThez BGE		SBARThez LaBSE		SBARThez SONAR
	R-L $\uparrow$	BertS $\uparrow$	R-L $\uparrow$	BertS $\uparrow$	R-L $\uparrow$	BertS $\uparrow$
FR	24.18	32.80	20.17	26.32	23.18	32.21
EN	20.29	28.30	18.87	24.89	21.56	30.03
ES	20.06	28.70	18.80	24.25	21.54	29.62
IT	19.89	28.63	18.25	24.38	20.82	29.47
PT	19.70	28.50	18.56	25.22	21.19	30.07

Table 5: Results on WikiLingua commun test-set.

	EN	AR	RU	UK	KO	JA	VI	ID	SW	HI
SBARThez-BGE (ours)	29.40	29.83	29.15	29.91	28.59	33.25	28.92	30.64	30.95	25.61
MT+BARThez	29.11	30.71	30.05	28.89	28.68	15.65	30.18	31.62	31.33	26.49
LLaMA-8B	25.00	26.33	24.61	25.61	24.19	29.23	26.12	25.61	21.48	20.68
#Test-Dev Samples	377	197	160	143	52	54	88	204	174	119

Table 6: Results on CrossSum using BertScore

\uparrow

– High-Resource Languages

	PA	UR	PS	MR	YO	SO	AZ	TA	MY	SI	GU	IG	RN	PCM
SBARThez-BGE (ours)	18.40	17.19	28.38	25.81	30.75	28.52	29.44	25.72	26.03	25.98	25.11	26.77	26.14	29.81
MT+BARThez	16.05	17.67	27.42	24.37	16.25	05.28	22.54	15.24	06.44	27.09	03.09	05.45	-	-
LLaMA-8B	14.62	21.61	25.99	19.89	23.96	21.64	24.22	15.32	21.65	24.03	19.38	17.92	22.57	25.53
#Test-Dev Samples	45	152	36	46	33	100	42	84	17	19	46	20	144	88

Table 7: Results on CrossSum using BertScore

\uparrow

– Low-Resource Languages .

Model	Train Data	Test Data	R-L $\uparrow$	BertS $\uparrow$
BARThez	Text	Text	29.11	38.90
Cascaded_1	-	Speech	23.43	32.55
Cascaded_2	-	Speech	27.82	38.54
SBARThez BGE	Text	Text	24.64	35.07
	Text	Speech	22.86	32.05
	Speech	Speech	23.99	34.64

Table 8: Results on the DECODA test-set.

Category	Method	R-L $\uparrow$	BertS $\uparrow$
GT Segmentation	-	24.64	35.07
Content-aware Segmentation	VAD	23.51	34.43
Content-aware Segmentation	Spk DIAR	24.18	34.95
Time-driven Segmentation	3sec fixed	23.83	34.23
	5sec fixed	24.42	34.98
	8sec fixed	23.80	34.22

Table 9: Segmentation results with SBARThez-BGE. Evaluation is made on speech segments using different segmentation methods.

Text Summarization

Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.

See all 142 papers

Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Oct 13, 2026Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad AbrarText Summarization Multilingual Language Model Evaluation
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation
Aug 11, 2026Yejin Jeon, Marie Maltais, Virginia Ceccatelli +2Speech Translation Speech Language Models
Mila - Quebec AI Institute · McGill University, Canada · Google DeepMind +1
BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion
Mar 12, 2026Varun Iyer, Cornelia CarageaText Summarization Language Model Decoding
University of Illinois Chicago

Oct 13, 2026J/K move · Enter open · S save

Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar

Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh

Text Summarization Multilingual Language Model Evaluation Summarization Evaluation Low-Resource Language Processing

Open paper AskPDF arXiv

Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.

Abstract

Text Summarization

Explore similar work

Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion