Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric

Sami Shames El Deen; Mariette Awad

cs.CLAug 21, 2026

Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric

Authors: Sami Shames El Deen, Mariette Awad

Organizations: American University of Beirut

Abstract

In this research, we introduce SAraBERT, an enhanced version of AraBERT which proposes inter-sentence transformer layers for extractive summarization tasks. To ensure that the summaries generated by SAraBERT achieve a high coverage of the document's main ideas, we propose Semantic Siamese Similarity, a novel evaluation metric that measures the level of similarity between two text inputs. We validated using BLEU, ROUGE, and Semantic Siamese similarity on Sarabert and published related models. Simulation results showed the effectiveness of our proposed model and motivate follow on research.

Figures & tables

Fig. 1 : Overview for the architecture of the original BERT model

Fig. 2 : Overview for the architecture of the SAraBERT model

Reference	ليللا يف ماعطلا لكأي يلع	ROUGE	Cosine Similarity
Candidate	ءاسم ماعطلا لوانتي وه	0.0	0.9074

TABLE I : An example comparing the ROUGE score and Cosine Similarity

Reference	ةحافت دلولا لوانت	Cos_Sim	L2 norm
Candidate 1	ةطق لفطلا لكأ	0.873	5.989
Candidate 2	ةهكاف لفطلا لكأ	0.867	5.686

TABLE II : An example showing the cosine similarity and L2 norm when comparing a reference to predictions

ةحافت دلولا لوانت	Cos_Sim	Rouge	L2 norm	FreqDiff
ةحافت لفطلا لكأ	0.957	0.222	3.637	1.000
لوانت لوانت ةحافت ةحافت دلولا	0.930	0.889	5.005	1.667

TABLE III : An example showing the cosine similarity, ROUGE score, L2 norm, and frequency difference when comparing a reference to predictions

SSS score	Interpretation
0-0.1	no clear resemblance or no similarity at all
0.1-0.2	the gist is clear, but not all context is covered
0.2-0.5	context is significantly covered using different words
$\geq$ 0.5	exact replica with slight modifications

TABLE IV : interpretation of SSS scores

	BLEU	METEOR	Duration (Docs/minute)
GoogleTrans	0.413 $\pm$ 0.386	0.24	30
mBart	0.413 $\pm$ 0.382	0.25	20

TABLE V : Translation Experiment Results

Fig. 3 : Sample English passage translated to Arabic and Diactritized

Metric	Value (Mean $\pm$ std)
OSMAN - Diacritics	78.45 $\pm$ 5.18
OSMAN + Diacritics	70.28 $\pm$ 8.95

TABLE VI : Average Readability Measures for Arabic

Fig. 4 : Histogram of readability comparisons between english and arabic with and without diacritics

Model	bleu_1	bleu_2	bleu_3	bleu_4	avg	max
SAraBert + Classifier	0.408	0.385	0.38	0.376	0.387	0.99
SAraBert + RNN	0.474	0.45	0.445	0.441	0.452	0.99
SAraBert + Transformer	0.391	0.368	0.364	0.36	0.371	0.99
AraBert Embeddings + K-Means	0.284	0.254	0.249	0.246	0.258	0.991
AraVec + K-Means	0.345	0.334	0.331	0.328	0.335	0.973
Bag of Words	0.361	0.351	0.348	0.345	0.352	0.98

TABLE VII : BLEU scores over Kalimat Dataset

Model	rouge_1	rouge_2	rouge_L	avg	min	max
SAraBert + Classifier	0.529	0.466	0.51	0.502	0.008	0.99
SAraBert + RNN	0.588	0.524	0.568	0.56	0.008	0.99
SAraBert + Transformer	0.512	0.444	0.491	0.482	0.017	0.99
AraBert Embeddings + K-Means	0.445	0.366	0.427	0.414	0.008	1.0
AraVec + K-Means	0.545	0.49	0.535	0.523	0.0	1.1
Bag of Words	0.47	0.415	0.46	0.448	0.0	1.1

TABLE VIII : ROUGE scores over Kalimat Dataset

Model	BLEU	ROUGE	Siamese Similarity	BS	TS-SS
Oracle Score	0.532 ± 0.27	0.699 ± 0.23	0.326 ± 0.3	0.83	0.003
SAraBert + Classifier	0.387 ± 0.3	0.502 ± 0.31	0.211 ± 0.158	0.76	0.018
SAraBert + RNN	0.452 ± 0.2	0.560 ± 0.26	0.282 ± 0.18	0.79	0.017
SAraBert + Transformer	0.371 ± 0.3	0.482 ± 0.31	0.240 ± 0.17	0.77	0.019
AraBert Embeddings + K-Means	0.258 ± 0.21	0.414 ± 0.25	0.216 ± 0.1	0.76	0.032
AraVec + K-Means	0.335 ± 0.28	0.523 ± 0.28	0.191 ± 0.18	0.74	0.039

TABLE IX : Results of different models under several evaluation metrics

Fig. 5 : Pearson Correlation between the similarity metrics

ID	Text
0	ةحافت ناك هماعط .ءاثالثلا موي احابص ةسردملا يف هماعط يماس ذيملتلا لكأ
1	بره
2	لكأ
3	ةحافت ناك هماعط .ءاثالثلا موي احابص ةسردملا يف هماعط يماس ذيملتلا لكأ
4	ةحافت ناك هماعط .ءاثالثلا موي احابص ةسردملا يف هماعط يماس ذيملتلا لوانت
5	ةحافت ناك هماعط .ءاثالثلا موي احابص ةسردملا يف هماعط يماس لوانت

TABLE X : List of a reference sentence (1) and set of candidate sentences(2 to 19)

Cosine

ROUGE

norm 2

SSS

TABLE XI : Results of computing SSS and BertScore (BS) along with other required computations for the candidates in Table[ X ] linked by ID

Text Summarization

Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.

See all 140 papers

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Jul 12, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaText Summarization Summarization Evaluation
Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM Evaluation Text Summarization
ServiceNow Canada
Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model Architectures Text Summarization
Independent Researcher · University of Memphis

Jul 12, 2026J/K move · Enter open · S save

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

Text Summarization Summarization Evaluation Automated Evaluation

Open paper AskPDF arXiv

Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.

Abstract

Text Summarization

Explore similar work

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios