cs.CLFeb 18, 2026

BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization

Authors: Ahmed Rafid, Rumman Adib, Fariya Ahmed, Ajwad Abrar, Mohammed Saidul Islam

Organizations: Department of Computer Science and Engineering, Islamic University of Technology, Bangladesh · York University, Canada

Abstract

Evaluating factual consistency is essential for reliable text summarization, particularly in high-stakes domains such as healthcare and news. However, most existing evaluation metrics overlook Bangla, a widely spoken yet under-resourced language, and often depend on reference summaries. We introduce BanglaSummEval, a reference-free, question-answering-based framework for evaluating factual consistency in Bangla summarization. The proposed method assesses both factual accuracy and content coverage through automatically generated questions and answers derived from the source document and the summary. A single multilingual instruction-tuned language model handles question generation, question answering, candidate answer extraction, and question importance weighting. This unified design reduces system complexity and computational cost. To capture semantic consistency beyond surface-level overlap, we use BERTScore-Recall for answer comparison. We validate BanglaSummEval on 300 human-written summaries from educational and medical domains, demonstrating strong correlation with expert human judgments (Pearson's r=0.694r = 0.694, Spearman's ρ=0.763ρ= 0.763). By providing interpretable, step-wise diagnostics alongside reliable evaluation scores, BanglaSummEval offers a practical and transparent solution for factual consistency evaluation in low-resource language settings.

Explore similar work

CardsList
  1. Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

    Oct 13, 2026Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad AbrarText SummarizationMultilingual Language Model Evaluation

  2. Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

    May 26, 2026Yuxuan Ye, Raul Santos-Rodriguez, Edwin SimpsonFactual Consistency EvaluationText Summarization

  3. ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

    Apr 12, 2026Suyoung Bae, CheolWon Na, Jaehoon Lee +3Factual Consistency EvaluationSummarization Evaluation