Summarization Evaluation

Follow this topic Ask about this topic

Summarization evaluation asks whether a short text captures the source’s important content, avoids unsupported claims, and is useful to its intended readers. Approaches range from lexical-overlap metrics such as ROUGE and semantic-similarity scores to factuality checks using questions and answers, evidence tracing, and content-unit coverage. Human ratings and LLM-as-a-judge systems are tested on benchmarks for news, long documents, dialogue, clinical texts, video, and multilingual settings. A central challenge is that fluency or similarity can hide omissions and factual errors, while automated judges may be inconsistent or miss reader-specific needs.

Latest papers

44 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

Abstracts or full text? Evaluating input scope and computational efficiency in biomedical lay summarization
Oct 15, 2026Egecan Çelik Evgin, İlknur Karadeniz, Olcay Taner YıldızText Summarization Summarization Evaluation
Department of Computer Science, Özyeğin University, İstanbul, Türkiye · Department of Computer Engineering, Galatasaray University, İstanbul, Türkiye · Department of Artificial Intelligence and Data Engineering, Özyeğin University, İstanbul, Türkiye
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Oct 13, 2026Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad AbrarText Summarization Multilingual Language Model Evaluation
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers
Oct 7, 2026Nadhem Zmandar, Mo El-Haj, Paul RaysonText Summarization Summarization Evaluation
UCREL NLP group, School of Computing and Communications, Lancaster University, UK
Interpreting at Write Time: A Policy Ablation for Multi-Goal Agent Memory
Oct 2, 2026Albert Sadowski, Jarosław A. ChudziakAgent Memory Management Long-Term Conversational Memory
Faculty of Electronics and Information Technology Warsaw University of Technology Warsaw, Poland
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Oct 2, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +4LLM-as-a-Judge Automated Evaluation
DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Faculty of Computer Science, Goethe University Frankfurt +1
Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening
Sep 30, 2026Thilo Tamme, Anton Hantel, Bijan Khosrawi-RadText Summarization Summarization Evaluation
Technical University of Munich · Massachusetts Institute of Technology · Leuphana University Lüneburg
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
Sep 27, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaHallucination in Language Models Factual Consistency Evaluation
International Institute of Information Technology Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India
Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study
Sep 24, 2026Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh +4Text Summarization Summarization Evaluation
Department of Computer Science, South Campus, University of Kashmir, Anantnag, India · IBM Research, San Jose, CA, USA · Manipal University Jaipur, Dehmi Kalan, Jaipur 303007, Rajasthan, India
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM Evaluation Text Summarization
ServiceNow Canada
Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation
Sep 1, 2026Himil Vasava, Ming JiangLLM-as-a-Judge Language Model Generation Evaluation
University of Wisconsin-Madison
Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric
Aug 21, 2026Sami Shames El Deen, Mariette AwadArabic Language Models Text Summarization
American University of Beirut
Assessing Quality of Experience in Natural Language Generation of German Text
Aug 19, 2026Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz +1Language Model Generation Evaluation Generative AI Evaluation
Technische Universität Berlin, Berlin, Germany. · Speech and Language Technology Lab, German Research Center for Artificial Intelligence (DFKI), Berlin, Germany.
Retrieved-Span Training for Efficient Query-Focused Meeting Summarization on QMSum
Aug 18, 2026Edward Xi YangText Summarization Summarization Evaluation
Ertas AI · Ertas AI Melbourne, Australia
The Plot Thins: Uniformity and Linearity in Literary Summaries
Aug 18, 2026Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1Text Summarization Computational Literary Studies
Computer Science Department, Cornell University · Information Science Department, Cornell University · School of Communication and Culture, Aarhus University
Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation
Aug 14, 2026Isabel Cachola, William Walden, Reno Kriz +1Summarization Evaluation LLM Evaluation
St. Edward’s University Austin, TX, USA · Johns Hopkins University Human Language Technology Center of Excellence Baltimore, MD, USA · Johns Hopkins University Baltimore, MD, USA
Averaging Bias: Human Faithfulness Annotations are not Locally Faithful
Jul 31, 2026Huajian Zhang, Yiyang Feng, Jiawei ZhouFactual Consistency Evaluation Summarization Evaluation
Stony Brook University
Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizers
Jul 23, 2026Vasudha Bhatnagar, Purnima Bindal, Vikas Kumar +1LLM Evaluation Text Summarization
Department of Computer Science, University of Delhi, Delhi, 110007, Delhi, India · Department of Statistics, Ramjas College, University of Delhi, Delhi, 110007, Delhi, India
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Jul 12, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaText Summarization Summarization Evaluation
Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India
A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3LLM Evaluation Long-Context Language Model Evaluation
CCSE, Beihang University · Fanyu AI Laboratory · MAIS, CASIA
Summarization is Not Dead Yet
Jun 6, 2026Dongqi Liu, Chenxi Whitehouse, Zheng Zhao +3LLM Evaluation Text Summarization
Saarland University, Max Planck Institute for Informatics · Tencent YouTu Research · University of Cambridge +2
Thinking Through Signs: PEEL as a Semiotic Scaffolding for Epistemically Accountable AI-Enabled Research
Jun 2, 2026Clarisse de Souza, Gabriel Barbosa, Simone Diniz Junqueira Barbosa +3LLM Evaluation AI Accountability
PUC-Rio, PUC-Behring Institute of Artificial Intelligence, Brazil · PUC-Rio, Brazil
Plans for Evaluating Structured Generative Search Summaries
May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation Evaluation Summarization Evaluation
Waseda University/Naver Corporation · Tokyo, Japan · Waseda University +2
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemLLM Evaluation Multimodal Large Language Models
School of Computing, Macquarie University, Sydney, Australia
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM Evaluation LLM Self-Refinement
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2Language Model Generation Evaluation Factual Consistency Evaluation
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering
Apr 27, 2026Woojun Jung, Junyeong KimVideo Summarization Summarization Evaluation
Department of Artificial Intelligence, Chung-Ang University
SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization
Apr 21, 2026Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu KaoLearning to Rank Text Summarization
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Artificial Intelligence Research Center, Chang Gung University +1
Calibrating Model-Based Evaluation Metrics for Summarization
Apr 19, 2026Hongye Liu, Dhanajit Brahma, Ricardo HenaoLanguage Model Calibration Summarization Evaluation
Duke University
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
Development, Evaluation, and Deployment of a Multi-Agent System for Thoracic Tumor Board
Apr 14, 2026Tim Ellis-Caleo, Timothy Keyes, Nerissa Ambers +5Text Summarization Summarization Evaluation
Division of Oncology, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA · Technology and Digital Solutions, Stanford Health Care, Palo Alto, CA · Department of Biomedical Data Science, Stanford University School of Medicine, Palo Alto, CA +4

Oct 15, 2026J/K move · Enter open · S save

Abstracts or full text? Evaluating input scope and computational efficiency in biomedical lay summarization

Egecan Çelik Evgin, İlknur Karadeniz, Olcay Taner Yıldız

Department of Computer Science, Özyeğin University, İstanbul, Türkiye · Department of Computer Engineering, Galatasaray University, İstanbul, Türkiye · Department of Artificial Intelligence and Data Engineering, Özyeğin University, İstanbul, Türkiye

Text Summarization Summarization Evaluation LLM Fine-Tuning

Open paper AskPDF arXiv

Biomedical research articles are mainly written for experts and often contain terms that are difficult for the general public to understand. Automated lay summarization has become a solution to make biomedical knowledge easier for patients and the public to understand. However, generating lay summaries from full-text articles requires more computing power because the articles are long and costly to process. Fine-tuning large language models for this task also requires substantial resources. This study asks two questions: (1) whether abstracts contain enough information to produce lay summaries as good as those based on full-text articles, and (2) whether few-shot learning performs as well as task-specific fine-tuning. The study used two public biomedical lay summarization datasets and two recent models, evaluated for readability, factual consistency, semantic similarity, and runtime efficiency. Results show that abstract-based summarization performs better overall. Across both datasets and models, abstract fine-tuning is 4.70 times faster, while abstract zero- and few-shot prompting is 5.31 times faster than full-text methods. These results show that abstract-based prompting reduces runtime cost while maintaining lay summary quality.