Overview

Summarization evaluation asks whether a short text captures the source’s important content, avoids unsupported claims, and is useful to its intended readers. Approaches range from lexical-overlap metrics such as ROUGE and semantic-similarity scores to factuality checks using questions and answers, evidence tracing, and content-unit coverage. Human ratings and LLM-as-a-judge systems are tested on benchmarks for news, long documents, dialogue, clinical texts, video, and multilingual settings. A central challenge is that fluency or similarity can hide omissions and factual errors, while automated judges may be inconsistent or miss reader-specific needs.

Latest papers

CardsList

48 papers4 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Discharge Summary Generation Is Not Just Summarization: A Reality Check for Evaluation

    Nov 10, 2026Radhika Dua, Frederick Wolff, Kyunghyun Cho +1Summarization EvaluationMedical Report Generation

  2. Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization

    Nov 6, 2026Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj +5Confidence Estimation in Language ModelsSummarization Evaluation

  3. SAUCE: Summary Analysis Using Conversation Entailment

    Nov 4, 2026Man-Ling Sung, Hemanth Kandula, Jeff Ma +2Summarization EvaluationAutomated Evaluation

  4. What Breaks When Facts Become Stories? The Distortions Source-Agreement Metrics Miss When Models Retell Real Events

    Oct 29, 2026Kwangsuk Park, JiWoong Yang, Vaniushkina KseniiaFactual Consistency EvaluationNarrative Generation

  5. Abstracts or full text? Evaluating input scope and computational efficiency in biomedical lay summarization

    Oct 15, 2026Egecan Çelik Evgin, İlknur Karadeniz, Olcay Taner YıldızText SummarizationSummarization Evaluation

  6. Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

    Oct 13, 2026Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad AbrarText SummarizationMultilingual Language Model Evaluation

  7. A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

    Oct 7, 2026Nadhem Zmandar, Mo El-Haj, Paul RaysonText SummarizationSummarization Evaluation

  8. Interpreting at Write Time: A Policy Ablation for Multi-Goal Agent Memory

    Oct 2, 2026Albert Sadowski, Jarosław A. ChudziakAgent Memory ManagementLong-Term Conversational Memory

  9. Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

    Oct 2, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +4LLM-as-a-JudgeAutomated Evaluation

  10. Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening

    Sep 30, 2026Thilo Tamme, Anton Hantel, Bijan Khosrawi-RadText SummarizationSummarization Evaluation

  11. CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

    Sep 27, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaHallucination in Language ModelsFactual Consistency Evaluation

  12. Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study

    Sep 24, 2026Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh +4Text SummarizationSummarization Evaluation

  13. Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

    Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM EvaluationText Summarization

  14. Assessing Quality of Experience in Natural Language Generation of German Text

    Aug 19, 2026Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz +1Language Model Generation EvaluationGenerative AI Evaluation

  15. The Plot Thins: Uniformity and Linearity in Literary Summaries

    Aug 18, 2026Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1Text SummarizationComputational Literary Studies

  16. Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

    Aug 14, 2026Isabel Cachola, William Walden, Reno Kriz +1Summarization EvaluationLLM Evaluation

  17. Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizers

    Jul 23, 2026Vasudha Bhatnagar, Purnima Bindal, Vikas Kumar +1LLM EvaluationText Summarization

  18. Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

    Jul 12, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaText SummarizationSummarization Evaluation

  19. A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization

    Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3LLM EvaluationLong-Context Language Model Evaluation

  20. Summarization is Not Dead Yet

    Jun 6, 2026Dongqi Liu, Chenxi Whitehouse, Zheng Zhao +3LLM EvaluationText Summarization

  21. Thinking Through Signs: PEEL as a Semiotic Scaffolding for Epistemically Accountable AI-Enabled Research

    Jun 2, 2026Clarisse de Souza, Gabriel Barbosa, Simone Diniz Junqueira Barbosa +3LLM EvaluationAI Accountability

  22. Plans for Evaluating Structured Generative Search Summaries

    May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation EvaluationSummarization Evaluation

  23. Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity

    May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemLLM EvaluationMultimodal Large Language Models

  24. LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation

    Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM EvaluationLLM Self-Refinement

  25. LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

    Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2Language Model Generation EvaluationFactual Consistency Evaluation