Summarization Evaluation

Follow this topic Ask about this topic

Summarization evaluation asks whether a short text captures the source’s important content, avoids unsupported claims, and is useful to its intended readers. Approaches range from lexical-overlap metrics such as ROUGE and semantic-similarity scores to factuality checks using questions and answers, evidence tracing, and content-unit coverage. Human ratings and LLM-as-a-judge systems are tested on benchmarks for news, long documents, dialogue, clinical texts, video, and multilingual settings. A central challenge is that fluency or similarity can hide omissions and factual errors, while automated judges may be inconsistent or miss reader-specific needs.

Latest papers

44 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization
Apr 12, 2026Suyoung Bae, CheolWon Na, Jaehoon Lee +3Factual Consistency Evaluation Summarization Evaluation
College of Computing and Informatics, Sungkyunkwan University, South Korea
TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarization
Apr 8, 2026Figen Eğin, Aytuğ OnanVideo Summarization Text Summarization
Department of Computer Engineering, Izmir Katip Celebi University, Cigli, Izmir, Türkiye · Department of Computer Engineering, Izmir Institute of Technology, Urla, Izmir, Türkiye
Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
Apr 7, 2026Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1LLM Evaluation Long-Context Language Model Evaluation
Information Science Cornell University · School of Communication and Culture Aarhus University
RadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings
Mar 24, 2026Sitong Zhou, Meliha Yetisgen, Mari OstendorfText Summarization Clinical Information Extraction
University of Washington
Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring
Mar 2, 2026Aditya Shukla, Yining Yuan, Ben Tamo +7Summarization Evaluation Clinical Language Model Evaluation
Georgia Institute of Technology · Shepherd Center
MOSAIC: Modular Opinion Summarization using Aspect Identification and Clustering
Mar 1, 2026Piyush Kumar Singh, Jayesh ChoudhariText Summarization Summarization Evaluation
Viator, TripAdvisor / London
Iterative Prompt Refinement for Dyslexia-Friendly Text Summarization Using GPT-4o
Feb 26, 2026Samay Bhojwani, Swarnima Kain, Lisong XuText Summarization Summarization Evaluation
University of Nebraska-Lincoln
BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization
Feb 18, 2026Ahmed Rafid, Rumman Adib, Fariya Ahmed +2Factual Consistency Evaluation Summarization Evaluation
Department of Computer Science and Engineering, Islamic University of Technology, Bangladesh · York University, Canada
LIDS: LLM Summary Inference Under the Layered Lens
Feb 18, 2026Dylan Park, Yingying Fan, Jinchi LvSummarization Evaluation LLM Evaluation
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Data Sciences and Operations Department, Marshall School of Business, University of Southern California, Los Angeles, CA 90089
SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics
Jan 22, 2026Silvia Casola, Ryan Soh-Eun Shim, Felicia Körner +2Summarization Evaluation Language Model Steering
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Language Science and Technology, Saarland University, Germany
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Jan 7, 2026Yao Dou, Benjamin Mamut, Wei XuLegal NLP LLM Evaluation
Georgia Institute of Technology
MAPLE: Medical Aspect-Based Summarization with Phrase-Level Evidence
Jan 6, 2026Bohao Chu, Hendrik Damm, Tabea M. G. Pakull +5Healthcare Text Summarization
University of Duisburg-Essen · University of Applied Sciences and Arts Dortmund · University Hospital Essen +1
VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
May 23, 2025Shenghui Chen, Po-han Li, Sandeep Chinchali +1Information Bottleneck Video Summarization
The University of Texas at Austin
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
Mar 21, 2025Jeremy Barnes, Naiara Perez, Alba Bonet-Jover +1Summarization Evaluation Automated Evaluation
HiTZ Center, University of the Basque Country (UPV/EHU) · Department of Software and Computing Systems, University of Alicante, Spain · GOI Institute, Basque Summer University (UEU)

Apr 12, 2026J/K move · Enter open · S save

ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

Suyoung Bae, CheolWon Na, Jaehoon Lee, Yumin Lee+2

College of Computing and Informatics, Sungkyunkwan University, South Korea

Factual Consistency Evaluation Summarization Evaluation Automated Evaluation Software Engineering Benchmarks

Open paper AskPDF arXiv

As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.