Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation
Authors: Isabel Cachola, William Walden, Reno Kriz, Mark Dredze
Organizations: St. Edward’s University Austin, TX, USA · Johns Hopkins University Human Language Technology Center of Excellence Baltimore, MD, USA · Johns Hopkins University Baltimore, MD, USA
The majority of work on summarization evaluation focuses on general summary quality (e.g., ROUGE, BERTScore) or specific desired properties (e.g., readability, factuality). However, these metrics fail to measure the utility of a summary to an individual user. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor. Query-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician. By contrast, a reader's background or persona (their role and expertise) is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader's needs. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM-as-judge metrics, fail basic perturbation tests of informational content. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person's background and use case. We find that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.
Table 1. We report the expected statistic values (Stat.) by the direction of change (Dir.) for Spearman rank correlation ( ρ ) and Monotonicity (M). See § 3.1 for details.
Figure 1. Annotation workflow example. Steps 1(c) - 1(d) continue for each query assigned to the annotator.
Table 2. Spearman’s ρ (left) and match-rate M (right) across all perturbation tests.
Table 3. α between metrics and human pairwise preferences. Chance agreement corresponds to α=0 . No metric family substantively exceeds chance, including persona-aware variants designed to capture informational satisfaction.
Table 4. Human evaluation results. ( 4(a) ) Per-variant win rates across the 2×2 design. ( 4(b) ) Aggregated personalized-vs-generic preferences by tournament position. ( 4(c) ) Logistic regression of the personalized-side win with cluster-robust standard errors.
Table 5. Per-metric similarity thresholds used in the agreement analysis ( Sections 6 and 3 ). When ∣metric(A)−metric(B)∣ is below the listed threshold, the metric’s decision is counted as “Neither.” Thresholds reflect the typical scale and variance of each metric.
Prompt: LLM Judge
Table 6. Prompt template for LLM Judges. Score rubrics in Table 7 . Source is also provided in prompt for consistency and relevance.
Score Rubrics
Table 7. Score rubrics for prompt in Table 6 .
Prompt: LLM Judge Annotator
Table 8. Prompt template for LLM Judge Annotator, in which the prompt reflects the instructions given to the human annotators. Score rubrics in Table 7 . Source is also provided in prompt for consistency and relevance.
Prompt: Persona Precision
Table 9. Prompts for Persona Precision metric.
Prompt: Persona Recall
Table 10. Prompts for Persona Recall metric.
Prompt: Perturbation Prompts
Table 11. Prompts for the synthetic perturbations tests, described in Section 3 .
Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.
Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval, a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and datasets will be released in GitHub for reproducibility.
Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.
Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA