Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation
Authors: Isabel Cachola, William Walden, Reno Kriz, Mark Dredze
Organizations: St. Edward’s University Austin, TX, USA · Johns Hopkins University Human Language Technology Center of Excellence Baltimore, MD, USA · Johns Hopkins University Baltimore, MD, USA
The majority of work on summarization evaluation focuses on general summary quality (e.g., ROUGE, BERTScore) or specific desired properties (e.g., readability, factuality). However, these metrics fail to measure the utility of a summary to an individual user. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor. Query-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician. By contrast, a reader's background or persona (their role and expertise) is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader's needs. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM-as-judge metrics, fail basic perturbation tests of informational content. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person's background and use case. We find that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.
Table 1. We report the expected statistic values (Stat.) by the direction of change (Dir.) for Spearman rank correlation ( ρ ) and Monotonicity (M). See § 3.1 for details.
Figure 1. Annotation workflow example. Steps 1(c) - 1(d) continue for each query assigned to the annotator.
Table 2. Spearman’s ρ (left) and match-rate M (right) across all perturbation tests.
Table 3. α between metrics and human pairwise preferences. Chance agreement corresponds to α=0 . No metric family substantively exceeds chance, including persona-aware variants designed to capture informational satisfaction.
Table 4. Human evaluation results. ( 4(a) ) Per-variant win rates across the 2×2 design. ( 4(b) ) Aggregated personalized-vs-generic preferences by tournament position. ( 4(c) ) Logistic regression of the personalized-side win with cluster-robust standard errors.
Table 5. Per-metric similarity thresholds used in the agreement analysis ( Sections 6 and 3 ). When ∣metric(A)−metric(B)∣ is below the listed threshold, the metric’s decision is counted as “Neither.” Thresholds reflect the typical scale and variance of each metric.
Prompt: LLM Judge
Table 6. Prompt template for LLM Judges. Score rubrics in Table 7 . Source is also provided in prompt for consistency and relevance.
Score Rubrics
Table 7. Score rubrics for prompt in Table 6 .
Prompt: LLM Judge Annotator
Table 8. Prompt template for LLM Judge Annotator, in which the prompt reflects the instructions given to the human annotators. Score rubrics in Table 7 . Source is also provided in prompt for consistency and relevance.
Prompt: Persona Precision
Table 9. Prompts for Persona Precision metric.
Prompt: Persona Recall
Table 10. Prompts for Persona Recall metric.
Prompt: Perturbation Prompts
Table 11. Prompts for the synthetic perturbations tests, described in Section 3 .
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA