cs.CLNov 4, 2026

SAUCE: Summary Analysis Using Conversation Entailment

Authors: Man-Ling Sung, Hemanth Kandula, Jeff Ma, William Hartmann, Matthew Snover

Organizations: RTX BBN Technologies, Cambridge MA, USA

Abstract

With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization

    Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3LLM EvaluationLong-Context Language Model Evaluation

  2. Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

    May 17, 2026Kaavya Chaparala, Thomas Thebaud, Jesús Villalba López +3Benchmark Construction

  3. Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

    Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM EvaluationText Summarization