With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
Figures & tables
Figure 1: Workflow of the SAUCE. The user can access the fact-text pairs to understand the missing or wrong information in the summary with corresponding faithfulness and coverage scores.
B: Oh no, I can remember my mother getting in trouble if, you know, one of was sick, and I know she probably didn’t make hardly anything, you know, compared to the work that she did.
A plays on MURPHY BROWN
A: He, he plays [laughter] on MURPHY BROWN.
Table 2: Examples of non-entailed pairs that are mis-classified as entailed. The first row confuses factual and hypothetical scenarios, which is also confusing to humans. Second row confuses the subject, the speaker vs. the person indicated by ‘he’.
Figure 2: Mean WER and BLEU scores of each language, and their summary scores on ground truth, MT and ASR+MT English transcriptions. We present the plots in orange for SAUCE, blue for NLI-based metrics, purple for LLM-as-judge metrics.
Summary from human translation (SAUCE Faithful : 0.94, SAUCE Coverage : 0.6, SummaC Conv : 0.6, SummaC zs : 0.47)
In the conversation, two individuals discuss various (a) personal matters involving health, family, and home renovations . One of them recently visited Joana to deliver some clothes, and discovered that Isa was sick with a cold. (b) Carlos, who drove the speaker, received no assistance during the visit. The conversation reveals that there isn’t much improvement in Isa’s condition, but there is a hope for becoming more independent. (c) Ernesto, a family member, has bought a car and is learning to drive , bringing some cheerful news. The family is also involved in home improvements, including adding a store and planting grass, enhancing their surroundings. However, the speakers acknowledge some ongoing challenges, emphasizing the importance of gratitude and support, especially for Carlos, who is not in a good place mentally. (d) Carlos’ surgery was postponed due to his cold and allergies, reminiscent of one speaker’s past experience with an adverse reaction to antibiotics, leading to a discussion about the risks of self-medication. …
In a conversation between two individuals, they discuss various topics, including ( a) personal encounters, family news, and health concerns. One participant mentions visiting Joana to drop off clothes and finding Isa unwell due to a cold. (b) There’s a discussion about Carlos, who seems to be undergoing challenges but not receiving much attention. The conversation shifts to family updates, (c) mentioning Ernesto’s new car purchase and driving lessons , as well as home renovations being done by several friends and family members like Cristi and Gaby. The tone reflects a mixture of envy and gratitude, emphasizing the importance of being thankful despite current hardships. They highlight the significance of family unity, particularly regarding Carlos’s need for support amidst his health issues. (d) Carlos is recovering but has had an operation postponed to November due to sinusitis-like symptoms. …
In the conversation, the speakers discuss (a) various personal and familial updates. Speaker A mentions visiting Joana to drop off clothes and encountering Isa, who is sick with a cold. Isa’s condition and the (b) lack of interest shown by Carlos, who accompanied A, are highlighted. The speakers discuss the need for A to reconsider certain decisions, but A resolutely rejects this notion, emphasizing independence and familial unity as priorities. The conversation shifts to news about (c) Ernesto planning to purchase a car , while renovations are underway on friends’ and family’s houses. B praises these efforts but expresses frustration with perceived stagnation in their situation. The speakers highlight gratitude towards God for their inner wellbeing and unity, despite external challenges. A underscores the importance of supporting (d) Carlos, particularly due to his health issues, which include a postponed surgery due to allergies. …
Table 3: Summaries and scores on Spanish conversation (sp_1578) on ground truth, and with MT, ASR+MT error. Sentences related to the same set of facts are colored and labeled with corresponding alphabets.
Metric
SAUCE Faithful
SAUCE Coverage
SummaC conv
SummaC zs
Kendall rank coef.
0.153
0.677
-0.266
0.008
Table 4: Tau coefficient of metrics on ground truth and ASR+MT conversations. Assumes summaries generated from ground truth are better than summaries generated from ASR+MT output.
Figure 3: Example of broken summary.
Metrics
SummaC Conv
SummaC ZS
SAUCE Faithful
SAUCE Coverage
Spanish norm
0.641
0.307
0.941
0.358
Spanish broke
0.598
0.104
0.356
0.000
Arabic norm
0.221
0.395
0.980
0.363
Table 5: Metrics on Eval sets. Spanish norm and Arabic norm are normal summaries, Spanish broke is broken summaries.
Figure 4: SAUCE Faithful and SAUCE Coverage on 4 LLMs across 4 languages translated into English.
Coef.
SAUCE F
SAUCE C
SC Conv
SC ZS
Acc.
0.259
0.203
0.228
0.139
Inf.
0.185
0.475
0.088
0.028
Table 6: Tau coefficient on human ranking of summaries compared with metrics. Notations as follow: Coef. - Kendall rank coefficient, SAUCE F - SAUCE Faithful , SAUCE C - SAUCE Coverage , SC - SummaC, Acc - Accuracy, Inf. - Informative.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Arabic
Mandarin
Russian
Spanish
Metric
Sub-metric
GT
MT
ASR
GT
MT
ASR
GT
MT
ASR
GT
MT
ASR
SummaC
SummaC Conv
0.237
0.230
0.226
0.212
0.213
0.212
0.217
0.217
0.213
0.758
0.769
0.789
SummaC ZS
0.385
0.363
0.364
0.298
0.305
0.280
0.318
0.312
0.327
0.325
0.313
0.326
AVG
0.311
0.296
0.295
0.255
0.259
0.246
0.268
0.264
0.270
0.541
0.541
0.558
G-EVAL
coherence
0.810
0.800
0.755
0.890
0.870
0.857
0.929
0.900
0.883
0.920
0.894
0.870
consistency
0.875
0.735
0.620
0.907
0.747
0.660
0.925
0.825
0.771
0.902
0.812
0.736
Appendix
Table 7: Full Benchmark Results for Cross-Lingual Conversational Summarization Metrics Under Varied Noise Conditions. Scores evaluate summary quality across Ground Truth (GT), Machine Translation (MT), and cascaded ASR-MT inputs, highlighting the robustness of SAUCE (Faithfulness and Coverage) compared to existing baselines.
Language
ASR (WER)
MT (BLEU)
ASR+MT(BLEU)
Arabic (avg)
0.497
0.314
0.211
CallHome dev
0.507
0.311
0.233
CallHome test
0.487
0.317
0.189
Mandarin (avg)
0.341
0.249
0.177
HUB5 dev
0.349
0.261
0.178
HUB5 test
0.333
0.237
0.177
Appendix
Table 8: WER and BLEU scores for each language on average, and their corresponding dev and test sets.
Subset of facts extracted from the conversation
Facts (a): Overview of conversation.
1. Carlos cannot undergo surgery until his health improves.
2. Someone had a total body allergy from pills Cristi bought.
3. Someone is allergic to antibiotics and cannot self-medicate.
4. A family is fixing their house and making it bigger.
5. The family is going to put a store in the house.
Appendix
Table 9: Example of Facts extracted Spanish conversation (sp_1578) on ground truth translation, grouped by the relevant topics. The colors and alphabets scheme aligns with Table 3 .
Arabic conversation (ar_4264 )
Fact 1.1: Ashraf is learning baseball, TUI, karate, and basketball at the summer school.
Fact 1.2: A decided to stop swimming lessons for the summer.
Related conversation line: A: well I stopped it in Summer because I go to swim with these Muslim sisters
GPT: A shares updates on Ashraf’s activities in a summer program learning various sports .
Mistral: A also mentioned that their son, Ahmad , is doing well and attending a summer school program where he is learning baseball, TUI, karate, basketball, and possibly karate.
DeepSeek: Updates on family members follow, including Ashraf attending summer school for activities like baseball and karate before the regular school term starts, while swimming lessons are paused .
Appendix
Table 10: Chunks of summaries with their corresponding facts on one Arabic conversation (ar_4264) and Mandarin conversation (ma_4249) on GPT, Mistral7B, DeepSeek-R1 and DeepSeek-R1-distil-qwen-14B. Blue indicates the information that are aligned with facts, red indicates incorrect information, orange indicates non-factual interpretation.
Despite the significant advancement of LLMs in conversation summarization, their evaluation remains limited by insufficient scenarios, input lengths, and sample sizes. Furthermore, existing benchmarks often omit frontier reasoning systems and efficient small models, or lack fine-grained, multi-dimensional assessments. To bridge these gaps, we propose OmniCSEval, a unified benchmark comprising 1,800 diverse conversations across six real-world scenarios, featuring context lengths ranging from 128 to 32k tokens. For fine-grained evaluation, we employ a bidirectional fact-checking framework that integrates key fact matching to assess completeness and conciseness, alongside summary fact verification to evaluate faithfulness. To ensure reliable assessment, we establish a human-LLM collaborative pipeline for key fact extraction and a multi-LLM consensus verifier for summary fact decomposition. Leveraging this framework, we evaluate 28 LLMs across four distinct categories grouped by reasoning capability and model scale. Our extensive empirical study reveals critical insights regarding the cross-scenario challenges current LLMs continue to face, the impacts of reasoning and scale, and the efficiency and adaptability of reasoning models. We also provide guidance for system selection in real-world deployments.
Weixiao Zhou, Gengyao Li, Xianfu Cheng +3
CCSE, Beihang University · Fanyu AI Laboratory · MAIS, CASIA
There are not enough established benchmarks for the task fo speech summarization. Creating new benchmarks demands human annotation, as LLMs could embed systemic errors and bias into datasets. We test ten annotation workflows varying input modality (audio, transcript, or both) and the inclusion of editing (self or peer-editing) to investigate potential quality tradeoffs from using human annotators to summarize audio. We compare human audio-based summaries to human transcript-based summaries to track the impact of the different information modalities on summary quality. We also compare the human outputs against four LLM benchmarks (three text, one audio) to examine whether human-written summaries are less informative than highly fluent automated outputs. We find that audio-based summaries are less informative and more compressed than transcript summaries. However, iterative peer-editing with audio mitigates this difference, enabling audio-based summaries to be as informative as their transcript counterparts and LLM summaries. These findings validate iterative peer-editing among human annotators for the creation of benchmarks informed by both lexical and prosodic information. This enables crucial dataset collection even in setting where transcripts are unavailable.
Kaavya Chaparala, Thomas Thebaud, Jesús Villalba López +3
Johns Hopkins University 3400 N. Charles St., Baltimore, MD 21218
Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.