SAUCE: Summary Analysis Using Conversation Entailment
Organizations: RTX BBN Technologies, Cambridge MA, USA
Abstract
With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
Figures & tables
| Selected chunk | Full conversation | |||||
|---|---|---|---|---|---|---|
| Thres | 0.9 | 0.95 | 0.98 | 0.9 | 0.95 | 0.98 |
| All | 77.0 | 79.2 | 77.0 | 68.1 | 71.4 | 74.2 |
| NoEnt. | 63.6 | 71.6 | 77.1 | 39.7 | 49.9 | 61.4 |
| Ent. | 88.4 | 85.7 | 80.0 | 92.2 | 89.7 | 85.0 |
| Extracted Fact | Chunked conversation |
|---|---|
| B’s mother got in trouble for getting sick | B: Oh no, I can remember my mother getting in trouble if, you know, one of was sick, and I know she probably didn’t make hardly anything, you know, compared to the work that she did. |
| A plays on MURPHY BROWN | A: He, he plays [laughter] on MURPHY BROWN. |
| Summary from human translation (SAUCE Faithful : 0.94, SAUCE Coverage : 0.6, SummaC Conv : 0.6, SummaC zs : 0.47) |
|---|
| In the conversation, two individuals discuss various (a) personal matters involving health, family, and home renovations . One of them recently visited Joana to deliver some clothes, and discovered that Isa was sick with a cold. (b) Carlos, who drove the speaker, received no assistance during the visit. The conversation reveals that there isn’t much improvement in Isa’s condition, but there is a hope for becoming more independent. (c) Ernesto, a family member, has bought a car and is learning to drive , bringing some cheerful news. The family is also involved in home improvements, including adding a store and planting grass, enhancing their surroundings. However, the speakers acknowledge some ongoing challenges, emphasizing the importance of gratitude and support, especially for Carlos, who is not in a good place mentally. (d) Carlos’ surgery was postponed due to his cold and allergies, reminiscent of one speaker’s past experience with an adverse reaction to antibiotics, leading to a discussion about the risks of self-medication. … |
| Summary from MT (SAUCE Faithful : 0.82, SAUCE Coverage : 0.44, SummaC Conv : 0.74, SummaC zs : 0.43) |
| In a conversation between two individuals, they discuss various topics, including ( a) personal encounters, family news, and health concerns. One participant mentions visiting Joana to drop off clothes and finding Isa unwell due to a cold. (b) There’s a discussion about Carlos, who seems to be undergoing challenges but not receiving much attention. The conversation shifts to family updates, (c) mentioning Ernesto’s new car purchase and driving lessons , as well as home renovations being done by several friends and family members like Cristi and Gaby. The tone reflects a mixture of envy and gratitude, emphasizing the importance of being thankful despite current hardships. They highlight the significance of family unity, particularly regarding Carlos’s need for support amidst his health issues. (d) Carlos is recovering but has had an operation postponed to November due to sinusitis-like symptoms. … |
| Summary from ASR + MT (SAUCE Faithful : 0.84, SAUCE Coverage : 0.2, SummaC Conv : 0.54, SummaC zs : 0.42) |
| In the conversation, the speakers discuss (a) various personal and familial updates. Speaker A mentions visiting Joana to drop off clothes and encountering Isa, who is sick with a cold. Isa’s condition and the (b) lack of interest shown by Carlos, who accompanied A, are highlighted. The speakers discuss the need for A to reconsider certain decisions, but A resolutely rejects this notion, emphasizing independence and familial unity as priorities. The conversation shifts to news about (c) Ernesto planning to purchase a car , while renovations are underway on friends’ and family’s houses. B praises these efforts but expresses frustration with perceived stagnation in their situation. The speakers highlight gratitude towards God for their inner wellbeing and unity, despite external challenges. A underscores the importance of supporting (d) Carlos, particularly due to his health issues, which include a postponed surgery due to allergies. … |
| Metric | SAUCE Faithful | SAUCE Coverage | SummaC conv | SummaC zs |
|---|---|---|---|---|
| Kendall rank coef. | 0.153 | 0.677 | -0.266 | 0.008 |
| Metrics | SummaC Conv | SummaC ZS | SAUCE Faithful | SAUCE Coverage |
|---|---|---|---|---|
| Spanish norm | 0.641 | 0.307 | 0.941 | 0.358 |
| Spanish broke | 0.598 | 0.104 | 0.356 | 0.000 |
| Arabic norm | 0.221 | 0.395 | 0.980 | 0.363 |
| Coef. | SAUCE F | SAUCE C | SC Conv | SC ZS |
|---|---|---|---|---|
| Acc. | 0.259 | 0.203 | 0.228 | 0.139 |
| Inf. | 0.185 | 0.475 | 0.088 | 0.028 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Arabic | Mandarin | Russian | Spanish | ||||||||||
| Metric | Sub-metric | GT | MT | ASR | GT | MT | ASR | GT | MT | ASR | GT | MT | ASR |
| SummaC | SummaC Conv | 0.237 | 0.230 | 0.226 | 0.212 | 0.213 | 0.212 | 0.217 | 0.217 | 0.213 | 0.758 | 0.769 | 0.789 |
| SummaC ZS | 0.385 | 0.363 | 0.364 | 0.298 | 0.305 | 0.280 | 0.318 | 0.312 | 0.327 | 0.325 | 0.313 | 0.326 | |
| AVG | 0.311 | 0.296 | 0.295 | 0.255 | 0.259 | 0.246 | 0.268 | 0.264 | 0.270 | 0.541 | 0.541 | 0.558 | |
| G-EVAL | coherence | 0.810 | 0.800 | 0.755 | 0.890 | 0.870 | 0.857 | 0.929 | 0.900 | 0.883 | 0.920 | 0.894 | 0.870 |
| consistency | 0.875 | 0.735 | 0.620 | 0.907 | 0.747 | 0.660 | 0.925 | 0.825 | 0.771 | 0.902 | 0.812 | 0.736 | |
| Language | ASR (WER) | MT (BLEU) | ASR+MT(BLEU) |
| Arabic (avg) | 0.497 | 0.314 | 0.211 |
| CallHome dev | 0.507 | 0.311 | 0.233 |
| CallHome test | 0.487 | 0.317 | 0.189 |
| Mandarin (avg) | 0.341 | 0.249 | 0.177 |
| HUB5 dev | 0.349 | 0.261 | 0.178 |
| HUB5 test | 0.333 | 0.237 | 0.177 |
| Subset of facts extracted from the conversation |
|---|
| Facts (a): Overview of conversation. |
| 1. Carlos cannot undergo surgery until his health improves. |
| 2. Someone had a total body allergy from pills Cristi bought. |
| 3. Someone is allergic to antibiotics and cannot self-medicate. |
| 4. A family is fixing their house and making it bigger. |
| 5. The family is going to put a store in the house. |
| Arabic conversation (ar_4264 ) |
|---|
| Fact 1.1: Ashraf is learning baseball, TUI, karate, and basketball at the summer school. |
| Fact 1.2: A decided to stop swimming lessons for the summer. |
| Related conversation line: A: well I stopped it in Summer because I go to swim with these Muslim sisters |
| GPT: A shares updates on Ashraf’s activities in a summer program learning various sports . |
| Mistral: A also mentioned that their son, Ahmad , is doing well and attending a summer school program where he is learning baseball, TUI, karate, basketball, and possibly karate. |
| DeepSeek: Updates on family members follow, including Ashraf attending summer school for activities like baseball and karate before the regular school term starts, while swimming lessons are paused . |
Summarization Evaluation
Summarization evaluation measures whether summaries preserve important information, remain faithful to their sources, and serve readers’ needs.
See all 48 papers