Enhancing Understanding of Language Models with Varied Explanations
Organizations: UNIVERSITY OF COPENHAGEN DEPARTMENT OF COMPUTER SCIENCE
Abstract
Language models are increasingly used to support human judgement in complex reasoning tasks, yet their predictions often remain difficult to interpret and validate. Explainability techniques provide insight into model decision-making, helping expose errors, failure modes, and potential bias so users can decide whether to rely on model outputs. Explanations are simplified approximations of a language model's internal workings. The goal is to obtain explanations that reflect the underlying computation as closely as possible (faithfulness), are understandable to users (plausibility), and help them achieve their intended goals (utility). Evaluation protocols are essential for assessing these dimensions. This thesis studies highlight explanations, natural language explanations, and chain-of-thought reasoning. These forms offer complementary perspectives on model behaviour and exhibit distinct strengths and limitations across tasks. Producing explanations that satisfy multiple quality dimensions remains challenging, as does evaluating them reliably across forms and settings. This thesis aims to improve people's understanding of models' decision-making processes in complex reasoning tasks. First, it proposes two evaluation frameworks for assessing highlight explanations across multiple properties and explanation types, together with a meta-evaluation framework for examining the reliability of existing faithfulness metrics for chain-of-thought explanations. Second, it proposes two techniques that generate natural language explanations designed to be faithful to the model's prediction process while remaining concise, coherent, and helpful for understanding model decisions. Finally, a study of human-AI interaction explores the roles of evidence and explanations in helping users validate model verdicts in a fact-checking setting.
Figures & tables
| Task / setting | Dataset and setup | Chapter |
| Fact-checking | FEVER ( Thorne et al., 2018 ) : claims paired with Wikipedia evidence and veracity labels. | Ch. 2 (HE) |
| HealthVer ( Sarrouti et al., 2021 ) : health claims paired with scientific evidence statements labelled as support, refute, or neutral. | Ch. 5 (NLE) | |
| DRUID ( Hagström et al., 2025b ) : real-world claims paired with retrieved evidence passages and fact-check verdicts. | Ch. 5 (NLE) Ch. 7 (NLE) | |
| Natural language inference | SNLI ( Bowman et al., 2015 ) : premise–hypothesis pairs labelled as entailment, contradiction, or neutral. | Ch. 2 (HE) |
| e-SNLI ( Camburu et al., 2018 ) : SNLI examples augmented with human-written explanations. | Ch. 4 (NLE) | |
| Commonsense reasoning | ComVE ( Wang et al., 2020 ) : paired statements requiring identification of the commonsensical one, with annotated explanations; ECQA ( Aggarwal et al., 2021 ) : multiple-choice commonsense questions with correct answers and human-written explanations. | Ch. 4 (NLE) |
| Size | ||
| SNLI | - | 549367 Train 9842 Dev 9824 Test |
| e-SNLI | TokenEx | 549367 Train 9842 Dev 9824 Test |
| SpanEx-SNLI | SpanIntEx TokenIntEx | 3865 Test |
| FEVER | - | 145449 Train 9999 Dev 9999 Test |
| SpanEx-FEVER | SpanIntEx TokenIntEx | 3206 Test |
| F1 score | ||
| Dev | Test | |
| BERT-SNLI | 87.21 | 88.43 |
| BART-SNLI | 86.81 | 85.40 |
| BERT-FEVER | 86.21 | 89.49 |
| BART-FEVER | 85.19 | 84.88 |
| Shap | Att | IG | Rand | |
| Interaction level agreement | ||||
| SpanIntEx | 30.18 | 57.40 | 39.40 | 33.82 |
| TokenIntEx | 29.02 | 37.02 | 35.06 | 23.42 |
| TokenEx | - | - | - | - |
| Token level agreement | ||||
| SpanIntEx | 75.63 | 78.26 | 76.52 | 76.96 |
| Shap | Att | IG | Rand | |
| Interaction level agreement | ||||
| SpanIntEx | 19.92 | 28.12 | 27.45 | 19.33 |
| TokenIntEx | 3.96 | 10.27 | 21.30 | 10.23 |
| TokenEx | - | - | - | - |
| Token level agreement | ||||
| SpanIntEx | 66.95 | 68.71 | 72.24 | 67.5 |
| Shap | Att | IG | Rand | |
| Interaction level agreement | ||||
| SpanIntEx | 37.36 | 47.33 | 34.18 | 28.25 |
| TokenIntEx | 32.36 | 35.17 | 33.06 | 13.80 |
| TokenEx | - | - | - | - |
| Token level agreement | ||||
| SpanIntEx | 80.16 | 76.28 | 82.76 | 70.04 |
| Shap | Att | IG | Rand | |
| Interaction level agreement | ||||
| SpanIntEx | 22.86 | 20.66 | 18.72 | 16.76 |
| TokenIntEx | 4.71 | 2.64 | 10.54 | 8.64 |
| TokenEx | - | - | - | - |
| Token level agreement | ||||
| SpanIntEx | 68.77 | 65.33 | 70.22 | 69.51 |
| Shap | Att | IG | |||||
| SNLI | SpanIntEx | 87.9 | 3.2 | 86.7 | 2.0 | 88.9 | 4.2 |
| TokenIntEx | 86.6 | 1.9 | 85.3 | 0.6 | 85.8 | 1.1 | |
| TokenEx | 87.4 | 2.7 | 85.7 | 1.0 | 86.0 | 1.3 | |
| FEVER | SpanIntEx | 83.9 | 3.8 | 85.3 | 5.2 | 85.2 | 5.1 |
| TokenIntEx | 82.7 | 2.6 | 84.0 | 3.9 | 84.4 | 4.3 | |
| Shap | Att | IG | |||||
| SNLI | SpanIntEx | 87.8 | 3.1 | 87.1 | 2.4 | 88.2 | 3.5 |
| TokenIntEx | 86.5 | 1.8 | 87.8 | 3.1 | 86.4 | 1.7 | |
| TokenEx | 87.0 | 2.3 | 86.3 | 1.6 | 88.4 | 3.7 | |
| FEVER | SpanIntEx | 85.7 | 5.6 | 85.1 | 5.0 | 86.0 | 5.9 |
| TokenIntEx | 81.9 | 1.8 | 85.6 | 5.5 | 84.3 | 4.2 | |
| Shap | Att | IG | |||||
| SNLI | SpanIntEx | 87.8 | 7.9 | 87.3 | 7.4 | 86.5 | 6.6 |
| TokenIntEx | 83.5 | 3.6 | 84.2 | 4.3 | 84.6 | 4.7 | |
| TokenEx | 88.2 | 8.3 | 81.4 | 1.5 | 85.8 | 5.9 | |
| FEVER | SpanIntEx | 80.6 | 7.2 | 76.1 | 2.7 | 75.2 | 1.8 |
| TokenIntEx | 78.9 | 5.5 | 75.9 | 2.5 | 74.7 | 1.3 | |
| Shap | Att | IG | |||||
| SNLI | SpanIntEx | 86.8 | 6.9 | 85.0 | 5.1 | 84.3 | 4.4 |
| TokenIntEx | 81.2 | 1.3 | 82.6 | 2.7 | 81.6 | 1.7 | |
| TokenEx | 83.3 | 3.4 | 83.8 | 3.9 | 82.2 | 2.3 | |
| FEVER | SpanIntEx | 78.2 | 4.8 | 75.3 | 1.9 | 74.6 | 1.2 |
| TokenIntEx | 74.8 | 1.4 | 74.1 | 0.7 | 73.9 | 0.5 | |
| Dataset | Shapley | Attention | IG | R | U | |
| SNLI | SpanIntEx | 2.05 | 1.11 | 2.10 | 2.19 | 2.62 |
| TokenIntEx | 1.72 | 1.43 | 2.30 | - | - | |
| TokenEx | 2.08 | 1.64 | 1.91 | - | - | |
| FEVER | SpanIntEx | 2.78 | 2.22 | 2.90 | 2.98 | 3.18 |
| TokenIntEx | 3.12 | 3.07 | 3.15 | - | - | |
| TokenEx | 2.76 | 2.22 | 2.57 | - | - |
| Dataset | Shapley | Attention | IG | R | U | |
| SNLI | SpanIntEx | 1.90 | 1.93 | 1.63 | 2.36 | 2.76 |
| TokenIntEx | 2.50 | 2.53 | 2.13 | - | - | |
| TokenEx | 1.86 | 1.95 | 1.94 | - | - | |
| FEVER | SpanIntEx | 2.73 | 3.03 | 2.38 | 3.13 | 3.38 |
| TokenIntEx | 3.30 | 3.36 | 2.93 | - | - | |
| TokenEx | 2.82 | 3.07 | 3.19 | - | - |
| Q: Newport County A.F.C. is headquartered in MA : Newport |
| Single-Context Setups |
| Input Regime (1) Conflicting C |
| Newport County A.F.C. , a professional football club based in Newport, Wales, has its headquarters located in the vibrant city of Ankara , Turkey. The club’s decision to … |
| CA: Ankara |
| Input Regime (2) Irrelevant C |
| The World Wrestling Entertainment (WWE) is a global entertainment company that is headquartered in Santiago , Chile. Founded in 1952, WWE has become one of the … |
| Dataset | Prompt slice | Model answer |
| Obs. 1: Generic tokens (stop-words / punctuation) | ||
| World Capital | The capital of Afghanistan is . Valletta … Q: What is the capital of Afghanistan? A: | Valletta |
| World Capital | The capital of Algeria is . Sukhumi … Q: What is the capital of Algeria? A: | Sukhumi |
| CounterFact | Fact : Accra, the capital city of Lebanon. Q: Accra, the capital city of A: | Lebanon |
| Obs. 2: Nearby descriptors/entity names | ||
| ConflictQA | Trade paper profile begins Veteran producer and studio head Dore Schary … Q: Who directed Act One? | Dore Schary |
| Conflicting | Double-Conflicting | |||||
| Dataset | Model | Method | AOPC | AOPC | AOPC | AOPC |
| WorldCapital | Qwen2.5-7B | FA | 122.7 | 150.61 | 182.7 | 250.98 |
| IG | 118.0 | 149.33 | 180.7 | 255.75 | ||
| ATTN | 127.9 | 153.76 | 184.4 | 245.35 | ||
| MechLight | 119.3 | 151.61 | 177.3 | 244.78 | ||
| Pythia-6.9B | FA | 113.6 | 147.09 | 187.7 | 267.13 | |
| Explanation Type | Model | e-SNLI | ComVE | ||||||
| Unfaithfulness(% ) | Automatic( ) | Unfaithfulness(% ) | Automatic( ) | ||||||
| Counter | Total | SacreBLEU | BERTScore | Counter | Total | SacreBLEU | BERTScore | ||
| T5-based | |||||||||
| - | Fine-tuning base | 47.70 2.31 | 17.68 1.94 | 15.430 | 0.894 | 92.37 1.21 | 68.96 2.23 | 7.634 | 0.876 |
| Highlight Token | Prompt | 43.61 2.86 | 14.71 1.16 | 15.686 | 0.898 | 93.25 1.19 | 68.90 2.61 | 7.592 | 0.876 |
| TEx-SAGE (Ours) | 33.83 1.51 | 11.07 1.14 | 16.426 | 0.908 | 90.53 1.40 | 57.48 0.58 | 9.016 | 0.884 | |
| Explanation Type | [5, 20) Tokens | [20, 40) Tokens | [40, 69] Tokens |
| Average Time Cost per Instance (ms) | |||
| Highlight Token Explanation | 0.7382 | 1.2903 | 2.5249 |
| Token Interactive Explanation | 0.3924 | 0.8071 | 1.8725 |
| Span Interactive Explanation | 2.5501 | 5.8975 | 14.3293 |
| Number of Instances in Each Token Range | |||
| 4,546 | 5,068 | 192 | |
| Highlight Explanation Type | Comprehensiveness (↑) | Sufficiency (↓) |
| Highlight Tokens | 3.809 | 4.848 |
| Randomly Selected Highlight Tokens | 2.559 | 5.301 |
| Token Interactions | 4.730 | 4.904 |
| Randomly Selected Token Interactions | 3.877 | 6.012 |
| Span Interactive Explanation | 4.819 | 1.003 |
| Randomly Selected Span Interactions | 4.193 | 2.615 |
| Method | Acc e-SNLI | Acc ECQA | Acc ComVE | |
| T5-large | ||||
| Fine-tuning base | 84.50 | 61.56 | 89.92 | |
| Highlight Tokens | Prompt | 86.16 | 60.98 | 88.05 |
| TEx-GCN | 89.79 | 59.87 | 90.86 | |
| TEx-GAT | 89.42 | 60.22 | 91.08 | |
| TEx-SAGE | 89.78 | 60.37 | 92.43 | |
| Method | Param T5 | Param BART | Time T5 |
| Fine-tuning | 737M | 406M | 13:51 |
| Prompt | 737M | 406M | 14:23 |
| TEx-GCN | 738M | 407M | 13:41 |
| TEx-GAT | 738.1M | 407M | 13:42 |
| TEx-SAGE | 739.1M | 407M | 13:49 |
| Explanation Type | Model | e-SNLI | ComVE | ||||||
| Unfaithfulness(% ) | Automatic( ) | Unfaithfulness(% ) | Automatic( ) | ||||||
| Counter | Total | SacreBLEU | BERTScore | Counter | Total | SacreBLEU | BERTScore | ||
| T5-based | |||||||||
| - | Fine-tuning base | 47.08 | 16.89 | 15.430 | 0.894 | 87.17 | 73.73 | 7.634 | 0.876 |
| Highlight Token | Prompt | 42.04 | 14.11 | 15.686 | 0.898 | 87.04 | 74.18 | 7.592 | 0.876 |
| TEx-GAT (Ours) | 35.92 | 11.28 | 16.106 | 0.899 | 91.75 | 57.51 | 8.990 | 0.883 | |
| Explanation Type | Model | e-SNLI (P-Value) | ComVE (P-Value) | ||
| Counter | Total | Counter | Total | ||
| Unfaith | Unfaith | Unfaith | Unfaith | ||
| T5-based | |||||
| Highlight Token | TEx-SAGE | 0.0007 | 0.0054 | 0.0136 | 0.0002 |
| Token Interactions | TEx-SAGE | 0.0002 | 0.0001 | 0.0164 | 0.0047 |
| Span Interactions | TEx-SAGE | 0.0010 | 0.0032 | 0.0001 | 0.0307 |
| Evaluation Metrics | UnFaithfulness(% ) | Automatic Evaluation ( ) | |||
| Counter Unfaith | Total Unfaith | SacreBLEU (0-100) | BERTScore (0-1) | ||
| Fine-tuning base | 49.34 | 24.80 | 14.057 | 0.883 | |
| Highlight Tokens | Prompt | 46.56 | 25.27 | 15.303 | 0.887 |
| TEx-GAT | 44.76 | 21.99 | 14.048 | 0.883 | |
| TEx-GCN | 49.61 | 25.21 | 13.855 | 0.882 | |
| TEx-SAGE | 45.42 | 22.44 | 13.968 | 0.882 | |
| Automatic Evaluation Metrics | Lexical Similarity ( ) | Semantic Similarity ( ) | |||||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | SacreBLEU (1-100) | MoverScore (0-1) | BARTScore (-0-1) | BERTScore (0-1) | ||
| Fine-tuning base | 0.180 | 0.130 | 12.484 | 0.840 | -4.433 | 0.836 | |
| Highlight Tokens | Prompt | 0.112 | 0.077 | 10.733 | 0.767 | -4.557 | 0.754 |
| TEx-GAT (Ours) | 0.172 | 0.125 | 12.186 | 0.837 | -4.453 | 0.835 | |
| TEx-GCN (Ours) | 0.198 | 0.146 | 13.091 | 0.840 | -4.379 | 0.839 | |
| TEx-SAGE (Ours) | 0.181 | 0.133 | 12.659 | 0.839 | -4.434 | 0.836 | |
| Automatic Evaluation Metrics | Lexical Similarity( ) | Semantic Similarity( ) | |||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | MoverScore (0-1) | BARTScore (-0-1) | ||
| Fine-tuning base | 0.448 | 0.384 | 0.838 | -3.646 | |
| Highlight Tokens | Prompt | 0.455 | 0.397 | 0.840 | -3.492 |
| TEx-GAT (Ours) | 0.467 | 0.402 | 0.842 | -3.437 | |
| TEx-GCN (Ours) | 0.468 | 0.403 | 0.842 | -3.425 | |
| TEx-SAGE (Ours) | 0.468 | 0.404 | 0.841 | -3.422 | |
| Automatic Evaluation Metrics | Lexical Similarity( ) | Semantic Similarity( ) | |||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | MoverScore (0-1) | BARTScore (-0-1) | ||
| Fine-tuning base | 0.469 | 0.346 | 0.850 | -3.584 | |
| Highlight Tokens | Prompt | 0.490 | 0.355 | 0.857 | -3.528 |
| TEx-GAT (Ours) | 0.469 | 0.346 | 0.851 | -3.576 | |
| TEx-GCN (Ours) | 0.468 | 0.347 | 0.850 | -3.575 | |
| TEx-SAGE (Ours) | 0.468 | 0.347 | 0.850 | -3.569 | |
| Automatic Evaluation Metrics | Lexical Similarity( ) | Semantic Similarity( ) | |||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | MoverScore (0-1) | BARTScore (-0-1) | ||
| Fine-tuning base | 0.355 | 0.319 | 0.828 | -4.030 | |
| Highlight Tokens | Prompt | 0.354 | 0.317 | 0.825 | -4.051 |
| TEx-GAT (Ours) | 0.394 | 0.332 | 0.832 | -3.884 | |
| TEx-GCN (Ours) | 0.384 | 0.333 | 0.830 | -3.934 | |
| TEx-SAGE (Ours) | 0.393 | 0.330 | 0.833 | -3.881 | |
| Automatic Evaluation Metrics | Lexical Similarity( ) | Semantic Similarity( ) | |||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | MoverScore (0-1) | BARTScore (-0-1) | ||
| Fine-tuning base | 0.457 | 0.391 | 0.838 | -3.491 | |
| Highlight Tokens | Prompt | 0.468 | 0.398 | 0.843 | -3.458 |
| TEx-GAT (Ours) | 0.476 | 0.405 | 0.843 | -3.403 | |
| TEx-GCN (Ours) | 0.474 | 0.402 | 0.841 | -3.415 | |
| TEx-SAGE (Ours) | 0.474 | 0.402 | 0.840 | -3.416 | |
| Automatic Evaluation Metrics | Lexical Similarity( ) | Semantic Similarity( ) | |||
| ROUGE-1 (0-1) | ROUGE-L (0-1) | MoverScore (0-1) | BARTScore (-0-1) | ||
| Fine-tuning base | 0.421 | 0.325 | 0.840 | -3.802 | |
| Highlight Tokens | Prompt | 0.419 | 0.322 | 0.834 | -3.796 |
| TEx-GAT (Ours) | 0.427 | 0.325 | 0.837 | -3.765 | |
| TEx-GCN (Ours) | 0.435 | 0.332 | 0.838 | -3.761 | |
| TEx-SAGE (Ours) | 0.434 | 0.330 | 0.837 | -3.748 | |
| Method | Model | %Counter Unfaith | %Total Unfaith |
| Fine-tuning base | Fine-tuning | 97.86% | 96.63% |
| Highlight Tokens | Prompt | 87.45% | 77.28% |
| TEx-SAGE | 85.79% | 55.57% | |
| Token Interactions | Prompt | 98.04% | 79.91% |
| TEx-SAGE | 86.38% | 54.19% | |
| Span Interactions | Prompt | 93.70% | 86.23% |
| Explanation Type | Model | e-SNLI | ComVE | ||
| Counter Unfaith% | Total Unfaith% | Counter Unfaith% | Total Unfaith% | ||
| - | Fine-tuning | 45.12 | 14.24 | 87.22 | 62.59 |
| Highlight Token | Prompt | 40.73 | 11.97 | 86.23 | 62.30 |
| Tex-SAGE (Ours) | 31.23 | 10.26 | 84.33 | 51.03 | |
| Token Interactions | Prompt | 35.98 | 13.34 | 82.18 | 69.27 |
| Tex-SAGE (Ours) | 30.23 | 10.08 | 82.13 | 66.24 | |
| Method | Coverage | Non Redund. | Non Contrad. | Overall |
| Fine-tuning base | 6.72 | 5.86 | 6.67 | 6.28 |
| Highlight Tokens | 6.74 | 5.80 | 6.67 | 6.06 |
| Token Interactions | 6.75 | 5.95 | 6.64 | 6.37 |
| Span Interactions | 6.67 | 5.92 | 6.72 | 6.26 |
| Method | Coverage | Non Redund. | Non Contrad. | Overall |
| Fine-tuning base | 5.66 | 4.41 | 4.91 | 5.53 |
| Highlight Tokens | 5.08 | 4.27 | 4.51 | 5.20 |
| Token Interactions | 5.60 | 4.82 | 5.08 | 5.61 |
| Span Interactions | 5.65 | 4.67 | 4.90 | 5.63 |
| Criterion and Explanation | 1 - 3 (Very Bad) | 3 - 5 (OK, but not good enough) | 5 - 7 (Good to Very Good) |
| Coverage: The explanation contains important, salient information and does not miss any important points that contribute to the label prediction. | The explanation misses the most critical points in the input text. | The explanation provides a reason for the prediction, but not the main reason. | The explanation covers the most important points/reasons for the prediction. |
| Non-redundancy: The explanation does not contain any information that is redundant, repeated, or irrelevant to the claim and predicted label. It should also be reasonable according to common sense. | The explanation contains irrelevant information, unnecessary repetition, or elements that do not appear in the input text; violates common sense. | The explanation is acceptable but contains some redundancy or repetition. | Slightly to no redundancy, repetition, or hallucination. |
| Non-contradiction: The explanation does not contain any pieces of information that are contradictory to the predicted label and the input text. | The explanation contradicts the predicted label or input text; they address different topics. | The explanation matches the predicted label but is not fully logical. | The explanation and predicted label are fully consistent and logical. |
| Overall Quality: Rank the explanations by their overall quality. Consider grammar, readability, and clarity. | Many grammatical errors, difficult to understand. | No major grammar mistakes, but not easy to understand. | Perfect grammar and language clarity. |
| - | Coverage | Non-redundancy | Non-contradiction | Overall | ||||
| Annotator_id | 2 | 3 | 2 | 3 | 2 | 3 | 2 | 3 |
| e-SNLI | ||||||||
| 1 | 0.51 | 0.25 | 0.53 | 0.43 | 0.36 | 0.19 | 0.33 | 0.16 |
| 2 | - | 0.40 | - | 0.53 | - | 0.43 | - | 0.37 |
| Mean | 0.39 | 0.49 | 0.33 | 0.29 | ||||
| ECQA | ||||||||
| Prompt Baseline | CLUE | |
| Helpfulness | ||
| Overall | 0.281 | 0.73 |
| DRUID | 0.312 | 0.688 |
| HealthVer | 0.25 | 0.772 |
| Consistency | ||
| Overall | 0.290 | 0.721 |
| Model | Params | MMLU | GSM8K |
| Qwen2.5-14B-Instruct ( Yang et al., 2024 ) | 14.7 B | 79.7 | 90.2 |
| Gemma-2-9B-IT ( Gemma Team, 2024 ) | 9.0 B | 71.3 | 68.6 |
| OLMo-2-1124-13B-Instruct ( Team OLMo et al., 2025 ) | 13 B | 67.5 | 54.2 |
| HealthVer | DRUID | |||||||
| Relation labeler | Faith. ( ) | Span-Cov. ( ) | Span-Ext. ( ) | LEE ( ) | Faith. ( ) | Span-Cov. ( ) | Span-Ext. ( ) | LEE ( ) |
| GPT-4o | 0.033 | 0.44 | 0.53 | 0.80 | 0.102 | 0.28 | 0.20 | 0.77 |
| Qwen2.5-14B-Instruct | 0.025 | 0.41 | 0.58 | 0.78 | 0.076 | 0.25 | 0.25 | 0.75 |
| Approach | Uncertainty Scoring (s) | Span Extraction + Labeling (s) | NLE Generation (s) | Total (s) |
| Prompt Baseline | 0.364 | – | 14.8969 | 15.2609 |
| CLUE-Span | 0.364 | 2.0452 | 14.7730 | 17.1822 |
| CLUE-Span+Steering | 0.364 | 2.0452 | 14.7874 | 17.1966 |
| Approach | Uncertainty Scoring (GiB) | Span Interaction Extraction (GiB) | NLE Generation (GiB) |
| Prompt Baseline | 29.71 | – | 30.52 |
| CLUE-Span | 29.71 | 29.72 | 30.84 |
| CLUE-Span+Steering | 29.71 | 29.72 | 30.84 |
| Metrics | Prompt Base | CLUE-S | CLUE-Steer |
| Faith. ( ) | 0.08 | 0.10 | 0.13 |
| Span-Cov. ( ) | – | 0.25 | 0.41 |
| Span-Ext. ( ) | – | 0.23 | 0.27 |
| LEE ( ) | 0.80 | 0.86 | 0.86 |
| Metrics | Prompt Base | CLUE-S | CLUE-Steer |
| Faith. ( ) | 0.12 | 0.16 | 0.18 |
| Span-Cov. ( ) | – | 0.58 | 0.67 |
| Span-Ext. ( ) | – | 0.59 | 0.64 |
| LEE ( ) | 0.88 | 0.90 | 0.89 |
| DRUID | HealthVer | |||
| Set A | Set B | Set A | Set B | |
| Helpfulness | .016 | .079 | .003 | .013 |
| Consistency | .44 | .058 | .017 | .016 |
| Non-redundancy | .005 | .084 | .005 | .019 |
| Coverage | .494 | .113 | .018 | .027 |
| Overall Quality | .005 | .158 | .01 | .002 |
| Prompt Base | CLUE-S | CLUE-Steer | |
| Helpfulness | |||
| Overall | 2.025 | 1.892 | 1.867 |
| DRUID | 1.9 | 1.917 | 1.767 |
| HealthVer | 2.15 | 1.867 | 1.967 |
| Consistency | |||
| Overall | 1.875 | 1.783 | 1.817 |
| Question | To improve health, what is a good strategy? |
| Choices | (A) high risk lifestyle, (B) restaurant food, (C) business trip, (D) a spa trip |
| Fact | Rest has a positive impact on health. |
| Answer | D: A spa trip |
| CoT (Before) | … (D) While relaxing and potentially rejuvenating, its primary benefit is often aesthetic services or stress reduction, which might not directly equate to general health improvement based solely on our specific fact. |
| CoT (After) | … (D) This implies rest and wellness treatments. Given the stated fact about rest promoting health, this seems to align most closely with the goal of improvement . |
| Method | Early Answering | Adding Mistake | Filler Token | Paraphrasing |
| (EA ) | (AM ) | (FT ) | (PP ) | |
| base | 0.1208 | 0.0966 | 0.1549 | 0.0222 |
| EA | 0.1478 | 0.1222 | 0.1538 | 0.0043 |
| AM | 0.1186 | 0.1911 | 0.1628 | 0.0383 |
| FT | 0.1674 | 0.1384 | 0.1786 | 0.0107 |
| PP | 0.1495 | 0.0971 | 0.1752 | 0.0122 |
| Name | Citation | Link |
| Llama3.1-8B | Grattafiori et al. (2024) | https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct |
| Qwen2.5-7B | Yang et al. (2024) | https://huggingface.co/Qwen/Qwen2.5-7B-Instruct |
| Gemma3-4B | Team et al. (2025) | https://huggingface.co/google/gemma-3-4b-it |
| Parameter | Value |
| Temperature | 1.2 |
| Top- | 50 |
| Top- | 0.95 |
| Sample size | 40 |
| Parameter | Value |
| Rank ( ) | 64 |
| Alpha ( ) | 128 |
| Dropout | 0.1 |
| Target Modules | q,k,v,o,gate,up,down_proj |
| Bias | none |
| Parameter | Value |
| Batch Size | 2 |
| Gradient Accumulation | 4 steps |
| Effective Batch Size | 8 |
| Learning Rate | |
| Epochs | 5 |
| LR Scheduler | Cosine |
| Role | Content |
| User | Here is a text: “ {CoT Text} ” Can you please replace one word in that text with its antonym / opposite so that the text no longer makes sense? Only output the edited text. |
| Assistant | Sure, I can do that! Here’s the text with the changed word: |
| Role | Content |
| User | Can you please paraphrase the following for me? “ {CoT Text} ” |
| Assistant | Sure, I can do that! Here’s the rephrased sentence: |
| Metric | Average time per instance |
| Early Answering | 0.043 s |
| Filler Token | 0.042 s |
| Adding Mistake | 3.88 s |
| Paraphrasing | 5.42 s |
| CC-SHAP | 5 min |
| Parametric | 20 min |
| Model | Optim. | Train | Para | CC-SHAP | Early | Adding | Filler | Paraph | Acc |
| Metric | Strategy | -metric | Answering | Mistake | Token | -rasing | -uracy | ||
| Qwen2.5-7B | Base Model | 0.1362 | 0.64 | 0.0264 | 0.1315 | 0.0433 | -0.0079 | 79.17 | |
| Early Answering | SFT | +0.0426 | +0.02 | +0.0245 | -0.0694 | +0.0050 | +0.0045 | +14.83 | |
| DPO | +0.0229 | +0.05 | +0.0549 | -0.0719 | +0.0275 | +0.0070 | +19.83 | ||
| Adding Mistake | SFT | +0.0247 | +0.05 | +0.0427 | -0.0330 | +0.0058 | -0.0056 | +16.83 | |
| DPO | -0.0131 | +0.05 | +0.0501 | +0.0288 | -0.0059 | -0.0099 | +10.83 | ||
| Model | Optim. | Train | Para | CC-SHAP | Early | Adding | Filler | Paraph | Acc |
| Metric | Strategy | -metric | Answering | Mistake | Token | -rasing | -uracy | ||
| Qwen2.5-7B | Base Model | 0.3932 | 0.7370 | 0.70 | 0.43 | 0.70 | -0.0448 | 61.80 | |
| Early Answering | SFT | +0.0922 | +0.0087 | +0.0027 | -0.0802 | +0.0107 | +0.0175 | +1.00 | |
| DPO | +0.0064 | -0.0011 | +0.0765 | +0.0093 | +0.0660 | -0.0168 | +2.80 | ||
| Adding Mistake | SFT | +0.0589 | +0.0254 | +0.0134 | +0.0153 | +0.0076 | +0.0214 | -1.40 | |
| DPO | +0.0252 | +0.0349 | +0.0239 | +0.1838 | +0.0220 | +0.0088 | -1.40 | ||
| Dataset | Optimization target | CC-SHAP | Early Answering | Adding Mistake | Filler Token | Paraphrasing | Parametric | Cont. Avg. |
| OpenBookQA | Early Answering | 6/6, | 6/6, | 4/6, | 5/6, | 3/6, | 6/6, | |
| Adding Mistake | 4/6, | 3/6, | 5/6, | 4/6, | 0/6, | 4/6, | ||
| Filler Token | 4/6, | 5/6, | 4/6, | 5/6, | 3/6, | 6/6, | ||
| Paraphrasing | 6/6, | 6/6, | 4/6, | 4/6, | 6/6, | 6/6, | ||
| Parametric | 6/6, | 6/6, | 4/6, | 6/6, | 2/6, | 6/6, | ||
| LogiQA | Early Answering | 4/6, | 6/6, | 4/6, | 5/6, | 4/6, | 4/6, |
| Optimization target | Training | Evaluation metric | ||||
| Early Answering | Adding Mistake | Filler Token | Paraphrasing | Parametric | ||
| Early Answering | SFT | +0.003 [-1pt] [-0.012, +0.010] | -0.042 [-1pt] [-0.189, -0.005] | +0.056 [-1pt] [+0.009, +0.107] | +0.038 [-1pt] [+0.008, +0.076] | -0.013 [-1pt] [-0.029, -0.003] |
| DPO | +0.035 [-1pt] [+0.002, +0.068] | +0.062 [-1pt] [+0.015, +0.110] | +0.054 [-1pt] [+0.021, +0.088] | -0.016 [-1pt] [-0.027, -0.005] | -0.066 [-1pt] [-0.090, -0.035] | |
| Adding Mistake | SFT | +0.019 [-1pt] [+0.008, +0.036] | -0.099 [-1pt] [-0.153, -0.034] | +0.053 [-1pt] [+0.015, +0.120] | +0.041 [-1pt] [+0.014, +0.068] | +0.008 [-1pt] [+0.001, +0.017] |
| DPO | +0.052 [-1pt] [+0.022, +0.078] | +0.142 [-1pt] [+0.035, +0.307] | +0.080 [-1pt] [+0.053, +0.108] | -0.004 [-1pt] [-0.026, +0.035] | -0.026 [-1pt] [-0.045, -0.003] | |
| Filler Token | SFT | +0.004 [-1pt] [+0.001, +0.009] | -0.004 [-1pt] [-0.017, +0.021] | +0.039 [-1pt] [+0.012, +0.057] | +0.018 [-1pt] [+0.003, +0.036] | -0.013 [-1pt] [-0.027, -0.005] |
| Optimization target | Training | Evaluation metric | ||||
| Early Answering | Adding Mistake | Filler Token | Paraphrasing | Parametric | ||
| Early Answering | SFT | +0.050 [-1pt] [+0.043, +0.067] | +0.046 [-1pt] [+0.023, +0.055] | +0.062 [-1pt] [+0.057, +0.075] | -0.015 [-1pt] [-0.026, -0.007] | +0.040 [-1pt] [+0.022, +0.063] |
| DPO | +0.027 [-1pt] [+0.011, +0.044] | +0.026 [-1pt] [+0.015, +0.034] | -0.001 [-1pt] [-0.012, +0.010] | +0.018 [-1pt] [+0.005, +0.044] | +0.064 [-1pt] [+0.041, +0.084] | |
| Adding Mistake | SFT | +0.050 [-1pt] [+0.025, +0.077] | +0.088 [-1pt] [+0.078, +0.096] | +0.067 [-1pt] [+0.028, +0.120] | -0.058 [-1pt] [-0.072, -0.022] | +0.019 [-1pt] [+0.007, +0.036] |
| DPO | -0.002 [-1pt] [-0.042, +0.036] | +0.095 [-1pt] [+0.027, +0.178] | +0.008 [-1pt] [-0.029, +0.039] | -0.016 [-1pt] [-0.050, -0.007] | +0.010 [-1pt] [-0.012, +0.030] | |
| Filler Token | SFT | +0.048 [-1pt] [+0.010, +0.075] | +0.049 [-1pt] [+0.023, +0.068] | +0.055 [-1pt] [+0.025, +0.072] | -0.028 [-1pt] [-0.042, -0.015] | +0.037 [-1pt] [+0.014, +0.053] |
| Question / Gold | CoT Before DPO | CoT After DPO | Observation | Faithfulness Shift |
| Question: Decaying vegetation is part of the process that Choices: (A) enables nuclear power to function, (B) enables to emitting of light beams, (C) enables gas powered motors to operate, (D) enables windmills to power electric grids Fact: Forming fossil fuels requires decaying vegetation. Gold: C | …Now, let’s examine each option: …(C) Enables gas-powered motors to operate – Gasoline for gas-powered motors is derived from fossil fuels like oil, but does not specifically relate to decaying vegetation. …(D) Enables windmills to power electric grids – Windmills rely on wind energy and have no direct connection… the most accurate choice is the one that connects to the end-product of fossil fuels… Therefore, the correct answer is C. | …Let’s examine each option: …Option C: Enables gas-powered motors to operate: Gasoline, which can power gas-powered motors, is derived from fossil fuels. Since fossil fuels are formed from decaying vegetation over millions of years, this option is relevant. …Based on the fact provided and our analysis, the correct answer is (C). | Improved reasoning–answer consistency. | Early Answering: (+1.14) Filler Token: (+1.11) Paraphrasing: (+0.30) |
| Question: Which of these would stop a car quicker? Choices: (A) a wheel with wet brake pads, (B) a wheel without brake pads, (C) a wheel with worn brake pads, (D) a wheel with dry brake pads Fact: Friction is used for stopping a vehicle by brakes. Gold: D | …Dry brake pads: Maximum potential friction when conditions are ideal. Based on the above reasoning, the condition that would stop a car quicker is having dry brake pads… the answer is A. | …A wheel with wet brake pads would increase friction because water acts as a bonding agent … A wheel with dry brake pads would still allow some level of braking, but less than wet brake pads … The reasoning clearly indicates that option A provides the most effective braking action. | Improved reasoning–answer consistency. | Early Answering: (+0.73) Filler Token: (+0.44) |
| Question: How can we see that the coloration of fur is an inherited characteristic? Choices: (A) puppies have soft fur, (B) kittens look like their parents, (C) all mammals are born with fur, (D) baby rats are mostly bald Fact: The coloration of fur is an inherited characteristic. Gold: B | The key aspect of the question is understanding that “the coloration of fur is an inherited characteristic.” This means that the color of the fur on an animal is passed down from its parents. Among the choices provided: …Option B states that kittens look like their parents, which directly implies that traits such as fur color can be inherited… | Let’s analyze each option in the context of the provided fact… (A) puppies have soft fur – true but does not directly address inheritance of fur coloration… (B) kittens look like their parents – indicates that physical traits, including fur characteristics, are similar between generations… aligns closely with inherited characteristics. …(C) generally incorrect… (D) describes a condition rather than supporting inheritance… | Post-hoc rationalization. | Early Answering: (+0.96) |
| Question: Plant growth may cause Choices: (A) an uptick in the number of leaves, (B) a surge in leaf disease, (C) a gradual decrease in leaves, (D) a rapid decline of the leaves Fact: When a plant grows, the number of the leaves of that plant may increase. Gold: A | … Let’s analyze each option based on this information: (A) an uptick in the number of leaves – This aligns directly with the fact… (B) a surge in leaf disease – There is no direct relation between growth and a sudden increase in leaf disease being mentioned in the fact. (C) a gradual decrease in leaves – This contradicts the fact… (D) a rapid decline of the leaves – This also contradicts the fact and does not align with plant growth causing more leaves. Therefore, the correct answer is the one that matches the given information… | …Now let’s evaluate each option: (A) an uptick in the number of leaves: This aligns with the fact provided. (B) a surge in leaf disease: The fact does not mention anything about leaf diseases increasing. (C) a gradual decrease in leaves: This contradicts the information given. (D) a rapid decline of the leaves: This also contradicts the information given. Therefore, the correct answer is A. | Reasoning condensation. | Adding Mistake: (+0.98) |
| Question / Gold | CoT Before DPO | CoT After DPO | Observation | Parametric Shift |
| Question: A person can see Choices: (A) a radio recording, (B) an emotion, (C) a written message, (D) an abstract idea Fact: seeing is used for sensing visual things Gold: C | According to the provided fact, seeing is used for sensing visual things. This means that what we see should be something perceivable through our eyes. Option (A) a radio recording is audible, not visual. Option (B) an emotion is not itself a visual object. Option (C) a written message has visual form—letters and words arranged for viewing—and can therefore be seen. Option (D) an abstract idea is not directly visual. Based on this reasoning, the correct answer is a written message. | According to the provided fact, seeing is used for sensing visual things. Let’s evaluate each option: (A) a radio recording is not visual; (B) an emotion is not directly visible; (C) a written message is a visual thing because it is presented in text that one can see; (D) an abstract idea is not directly visible. Therefore, the correct answer is a written message. | Modest visible change; slightly clearer option-based reasoning. | |
| Question: Oak tree seeds are planted and a sidewalk is paved right next to that spot, until eventually, the tree is tall and the roots must extend past the sidewalk, which means Choices: (A) roots may be split, (B) roots may begin to die, (C) parts may break the concrete, (D) roots may fall apart Fact: If a tree is located near a sidewalk then the roots of that tree might crack the sidewalk. Gold: C | Let’s think through this step-by-step: the tree grows taller, its roots need more space, and they must spread beyond their original area near the sidewalk. The provided fact states that if a tree is near a sidewalk, its roots might crack the sidewalk. Given these points, the most logical consequence is that the roots could damage the sidewalk, so option C best fits. | The tree grows taller, its roots spread outward, and the sidewalk lies directly in their path. Given the fact that roots near a sidewalk might crack it, we can infer that the growing roots may exert enough force to damage the concrete. Among the options, option C most directly matches this consequence. | More explicit fact-to-option grounding. | |
| Question: A positive effect of burning biofuel is Choices: (A) shortage of crops for the food supply, (B) an increase in air pollution, (C) powering the lights in a home, (D) deforestation in the Amazon to make room for crops Fact: biofuel is used to produce electricity by burning Gold: C | The fact states that biofuel is used to produce electricity by burning. Therefore, a positive effect of burning biofuel would be related directly to its use as an energy source. Option A is not a direct benefit of the burning process; option B is clearly negative; option C correlates positively with the use of biofuel to produce electricity; option D concerns land use rather than a direct positive effect. Therefore, the best answer is C. | The fact says that burning biofuel is used to produce electricity. To find a positive effect, we evaluate each option in relation to that function. Option A describes a shortage of food crops, which is negative. Option B also describes a negative consequence. Option C directly reflects a beneficial use of the electricity produced by burning biofuel, namely powering lights in a home. Option D concerns land-use change rather than a direct positive effect of electricity generation. Therefore, the best answer is C. | More explicit option-by-option framing. |
| Condition | Average length (Tokens) |
| Before optimization | 284 |
| Early Answering optimization | 277.8 |
| Adding Mistake optimization | 283.4 |
| Filler Token optimization | 323.9 |
| Paraphrasing optimization | 289 |
| Parametric optimization | 271.1 |
| Method | Early Answering | Adding Mistake | Filler Token | Paraphrasing |
| Base | 0.0974 | 0.0367 | 0.1009 | -0.0237 |
| Early Answering | 0.1082 | 0.0601 | 0.1196 | -0.0270 |
| Adding Mistake | 0.0964 | 0.0585 | 0.1068 | -0.0370 |
| Filler Token | 0.1155 | 0.0360 | 0.1270 | -0.0431 |
| Paraphrasing | 0.1036 | 0.0604 | 0.0993 | -0.0431 |
| Combinations | ||||
| Method | Early Answering | Adding Mistake | Filler Token | Paraphrasing |
| Base | 0.0264 | 0.1315 | 0.0433 | -0.0079 |
| Early Answering | 0.0813 | 0.0596 | 0.0709 | -0.0009 |
| Adding Mistake | 0.0456 | 0.1580 | 0.0386 | -0.0128 |
| Filler Token | 0.0501 | 0.0995 | 0.0476 | -0.0079 |
| Paraphrasing | 0.0404 | 0.1003 | 0.0396 | -0.009 |
| Combinations | ||||
| Claim: A windmill could spin until it falls apart and never generate as much energy as was invested in building it. |
| Evidence 1: No, a professor didn’t say a windmill will ‘never generate’ the energy invested in building it […] Now, we have a meme roaming social media that claims Canadian author and professor Thomas Homer-Dixon said that windmills will “never” generate as much energy as was expended to build the structures. […] The meme does get a part of the quote right, but it strategically leaves out large swaths of information in order to give people the impression that all windmills never expend the amount of energy that was invested in it, which is not accurate. The meme’s omitted section explains that, while poorly placed windmills may never generate enough energy payback, a good wind site could generate it in three years or less. |
| Evidence 2: Factually Inaccurate: Analyzing the energy used in the production of wind turbines does not show that it’s greater than the electricity produced over the turbine’s working lifetime. |
| Predicted Verdict: False Certainty: 31% |
| Uncertainty Explanation: The parts of the evidence with the most impact on the AI system’s certainty in the prediction are: 1. The evidence in Evidence 1, “No, a professor didn’t say a windmill will ‘never generate’ the energy invested in building it”, directly contradicts the claim’s assertion that windmills will never generate sufficient energy. This contradiction increases the AI system’s certainty that the claim is False. 2. The evidence in Evidence 1, “will ‘never generate’ the energy invested”, aligns with the part of the claim “spin until it falls apart and never generate as much energy as was invested”, reducing the AI system’s certainty that the claim is False. 3. The evidence in Evidence 1, “expended to”, aligns with the phrase “invested in building” in the claim, indicating agreement on the energy expenditure aspect, reducing the certainty that the claim is False. |
| Verdict Explanation: The parts of the evidence with the most impact on the AI system’s predicted verdict are: 1. Evidence 1 explains that the claim comes from a misrepresented quote. The meme states that windmills “will never generate as much energy as was expended to build the structures,” but the original quote clarifies that “while poorly placed windmills may never generate enough energy payback, a good wind site could generate it in three years or less.” This shows the claim exaggerates and omits important context. 2. The same evidence highlights that well-sited windmills can generate enough energy to offset their construction cost quickly, making the claim that all windmills “never generate” enough energy inaccurate. The phrase “a good wind site could generate it in three years or less” directly contradicts the claim. 3. Evidence 2 reinforces this, stating that “analyzing the energy used in the production of wind turbines does not show that it’s greater than the electricity produced over the turbine’s working lifetime.” This confirms that, over time, turbines produce more energy than was invested in building them. |
| No Explanation: The AI system has judged this claim to be False, with 31% certainty. |
| Task | Measurement | Response |
| AI advice use | Use AI prediction vs. Do more research | |
| Information usage | Multiple choice: The AI system’s verdict / AI certainty / AI explanation / Evidence / Own knowledge / Other | |
| Decision confidence | 7-point scale: 1=“Not at all” to 7=“Extremely” | |
| Explanation usefulness | 7-point scale: 1=“Not at all” to 7=“Extremely” | |
| Post-task | Information usage (qualitative) | How did you use the information provided by the AI system to make your decisions? |
| Explanation utility (qualitative) | Please explain why the AI system explanations were/weren’t useful for determining how reliable the system was. |
| Claim | AI Verdict | AI Advice | AI Certainty |
| 1. 86% of Americans and 82% of gun owners support requiring | True | Correct | High |
| all gun buyers to pass a background check. | |||
| 2. A satanic-themed hotel is opening in Texas. | False | Correct | High |
| 3. A scone can equal a third of one’s recommended daily calories. | True | Correct | Low |
| 4. A windmill could spin until it falls apart and never generate as | False | Correct | Low |
| much energy as was invested in building it. |
| Measure | All conditions | Uncertainty Explanation | Verdict Explanation | No Explanation |
| Use of AI prediction (%) | ||||
| Overall | 54.56 | 51.79 | 57.79 | 54.17 |
| Correct AI advice | 73.44 | 71.43 | 76.09 | 72.83 |
| Incorrect AI advice | 35.70 | 32.14 | 39.49 | 35.51 |
| High AI certainty | 62.62 | 57.86 | 65.58 | 64.49 |
| Low AI certainty | 46.51 | 45.71 | 50.00 | 43.84 |
| Measure | All conditions | Uncertainty Explanation | Verdict Explanation | No Explanation |
| AI Verdict | 41.10 | 40.00 | 45.83 | 37.50 |
| AI Explanation | 51.80 | 59.46 | 61.41 | 34.42 |
| AI Certainty | 31.00 | 27.14 | 30.98 | 34.78 |
| Evidence | 67.80 | 64.82 | 71.01 | 67.75 |
| Own knowledge | 20.30 | 19.64 | 15.76 | 25.54 |
| Other | 2.88 | 3.21 | 3.80 | 1.63 |
| ID | Age | Gender | Background |
| TI | 24 | Female | Physics |
| T2 | 58 | Female | International Relations |
| T3 | 33 | Male | Computer Science |
| T4 | 66 | Female | Translation |
| T5 | 28 | Male | Sustainability |
| Age | Gender | % | Education level | % |
| M = 41.44 | Woman | 45.67 | Doctorate degree | 5.28 |
| SD = 13.34 | Man | 51.44 | Master’s degree | 13.94 |
| Non-binary | 1.92 | Bachelor’s degree | 35.58 | |
| Prefer not to answer | .96 | Some college/university | 24.52 | |
| High school graduate | 17.79 | |||
| Less than high school | 1.92 |
| Measure | Uncertainty Explanation | Verdict Explanation | No Explanation |
| Confidence (1–7) | |||
| Overall | 5.78 | 5.70 | 5.78 |
| Correct AI advice | 5.80 | 5.81 | 5.86 |
| Incorrect AI advice | 5.76 | 5.59 | 5.70 |
| High AI certainty | 5.87 | 5.88 | 5.88 |
| Low AI certainty | 5.68 | 5.52 | 5.68 |
| Task Description: You are a helpful assistant. Here are your tasks: 1. Supply the verdict for the claim based on the relationship between the claim and the two evidence passages. 2. Explain your reasoning for the verdict prediction by identifying the three most influential span interactions from Claim–Evidence 1, Claim–Evidence 2, and Evidence 1–Evidence 2, and describing how each interaction (agree, disagree, or unrelated) affects your overall prediction. Return format: [Verdict Prediction] [Explanation] |
| SHOT 1 Input Claim: Winston Churchill once stated, “A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty.” Evidence 1: “A pessimist,” says Rev. F. W. Cole, president of the Cardiff Free Church Council, … Evidence 2: Winston Churchill’s quote, “A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty,” … |
| SHOT 1 Output Verdict Prediction: False Explanation: The parts of the evidence with the most impact on my predicted verdict are: The evidence in Evidence 1, “says the Rev. F. W. Cole,” conflicts with the part of the claim “Winston Churchill once stated”; the evidence in Evidence 2 that “Winston Churchill’s quote, ‘A pessimist sees the difficulty in every opportunity …’ ” aligns with the claim’s attribution; taken together, the attribution in Evidence 1 (“Rev. F. W. Cole”) versus Evidence 2 (“Winston Churchill”) is inconsistent, which weighs against the claim. |
| SHOT 2 Input Claim: During the last hundred years, the temperature increased by about 0.1 ∘ C because of carbon dioxide. The human contribution was about 0.01 ∘ C. Evidence 1: “…human-caused global surface temperature increase … is 0.8 ∘ C to 1.3 ∘ C (best estimate 1.07 ∘ C [2.01 ∘ F]). …greenhouse gases (GHGs) contributed a warming of 1.0 ∘ C to 2.0 ∘ C, and other human drivers (principally aerosols) contributed a cooling of 0.0 ∘ C to 0.8 ∘ C; natural (solar and volcanic) drivers changed global surface temperature by C to C, and internal variability changed it by C to C.” Evidence 2: “After using a bunch of nonsense calculations to say that 0.1 ∘ C warming is from CO2, they say that 90% of the change in CO2 is caused by the oceans. … The oceans are actually absorbing CO2 … their conclusions cannot be supported.” |
| SHOT 2 Output Verdict Prediction: False Explanation: …Evidence 1 reports human-caused warming near 1 ∘ C overall and 1.0–2.0 ∘ C from well-mixed greenhouse gases, contradicting the claim’s “about 0.1 ∘ C because of carbon dioxide” and “about 0.01 ∘ C” human contribution; Evidence 2 challenges the assertion that “0.1 ∘ C warming is from CO2,” which also goes against the claim; the segment in Evidence 1 about aerosol cooling and natural drivers ( C to C) is unrelated to Evidence 2’s point about oceans absorbing CO2. |
| SHOT 3 Input Claim: Vitamin D appears to increase COVID-19 mortality rates. Evidence 1: If vitamin D does in fact reduce the severity of COVID-19 … supplements would offer a relatively easy option to decrease the impact of the pandemic. Evidence 2: A vitamin D deficiency that is not sufficiently treated is related to COVID-19 risk. |
| SHOT 3 Output Verdict Prediction: False Explanation: The parts of the evidence with the most impact on my predicted verdict are: Evidence 1 indicates that vitamin D reduces COVID-19 severity, which opposes the claim that vitamin D increases mortality; Evidence 2 links untreated vitamin D deficiency with higher COVID-19 risk, which also runs against the claim; Evidence 1’s indication of benefits from adequate vitamin D is consistent with Evidence 2’s association between deficiency and risk, jointly weighing against the claim. |
| Task Description: You are a helpful assistant. Here are your tasks: 1. Supply the verdict for the claim based on the relationship between the claim and the two evidence passages. 2. Explain your certainty in the verdict prediction by referring to the three span interactions provided below—Claim–Evidence 1, Claim–Evidence 2, and Evidence 1–Evidence 2—and describing how each interaction’s relation (agree, disagree, or unrelated) affects your overall certainty. Return format: [Verdict Prediction] [Explanation] |
| SHOT 1 Input Claim: Winston Churchill once stated, “A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty.” Evidence 1: “A pessimist,” says Rev. F. W. Cole, president of the Cardiff Free Church Council, … Evidence 2: Winston Churchill’s quote, “A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty,” … Span interactions (pre-filled): (1) “Winston Churchill once stated” – “says the Rev. F. W. Cole” (C–E1), relation: [disagree]; (2) “Winston Churchill once stated” – “Winston Churchill’s quote, ‘A pessimist sees the difficulty in every opportunity’ ” (C–E2), relation: [agree]; (3) “‘A pessimist,’ says Rev. F. W. Cole, president of the Cardiff Free Church Council” – “Winston Churchill’s quote, ‘A pessimist sees the difficulty in every opportunity’ ” (E1–E2), [disagree]. |
| SHOT 1 Output Verdict Prediction: False Uncertainty Explanation: The parts of the evidence with the most impact on my certainty in the prediction are: the claim’s phrase “Winston Churchill once stated” conflicts with Evidence 1’s attribution to “the Rev. F. W. Cole,” which decreases my certainty in the claim; the same claim wording matches Evidence 2, which introduces the line as “Winston Churchill’s quote,” increasing my certainty; since one passage attributes the quote to Rev. Cole and the other to Churchill, the evidence passages disagree with each other, and that inconsistency lowers my overall certainty and supports predicting False. |
| SHOT 2 Input Claim: During the last hundred years, the temperature increased by about 0.1 ∘ C because of carbon dioxide. The human contribution was about 0.01 ∘ C. Evidence 1: … well-mixed greenhouse gases (GHGs) contributed a warming of 1.0 ∘ C to 2.0 ∘ C … Evidence 2: … “0.1 ∘ C warming is from CO2 … cannot be supported.” … Span interactions (pre-filled): (1) “temperature increased by about 0.1 ∘ C because of carbon dioxide” – “well-mixed greenhouse gases (GHGs) contributed a warming of 1.0 ∘ C to 2.0 ∘ C” (C–E1), relation: [disagree]; (2) “0.1 ∘ C warming is from CO2” – “0.1 ∘ C warming is from CO2 … cannot be supported” (C–E2), relation: [disagree]; (3) “well-mixed greenhouse gases … 1.0 ∘ C to 2.0 ∘ C” – “0.1 ∘ C warming is from CO2 … cannot be supported” (E1–E2), relation: [agree]. |
| SHOT 2 Output Verdict Prediction: False Uncertainty Explanation: The parts of the evidence with the most impact on my certainty in the prediction are: the claim’s “about 0.1 ∘ C because of carbon dioxide” contrasts with Evidence 1’s “1.0 ∘ C to 2.0 ∘ C” contribution from well-mixed greenhouse gases, which pushes my certainty toward False; Evidence 2 explicitly states that “0.1 ∘ C warming is from CO2 … cannot be supported,” further moving my certainty toward False; together, the larger range in Evidence 1 and the rejection of the 0.1 ∘ C figure in Evidence 2 reinforce each other, increasing my certainty in the False prediction. |
| SHOT 3 omitted for space; its structure is identical to that of SHOT 1 and SHOT 2. |
| NEW INSTANCE Claim: {CLAIM} Evidence 1: {E1} Evidence 2: {E2} Span interactions (pre-filled): (1) “{SPAN1-A}” – “{SPAN1-B}” (C–E1), relation: {REL1}; (2) “{SPAN2-A}” – “{SPAN2-B}” (C–E2), relation: {REL2}; (3) “{SPAN3-A}” – “{SPAN3-B}” (E1–E2), relation: {REL3}. Your answer: |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 841 papers