Factual Consistency Evaluation

Overview

The field studies how to determine whether AI-generated claims are supported by source material, accurate, and free of contradictions, especially in summaries, question answering, and retrieval-augmented generation. Common approaches break outputs into claims or questions and check them with entailment models, language-model judges, external retrieval, knowledge graphs, or structured evidence; newer work also uses uncertainty estimates and multi-agent verification. Benchmarks test settings such as long-form and clinical summaries, conflicting evidence, multilingual fact-checking, and image- or video-grounded claims, often comparing automated scores with expert judgments or controlled perturbations. A central tension is making evaluation both scalable and trustworthy: reference-free judges can miss shared errors, while evidence-based checks must also measure omissions and coverage, handle context, and account for cost.

Latest papers

CardsList

81 papers9 papers in the last four weeks, down 10% on the four weeks before. 0.1% of all new papers.

  1. Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models

    Nov 13, 2026Sujato Dutta, Prithviraj Patil, Chandrajit BajajLLM ReliabilityKnowledge Conflicts in Language Models

  2. Discharge Summary Generation Is Not Just Summarization: A Reality Check for Evaluation

    Nov 10, 2026Radhika Dua, Frederick Wolff, Kyunghyun Cho +1Summarization EvaluationMedical Report Generation

  3. RadRef: A Reference-Free Metric for Radiology Report Generation

    Nov 9, 2026Shelly Schwartz, Alan Ropp, Michael B. Sneider +3Radiology Report EvaluationRadiology Report Generation

  4. SAUCE: Summary Analysis Using Conversation Entailment

    Nov 4, 2026Man-Ling Sung, Hemanth Kandula, Jeff Ma +2Summarization EvaluationAutomated Evaluation

  5. ViAdminShift: A Vietnamese Benchmark for Event-Level Knowledge Updates Built from the 2025 Administrative Reform

    Nov 2, 2026Phan Binh Nguyen Lam, Nguyen Thai AnhFactual Knowledge in Language ModelsLanguage Modeling

  6. Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers

    Oct 30, 2026Ziyuan Wang, Genliang Zhu, Xiaobo ChenLanguage Model AuditingText Generation

  7. What Breaks When Facts Become Stories? The Distortions Source-Agreement Metrics Miss When Models Retell Real Events

    Oct 29, 2026Kwangsuk Park, JiWoong Yang, Vaniushkina KseniiaFactual Consistency EvaluationNarrative Generation

  8. NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

    Oct 23, 2026Matt Wilde, Jillian Fisher, Andrew B. Hall +2Generative AI EvaluationFactual Consistency Evaluation

  9. PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

    Oct 15, 2026Dingjie Song, Zhiling Yan, Ziqian Zhang +2Automated Fact-CheckingFactual Consistency Evaluation

  10. Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability

    Oct 5, 2026Zhuomin Chen, Jingchao Ni, Xu Zheng +4Time Series ReasoningTime Series QA

  11. Semantic Uncertainty Quantification Needs Factual Equivalence

    Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language ModelsFactual Consistency Evaluation

  12. MIC: Explaining Image-Claim Inconsistencies in AI-Generated Multimodal Misinformation

    Sep 27, 2026Ruihong Zeng, Jonathan Tonglet, Preslav Nakov +1Factual Consistency EvaluationAutomated Fact-Checking

  13. CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

    Sep 27, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaHallucination in Language ModelsFactual Consistency Evaluation

  14. Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality

    Sep 23, 2026Jiaju Huang, Hao Yang, Xinyu Ma +6Radiology Report GenerationLLM-as-a-Judge

  15. Calibration Is Not Verification: Answer-Level Conformal Guarantees for Mixture-of-Agents

    Sep 22, 2026Nada Rahali, Zijia Wang, Zhi-Song LiuFactual Consistency EvaluationConformal Prediction

  16. Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

    Sep 14, 2026Sarra Gharsallah, Adele Robaldo, Mariia Tokareva +5LLM EvaluationLLM-as-a-Judge

  17. Grounded Adjudication of Variations across Extracted TimeLines (GAVEL): Comparing Clinical Timelines Against Their Case Reports

    Sep 11, 2026Jack Cummins, Sayantan Kumar, Ketan Tamirisa +1Clinical Information ExtractionLLM-as-a-Judge

  18. Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

    Sep 3, 2026Joe Cecil, Marjorie FreedmanHallucination in RAGLLM-as-a-Judge

  19. How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

    Sep 1, 2026Elitsa Yotkova, Violeta Kastreva, Petar Velkov +4LLM Answer VerificationLanguage Model Generation Evaluation

  20. LOOMSUM: Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarization

    Aug 31, 2026Meng Zhou, Wenhao You, Wei Yuan +1Factual Consistency EvaluationText Summarization