Language Model Error Detection

Overview

Language model error detection studies how to recognize mistakes such as hallucinated facts, faulty reasoning, incorrect code or tool calls, and errors in specialized domains. Methods use output signals such as token probabilities, entropy, and agreement across samples or models, as well as internal activations, probes, retrieval, execution tests, and formal verifiers. Benchmarks span question answering, mathematical reasoning, coding, clinical and legal tasks, and multilingual settings, but results can be distorted by benchmark artifacts or fail to transfer across models and domains. A central challenge is that detecting an error does not ensure that a system can correct it: confident mistakes may be hard to observe, while interventions can introduce new errors.

Latest papers

CardsList

113 papers27 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

  1. Language Models Meet Vibration Testing for LEO Constellation Satellites with Calibrated Response Prediction from Few-show Sweeps

    Nov 16, 2026Jun Wu, Deming Li, Zhenming Zheng +3LLM ReliabilityLanguage Model Error Detection

  2. Clean Tickets, Wrong Orders: Catching Silent Errors with a Language Model in Your Browser Tab

    Nov 13, 2026Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad +2Language Model Error DetectionSemantic Parsing

  3. Auditing AI peer reviewers: a benchmark on real scientific papers

    Nov 12, 2026Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro +2Automated Peer ReviewLanguage Model Error Detection

  4. Detection Is Not Early Warning: Evaluating Token-Level Alarms on Language Model Failures

    Nov 12, 2026Marco De Negri, Luca Sartori, Lorenzo Baggi +5Language Model Error DetectionLanguage Model Probing

  5. Catch It Where It Happens: Real Time Error Correction During LLM Reasoning

    Nov 9, 2026Rotem Elimelech, Michael Shalyt, Ido KaminerLanguage Model Error DetectionLLM Self-Correction

  6. When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation

    Nov 6, 2026Yuehong Cassandra Tai, Qunhua LiLLM-Assisted AnnotationLLM Auditing

  7. Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations

    Nov 5, 2026Xiaobo Chen, Genliang Zhu, Ziyuan Wang +1regression testingAutomated Program Repair

  8. Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency

    Nov 5, 2026Shuyi Miao, Zheng Li, Xu Xie +4Language Model Self-AssessmentLLM Evaluation

  9. Rewrite, Then Veto: Rewriter-Agnostic Edit Gating for LLM-Based Post-OCR Correction

    Nov 3, 2026Jorge Ramón Fonseca CachoOCR Post-ProcessingHistorical Document Processing

  10. MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection

    Nov 3, 2026Omar NusratClinical Language Model EvaluationLanguage Model Error Detection

  11. MemShift: Separating Detection from Repair in LLM-Derived Personal Memory

    Nov 2, 2026Zichuan Xu, Haochen Zhang, Yuyang Dong +2Memory-Augmented Language ModelsLanguage Model Error Detection

  12. From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines

    Nov 1, 2026Hira Naveed, Scott Barnett, Stefanus Kurniawan +4Fault LocalizationRAG Evaluation

  13. Neighborhood Elites: Quality-Diversity Search for Uncertainty Quantification in Question Answering

    Oct 29, 2026Mesut Ceylan, Arda Baris Basaran, Elgar Fleisch +1Uncertainty QuantificationLLM Uncertainty Estimation

  14. TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

    Oct 28, 2026Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad +1LLM EvaluationLanguage Model Reasoning Evaluation

  15. From Error Prediction to Causal Influence: Auditing SAE Features in Multiple-Choice QA

    Oct 27, 2026Aria Nourbakhsh, Adelaide Danilov, Oleksandr Marchenko Breneur +1Sparse AutoencodersCausal Interpretability

  16. INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM

    Oct 26, 2026Timo OksanenLanguage Model Error DetectionDocument Understanding

  17. Adaptive Transfer of Representational Signatures for LLM Reliability Detection

    Oct 24, 2026Yongtong Li, Yixuan Florence Wu, Jiawei Zhang +1Language Model Error DetectionLLM Reliability

  18. B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking

    Oct 21, 2026Meysam Sabbaghan, Doina CarageaLanguage Model Error DetectionStance Detection

  19. Detecting Errors in Large Language Model Responses: The Incremental Predictive Value of Hidden States

    Oct 20, 2026Ningzhi Li, Xiaojun Ma, Xiaofei WangLanguage Model Error DetectionLLM Hallucination Detection

  20. Learning When to Abort in Transformer Planning: A Mechanistic Study

    Oct 18, 2026Shi Feng, Qimin Zhong, Siwei Wang +4Language Model Error DetectionTransformer Interpretability

  21. Self-Guided LLM Reasoning for Error Diagnosis in Vehicle Routing Problems

    Oct 15, 2026Ni Zhang, Jianan Zhou, Yue-Jiao Gong +1Language Model Error DetectionVehicle Routing