LLM Reliability

LLM: Large Language Model

Overview

It asks not only whether a model gets an answer right, but whether it knows when it may be wrong, follows constraints, uses evidence faithfully, and behaves consistently across repeated or multi-turn use. Researchers test these properties with adversarial and perturbation benchmarks, repeated-run and cross-language evaluations, domain-specific tasks, and audits of LLM judges and tool-using agents. Common approaches include estimating uncertainty from logits, hidden representations, or response agreement; selective abstention and conformal calibration; retrieval and symbolic verification; and training methods that reward grounded answers, correction, or hesitation. The central tension is that benchmark accuracy and verbal confidence often fail to predict reliability on individual cases, long tasks, or shifted inputs, while stronger checks and abstention trade coverage, latency, and cost for safer decisions.

Latest papers

CardsList

866 papers92 papers in the last four weeks, down 21% on the four weeks before. 0.7% of all new papers.

  1. Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

    Nov 19, 2026Terence D. Agbeyegbe, Toneukarin G. AgbeyegbeLLM ReliabilitySelf-Consistency in Language Models

  2. Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance

    Nov 19, 2026Yinuo Liu, Yunlong Lyu, Zichao Luo +4AI-Assisted Software DevelopmentSoftware Maintenance

  3. Coordinated Use of Large Language Models in Academic Research

    Nov 19, 2026Vladislav MarinovMulti-LLM CollaborationLLM Reliability

  4. Reliability is a user-specific property of large language models

    Nov 18, 2026Xingyu ZhaoLLM ReliabilityLLM Evaluation

  5. When Do LLMs Model the Right Target? Problem Formulation in High-Stakes Risk Prediction

    Nov 18, 2026Jennifer Zhang, Amanda CostonLLM Reliability

  6. FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

    Nov 17, 2026Yi-Cheng Lai, Hen-Hsen HuangKnowledge Conflicts in Language ModelsLLM Evaluation

  7. Scientific Coding with Large Language Models: Evidence from Finite Element Analysis

    Nov 16, 2026Erfan Hamdi, Joel Shor, Emma LejeuneScientific Code GenerationLLM Reliability

  8. Language Models Meet Vibration Testing for LEO Constellation Satellites with Calibrated Response Prediction from Few-show Sweeps

    Nov 16, 2026Jun Wu, Deming Li, Zhenming Zheng +3LLM ReliabilityLanguage Model Error Detection

  9. NEBench R1: a Reasoning Benchmark for Evaluating Nuclear Domain Competence in Large Language Models

    Nov 15, 2026Liyan Wang, Longze Li, Mohamed Hassan +4Language Model Reasoning EvaluationLLM Reliability

  10. The Silent Cost of Conversation History in LLM Reasoning

    Nov 15, 2026Ali ŞenolMulti-Turn Language Model ReasoningLLM Reliability

  11. Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems

    Nov 15, 2026Thilo Tamme, Anton Hantel, Jakob ScheppersLLM AuditingLLM Reliability

  12. Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety

    Nov 13, 2026Christopher M. Stewart, Kelsey Kraus, J. Elliott Casal +1Language Model Safety EvaluationLLM Reliability

  13. Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models

    Nov 13, 2026Sujato Dutta, Prithviraj Patil, Chandrajit BajajLLM ReliabilityKnowledge Conflicts in Language Models

  14. Maximizing LLM Utility While Managing LLM Fallibility

    Nov 12, 2026Ekta U. Samani, Christopher G. AtkesonLLM ReliabilityLLM Decision-Making

  15. When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision

    Nov 12, 2026Yupeng Tang, Mingfeng LinLLM ReliabilityLLM Evaluation

  16. Benchmarking AI risk behaviors

    Nov 12, 2026Luhan Mikaelson, Robert Adragna, Emmett Brosowsky +1Behavioral Consistency of LLM AgentsLanguage Model Safety Evaluation

  17. Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

    Nov 11, 2026Xin Liu, Xu Han, Yuxin Wang +3LLM Reliability

  18. Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines

    Nov 10, 2026Mo Zhou, Yiwei ZhangFraming Effects in Language ModelsLLM Reliability