LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable
Sep 3, 2026Shai Vardi, João SedocLLM Reliability LLM Decision-Making
University of South Florida, Muma College of Business · New York University
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
Sep 3, 2026Dmitrij ŻatuchinLLM Auditing LLM Reliability
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models
Sep 3, 2026Ross Tieman, Evan MarkouLLM Reliability Language Model Ensembles
Fenner School of Environment & Society · Australian National University · School of Computing
Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversation
Sep 3, 2026Yuhe Wu, Guangyu Wang, Yujie Chen +7Moral Reasoning in Language Models LLM Reliability
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Chinese University of Hong Kong, Shenzhen · 3Dongbei University of Finance and Economics
Knowing Is Not Enough: Information Retrievability as a Precondition to Effective LLM Oversight
Sep 2, 2026Xinyu Fu, Narayan Ramasubbu, Dennis GallettaLLM Reliability Human Oversight of AI
Robinson College of Business Georgia State University 55 Park Place NE, Atlanta, GA 30303, USA · School of Business University of Pittsburgh 3950 Roberto and Vera Clemente Dr, Pittsburgh, PA 15260, USA
Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades
Sep 1, 2026Dushyant RajputLLM Inference Efficiency LLM Answer Verification
AltSlate Labs LLP
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Sep 1, 2026Muhammed Saeed, Simon RazniewskiLLM Evaluation LLM Auditing
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
Towards a Reliable and Practical Eval Pipeline
Sep 1, 2026Emma Thuong Nguyen, Abhishek GhoseLLM Evaluation LLM-as-a-Judge
Salesforce
Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning
Aug 31, 2026Yue Zhou, Yuan Wu, Yi ChangVLM Evaluation Large Vision-Language Models
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University
WildSEEK: Evaluating Language Models for Information-Seeking
Aug 31, 2026Tanise Ceron, Joachim Baumann, Elisa Bassignana +3LLM Safety Benchmarks LLM Sycophancy
1Bocconi University · 2Stanford University · 3IT University of Copenhagen +1
Three Ways Classical Test Theory Can Mislead About LLM Judges
Aug 31, 2026Louis Yiven ZhuLLM-as-a-Judge Psychometric Validation
University of Oxford
Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
Aug 31, 2026Ante Kapetanovic, Tomislav Duricic, Andro Mercep +1LLM Evaluation LLM Reranking
Infobip · Split, Croatia · Zagreb, Croatia
More Capable, Less Faithful: A Multilingual Analysis of Mathematical (Un)Solvability Detection in LLMs
Aug 31, 2026Maria-Eleni Zoumpoulidi, Nikolaos Xiros, Georgios ParaskevopoulosMultilingual Language Model Evaluation LLM Reliability
Institute for Language and Speech Processing, Athena Research Center, Greece
Review Before Trust: Source-Grounded Integrity Gates for AI-Assisted Personal Health Records
Aug 30, 2026Nora Girda, Adrian GrozaData Provenance LLM Reliability
Technical University of Cluj-Napoca, Cluj-Napoca, Romania
EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoning
Aug 30, 2026Peilin Liu, Zhiquan Ji, Jinglong PingEvidence-Grounded Reasoning LLM Reliability
College of Software, Jilin University · School of Urban Planning and Design, Peking University
Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered
Aug 29, 2026Aryo Pradipta Gema, Neel Rajani, Rohit Saxena +2CoT Faithfulness LLM Auditing
University of Edinburgh · University of Edinburgh, Miniml.AI
Validating FKG.in: Soundness Assessment in LLM-Augmented Indian Food Knowledge
Aug 29, 2026Saransh Kumar Gupta, Armaan Shah, Lipika Dey +2LLM Auditing LLM Reliability
Department of Computer Science, Ashoka University, Sonipat, Haryana 131029, India · Mphasis AI and Applied Tech Lab, Ashoka University, Sonipat, Haryana 131029, India · Koita Centre for Digital Health, Ashoka University, Sonipat, Haryana 131029, India +1
Credal Large Language Models for Semantic Commitment under Uncertainty
Aug 24, 2026Shireen Kudukkil Manchingal, Sofiia Nikolenko, Fabio CuzzolinSelective Prediction Confidence Estimation in Language Models
Oxford Dynamics Oxford · Ludwig-Maximilians-Universität München Munich · Institute for Artificial Intelligence, Data Analysis and Systems (AIDAS) School of Engineering Computing & Mathematics Oxford Brookes University, Oxford, UK
Decomposing Wrong-Consensus Agreement in LLM Self-Consistency
Aug 19, 2026Lizhuo Zhang, Mengmeng Tang, Chenfeng Long +2LLM Evaluation LLM Reliability
College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China · Yuelushan Laboratory, Changsha 410128, China · School of Computer Science and Technology, Changsha University of Science and Technology, Changsha 410114, China +1
Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection
Aug 18, 2026Bin Li, Dongdong Wang, Siyang LuPost-Hoc Calibration Online Anomaly Detection
School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China
CAPRI: Contract-Aware Proof Repair for Isabelle
Aug 13, 2026Jim Woodcock, Gabriel Leite, Augusto Sampaio +1LLM Auditing Interactive Theorem Proving
Southwest University, China · Aarhus University, Denmark · University of York, UK +2
Self-Referential Induction Increases Response Instability Relative to Unresolvable and Verifiable Questions in Large Language Models
Aug 13, 2026Paras Balani, Subhrakanta PandaLLM Prompting LLM Reliability
Department of Mathematics and Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India · Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India
Explanatory Engagement Under Rare Anomalous Failure: Asymptotic Rarity in Model Behavior (or: The Asymptotic AI)
Aug 13, 2026Sam MaoLanguage Model Self-Assessment LLM Reliability
New York University | Interactive Media Arts
Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
Aug 12, 2026Haoyuan ZhuInstruction Following LLM Reliability
Department of Electronic and Electrical Engineering University of Sheffield
Is this Citation on Point?
Aug 12, 2026Apurv VermaLegal Citation Verification LLM Reliability
Bloomberg, New York, NY, USA.
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Aug 12, 2026Jiarui Ma, Jianghan Wang, Yuheng Ma +2LLM Evaluation Benchmark Design
Southern University of Science and Technology School of Microelectronics Shenzhen, China
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Aug 12, 2026Sen Xu, Wei Wang, Shixi Liu +5Test-Time Scaling LLM Reliability
Sina Weibo Inc.
Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
Aug 12, 2026Mariya I. VasilevaLLM Evaluation Instruction Following
Work done while at Meta Superintelligence Labs.
EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval
Aug 12, 2026Huiqi Miao, Xinbao Sun, Bo Wang +6Retrieval-Augmented Generation Instruction Following
Jiutian Research, China Mobile, Beijing, China
TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
Aug 11, 2026Valentin Rodionov, Shamil AssylbekovLLM Evaluation LLM Reliability
Intellicat Cleveland, OH 44106

Sep 3, 2026, cs.AIJ/K move · Enter open · S save

Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable

Shai Vardi, João Sedoc

University of South Florida, Muma College of Business · New York University

LLM Reliability LLM Decision-Making

Open paper AskPDF arXiv

Large language models are increasingly used to support organizational decisions, yet users often lack a principled basis for assessing whether to rely on a specific recommendation. Existing approaches typically evaluate broad model properties, such as reliability, uncertainty, or robustness, or focus on user trust, rather than the underlying basis for relying on an individual recommendation. Adapting theoretical foundations from epistemology, we introduce epistemic warrant, a decision-level construct that characterizes the stability of a model's preference and the scope over which that preference holds. We operationalize this construct through a four-tier reliance certificate for pairwise recommendations, distinguishing among unstable, context-dependent, locally supported, and broadly supported recommendations. We validate the construct using contemporary methodologies: known-groups tests successfully recover expert-prespecified warrant orderings, and stronger warrants systematically align with independent consensus from crowd workers. Furthermore, we demonstrate that epistemic warrant provides information distinct from verbalized confidence and is not readily explained by decision difficulty. Ultimately, this framework offers a theoretically grounded, implementable approach for characterizing the warrant of individual LLM recommendations when objective ground truth is unavailable.

Momentum

Related topics

Latest papers 507