LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
May 11, 2026Jasmine Qi, Danylo Dantsev, Muyang SunPost-Hoc Calibration LLM-as-a-Judge
Indeed Inc.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1LLM Evaluation LLM Reliability
Carnegie Mellon University
Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
May 11, 2026Nikita Kezins, Urbas Ekka, Pascal Berrang +1Neural Network Verification Language Model Safety Evaluation
Delft University of Technology · University of Birmingham & Zeroth Research
ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Models
May 11, 2026Wentao Qiu, Guanran Luo, Zhongquan Jian +3Abductive Reasoning Bayesian Networks
School of Informatics, Xiamen University, Xiamen, China · School of Computer and Data Science, Minjiang University, Fuzhou, China · School of Film, Xiamen University, Xiamen, China +2
Trust or Abstain? A Self-Aware RAG Approach
May 11, 2026Xi Zhu, Ziqi Wang, Kai Mei +5Retrieval-Augmented Generation Knowledge Conflicts in Language Models
Rutgers University · University of Illinois Urbana-Champaign
Can We Trust LLMs for Mental Health Screening? Consistency, ASR Robustness, and Evidence Faithfulness
May 10, 2026Erfan Loweimi, Sofia de la Fuente Garcia, Samira Loveymi +2Healthcare Mental Health
Usher Institute, Edinburgh Medical School, University of Edinburgh, Edinburgh, UK · Department of Computer Engineering, Ahvaz Campus, Islamic Azad University, Ahvaz, Iran · School of Health and Social Care, Edinburgh Napier University, Edinburgh, UK
APCD: Adaptive Path-Contrastive Decoding for Reliable Large Language Model Generation
May 10, 2026Tianyu Zheng, Hong Wu, Jiaji ZhongLanguage Model Decoding LLM Reliability
University of Electronic Science and Technology of China
The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
May 9, 2026Rania Elbadry, Ahmed Heakl, Fan Zhang +4LLM Interpretability LLM Reliability
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · INSAIT, Sofia University
A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability
May 9, 2026Hamed Omidvar, Vahideh AkhlaghiLLM Inference Efficiency AI Agent Reliability
INTELLERCE LLC San Jose, CA, USA
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
May 9, 2026Xiang Feng, Jiawei Zhou, Zhangfeng Huang +6LLM Grounding LLM Answer Verification
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · Alibaba Group, Hangzhou, China · Independent Researcher +1
A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
May 8, 2026Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3LLM Evaluation Confidence Estimation in Language Models
University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH
Sanity Checks for Long-Form Hallucination Detection
May 8, 2026Geigh Zollicoffer, Minh Vu, Hongli Zhan +2LLM Evaluation Hallucination Detection
Los Alamos National Laboratory · The University of Texas at Austin · University of British Columbia
Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
May 8, 2026Sree Bhattacharyya, Samarth Khanna, Leona Chen +3LLM Evaluation Confidence Estimation in Language Models
Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.
Reliable Chain-of-Thought via Prefix Consistency
May 8, 2026Naoto Iwase, Yuki Ichihara, Mohammad Atif Quamar +1Self-Consistency Decoding Reasoning Consistency in Language Models
Nagoya University · Nara Institute of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence +1
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5LLM Alignment Direct Preference Optimization
Zhejiang University · Université de Montréal · Sun Yat-sen University +1
Limits of Reliability and Scaling in Language Models
May 8, 2026Subhabrata MajumdarLanguage Model Scaling Laws Autoregressive Language Modeling
Indian Institute of Management Bangalore Bengaluru, India
Hallucination Detection via Activations of Open-Weight Proxy Analyzers
May 8, 2026Akshita Singh, Prabesh Paudel, Siddhartha RoyHallucination in Language Models LLM Reliability
Khoury College of Computer Sciences Northeastern University
The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
May 8, 2026Zekai Tong, Ruiyao Xu, Aryan Shrivastava +2LLM Evaluation LLM Reliability
University of Chicago · 2Northwestern University
Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
May 7, 2026Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg +4Language Model Error Detection Language Model Generation Evaluation
University of Illinois Chicago · Microsoft
Towards Generation-Efficient Uncertainty Estimation in Large Language Models
May 7, 2026Mingcheng Zhu, Yu Liu, Tingting ZhuLLM Inference Efficiency Uncertainty Quantification
University of Oxford
Knowing but Not Correcting: Routine Task Requests Suppress Factual Correction in LLMs
May 7, 2026Zixuan Chen, Hao Lin, Zizhe Chen +6LLM Self-Correction LLM Reliability
1Shanghai Jiao Tong University · 2The Chinese University of Hong Kong · 3Ant Group
Hallucination as an Anomaly: Dynamic Intervention via Probabilistic Circuits
May 7, 2026Erik Nielsen, Elia Cunegatti, Marcus Vukojevic +1Probabilistic Circuits LLM Hallucination Mitigation
Department of Information Engineering and Computer Science University of Trento Trento, Italy
ReFlect: An Effective Harness System for Complex Long-Horizon LLM Reasoning
May 7, 2026Fan HuangLLM Self-Correction LLM Reliability
Indiana University Bloomington
Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals
May 6, 2026Gijs van DijkLanguage Model Error Detection Confidence Estimation in Language Models
Utrecht University
NoisyCoconut: Counterfactual Consensus via Latent Space Reasoning
May 6, 2026Michael Jerge, David EvansSelective Prediction LLM Inference
University of Virginia
Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
May 6, 2026Aofan Liu, Jingxiang MengPrompt Sensitivity LLM Evaluation
1Peking University. · University of Chicago.
MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following
May 5, 2026Jaeyun Lee, Junyoung Koh, Zeynel Tok +2LLM Evaluation LLM-as-a-Judge
University of Oxford · Yonsei University
Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference
May 5, 2026Yi LiuLLM Inference Test-Time Scaling
York University
Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
May 4, 2026Yucheng DuRepresentation Geometry LLM Reliability
University of Southern California
A multilingual hallucination benchmark: MultiWikiQHalluA
May 4, 2026Freja Thoresen, Dan Saattrup SmartMultilingual Language Models Multilingual Language Model Evaluation
Alexandra Institute · Rued Langgaards Vej 7, 2300 København

May 11, 2026, cs.LGJ/K move · Enter open · S save

VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference

Jasmine Qi, Danylo Dantsev, Muyang Sun

Indeed Inc.

Post-Hoc Calibration LLM-as-a-Judge LLM Answer Verification Language Model Generation Evaluation Confidence Estimation in Language Models LLM Reliability+1

Open paper AskPDF arXiv

LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be trusted. Token log-probabilities, the standard post-hoc confidence signal, are unavailable for many commercial LLMs and, even when accessible, saturate above 0.999 with structured JSON output. We introduce VERDI (VERification-Decomposed Inference), a method that extracts confidence from the reasoning trace a structured judge already produces, with no additional inference calls. VERDI decomposes each verification-style evaluation into sub-checks and derives three structural signals: Step-Verdict Alignment, Claim-Level Margin, and Evidence Grounding Score. We combine them with Platt-scaled logistic regression. On three public benchmarks, VERDI achieves AUROC 0.72-0.91 on GPT-4.1-mini and 0.66-0.80 on GPT-5.4-mini. On Qwen3.5-4B/9B/27B, where answer-token logprobs are anti-calibrated (higher confidence on errors, AUROC 0.32-0.49), VERDI achieves 0.56-0.70. We additionally validate on a production system with eight rubrics (AUROC 0.73-0.88 on factual rubrics), demonstrate cross-model transfer (AUROC 0.66-0.69), and show that a 33M-parameter NLI (Natural Language Inference) model provides a scalable alternative to regex extraction.

Momentum

Related topics

Latest papers 507