LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

ARBITER: Reasoning Trajectory Basins and Majority Vote Failures in Test-Time Sampling
May 25, 2026Meng Cai, Lars Kulik, Farhana ChoudhurySelf-Consistency Decoding Test-Time Scaling
School of Computing and Information Systems University of Melbourne
Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction
May 24, 2026João Sedoc, Baotong Zhang, Dean FosterLLM Answer Verification Selective Prediction
New York University
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
May 24, 2026Yoav Gur-Arieh, Ana Marasović, Mor GevaCoT Faithfulness LLM Evaluation
Tel Aviv University · University of Utah
Graph Alignment Topology as an Inductive Bias for Grounding Detection
May 21, 2026Paul Landes, Pranav Herur, Adam Cross +1LLM Grounding Hallucination in Language Models
†Department of Pediatrics, University of Illinois College of Medicine Peoria · ♣University of Illinois Urbana-Champaign · ♠Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign +1
Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer
May 21, 2026Jewon Yeom, Jaewon Sok, Heejun Kim +3LLM Interpretability Hallucination in Language Models
Graduate School of Data Science, Seoul National University · Department of Rural Systems Engineering, Seoul National University · Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology +1
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models
May 20, 2026Divyaksh Shukla, Ashutosh ModiConfidence Estimation in Language Models Language Model Calibration
Indian Institute of Technology Kanpur (IIT Kanpur)
Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models
May 20, 2026Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat MasoodLanguage Model Safety Evaluation Healthcare
Macquarie University, Sydney, Australia · The University of New South Wales, Sydney, Australia
Reading Calibrated Uncertainty from Language Model Trajectories
May 19, 2026Aliai Eusebi, Alexander Herzog, Xiaoyu Liang +3Confidence Estimation in Language Models LLM Reliability
University College London, London, United Kingdom · Core46, London, United Kingdom
BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation
May 19, 2026Zijun Jia, Yuanchang Ye, Sen Jia +6Retrieval-Augmented Generation Selective Prediction
Beihang University · Zhejiang University of Finance & Economics · Shenzhen Institute of Advanced Technology +1
LLM Benchmark Datasets Should Be Contamination-Resistant
May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1LLM Evaluation Benchmark Contamination
The Pennsylvania State University, University Park, PA, USA.
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
May 19, 2026Xiaoou Liu, Tiejin Chen, Dengjia Zhang +3Uncertainty Quantification Confidence Estimation in Language Models
1Arizona State University · 2Johns Hopkins University · 3Purdue +1
Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
May 19, 2026Tiejin Chen, Longchao Da, Xiaoou Liu +1LLM Evaluation Uncertainty Quantification
School of Computing and Augmented Intelligence, Arizona State University
PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts
May 16, 2026Khizar Hussain, Murat KantarciogluLLM Evaluation Language Model Error Detection
Virginia Tech
Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
May 16, 2026Shuo Liu, Ding Liu, Shi-Ju RanConfidence Estimation in Language Models LLM Reliability
School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.
Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning
May 15, 2026Shireen Kudukkil Manchingal, Abhey Kalia, Fernanda Gonçalves +1LLM Answer Verification Energy-Based Models
Oxford Dynamics · Harwell Science and Innovation Campus, United Kingdom
Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments
May 15, 2026Izabella Krzeminska, Michal Butkiewicz, Ewa KomkowskaLLM Evaluation Psychometric Validation
Orange Research, AI Center, Warsaw, Poland
Calibrating LLMs with Semantic-level Reward
May 15, 2026Fengfei Yu, Ruijia Niu, Dongxia Wu +2Language Model Calibration LLM Reliability
Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA · Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, California, USA · Department of Statistics, Stanford University, Stanford, California, USA
LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
May 14, 2026Philipp Mondorf, Samuel J. Bell, Jesse Dodge +1LLM Evaluation Neural Network Robustness
FAIR at Meta · LMU Munich · Munich Center for Machine Learning +1
Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations
May 13, 2026Qisong He, Jinwei Hu, Xinmiao Huang +3Belief Revision LLM Grounding
School of Computer Science & Informatics, University of Liverpool
Negation Neglect: When models fail to learn negations in training
May 13, 2026Harry Mayne, Lev McKinney, Jan Dubiński +3LLM Reliability LLM Safety
University of Oxford · University of Toronto · 3Warsaw University of Technology +6
Discovery of Hidden Miscalibration Regimes
May 13, 2026Katarzyna Kobalczyk, Mihaela van der SchaarPost-Hoc Calibration Language Model Calibration
University of Cambridge
Proof-Carrying Certificates for LLM Pipelines: A Trust-Boundary Architecture
May 13, 2026George KoomullilLLM Grounding LLM Auditing
Ascendr, Inc.
LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information
May 13, 2026Stef van BuurenLLM Evaluation Confidence Estimation in Language Models
1. TNO - Netherlands Organization for Applied Scientific Research, Leiden · 2. Dept. of Methodology and Statistics, University of Utrecht
Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling
May 13, 2026Adil AminLLM Alignment LLM Interpretability
ZEHEN Labs
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
May 12, 2026Buyun Liang, Jinqi Luo, Liangzu Peng +6Adversarial Prompt Generation Adversarial Attacks
University of Pennsylvania
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
May 12, 2026Zenghui Zhou, Man Li, Xiaoke Fang +3LLM Evaluation Automated Software Testing
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
May 12, 2026Chen Li, Xiaoling Hu, Songzhu Zheng +2Confidence Estimation in Language Models Language Model Calibration
Stony Brook University, NY, USA · Massachusetts General Hospital and Harvard Medical School, MA, USA · Morgan Stanley, NY, USA
Classifier Context Rot: Monitor Performance Degrades with Context Length
May 12, 2026Sam Martin, Fabien RogerLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Anthropic Fellows Program · Anthropic
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
May 12, 2026Xiaoyuan Li, Yuzhe Wang, Moxin Li +6LLM Evaluation Benchmark Construction
University of Science and Technology of China · 2Alibaba Group · 3National University of Singapore
Assessing and Mitigating Miscalibration in LLM-Based Social Science Measurement
May 12, 2026Jinyuan Wang, Ningyuan Deng, Yi YangLanguage Model Calibration LLM Reliability
The Hong Kong University of Science and Technology

May 25, 2026, cs.LGJ/K move · Enter open · S save

ARBITER: Reasoning Trajectory Basins and Majority Vote Failures in Test-Time Sampling

Meng Cai, Lars Kulik, Farhana Choudhury

School of Computing and Information Systems University of Melbourne

Self-Consistency Decoding Test-Time Scaling LLM Reliability LLM Reasoning Majority Voting

Open paper AskPDF arXiv

When language models use test-time sampling, they generate multiple reasoning trajectories and select an answer by majority vote. We show that these trajectories are not independent: for a given question, they concentrate into a small number of clusters, or reasoning basins, each defined by a normalized final answer and the solutions that reach it. A majority vote therefore selects the most stable basin rather than the most accurate one, which creates wrong-majority failures where the correct answer is present but outvoted. We introduce ARBITER, a model-agnostic approach that models interactions between basins using only the base model's own sampled outputs, hidden states, and derived evidence. Most direct correction strategies fail; ARBITER instead uses conservative additive evidence on top of consensus. In its simplest parameter-free form, ARBITER-Δ adds same-model evidence to the majority prior, while ARBITER-Enc augments this with bounded residual signals from hidden states over complete solutions. On GSM8K with Qwen3-4B, consensus over K=24 samples achieves around the mid-94% range, while a same-pool top-2 oracle reaches around the mid-96% range. ARBITER recovers a subset of these cases using zero external information. Across three model families and three math benchmarks, it yields consistent gains with no net-negative cases; for example, on Llama-3.1-8B MMLU-HS-Math, it improves accuracy from the mid-78% range to the mid-82% range, recovering about 22% of the available oracle headroom, indicating that this headroom can be partially recovered from the sample pool itself.

Momentum

Related topics

Latest papers 507