LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs
Aug 11, 2026Utkarsh BahugunaLLM Reliability LLM Uncertainty Estimation
Scaler School of Technology
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
Aug 11, 2026Zhuang WangFault-Tolerant Distributed Training LLM Reliability
Independent Researcher
ReLTEx: Reliable LLM-based Taxonomy Expansion
Aug 11, 2026Zeinab Ghamlouch, Mehwish AlamLLM Hallucination Mitigation LLM Reliability
Télécom Paris, Institut Polytechnique de Paris, France
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
Logit-Boundary Geometric Belief Interfaces and Sparse Sheaf-Enclave Protocols: A Self-Contained Substrate for Secure Network Electronic Health Record (EHR) Interoperability
Aug 10, 2026Alvin Spivey, Yu HuangLLM Reliability Electronic Health Records
Light Imaging Technologies, Inc.
Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation System
Aug 10, 2026Yanlin YuLLM Reliability Machine Translation
University of Tsukuba, Japan · Doctoral Program in International Public Policy · Degree Programs in Humanities and Social Sciences
Theory-Guided Deception Detection: A RAG-Based Artificial Intelligence Exploration
Aug 9, 2026David M. Markowitz, Timothy R. LevineRetrieval-Augmented Generation Deception in Language Models
Department of Communication, Michigan State University, East Lansing, MI 48824 · Department of Communication, University of Oklahoma, Norman, OK 73019
Can Open-Weight Models Compete on Financial Text Comprehension?
Aug 9, 2026Jan SpörerLLM Evaluation Financial QA
University of St. Gallen, Switzerland
Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing
Aug 9, 2026Minhan Cho, Jimin KweonLLM Evaluation Reasoning Consistency in Language Models
Remember & Company AI Lab · DSAIL, Dept. of Applied AI · Sungkyunkwan University +2
On the Robustness of LLMs' Internal Representation of Code Correctness
Aug 8, 2026Francisco Ribeiro, Sohaila Abdulsattar, Renata Gonzalez +2LLM Reliability Language Model Robustness
New York University Abu Dhabi Abu Dhabi, United Arab Emirates
REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment
Aug 8, 2026Zhengze Huang, Luyang Yu, Di Hong +5LLM Hallucination Mitigation LLM Reliability
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Fudan University · The Hong Kong Polytechnic University
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
Aug 7, 2026Yiyao Zhang, Diksha Goel, Hussain Ahmad +2LLM-as-a-Judge LLM Reliability
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia · CSIRO’s Data61, Clayton, VIC, Australia · School of Computer Science and Information Technology, Adelaide University, Adelaide, SA, Australia
Learning When to Trust via Selective Context Preference Optimization
Aug 6, 2026Xian Sun, Wei Chow, Yingshuo Wang +4Knowledge Conflicts in Language Models LLM Reliability
1Duke University · 2National University of Singapore · 3UC Berkeley +3
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Aug 6, 2026Ro Encarnación, Tina Behzad, Emma Lurie +1LLM Safety Benchmarks LLM Evaluation
University of Pennsylvania · Stony Brook University
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Aug 6, 2026Hamed Damirchi, Ignacio Meza De la Jara, Damith Ranasinghe +2Language Model Error Detection LLM Interpretability
Australian Institute for Machine Learning · Adelaide University · Naval Group Pacific +1
SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specifications
Aug 5, 2026Yixuan Wang, Licheng Luo, Yu Fu +3Selective Prediction LLM Reliability
University of California, Riverside, Riverside, CA, USA · City University of Hong Kong, Hong Kong SAR, China
PPDL: LLM-Based Flows as Probabilistic Programs
Aug 5, 2026Louis Mandel, Guillaume Baudart, Mandana Vaziri +1LLM Inference LLM Reliability
IBM, New York, USA · Université Paris Cité, Inria, CNRS, IRIF, France
Relational Response Fields: A General Theory of Black-Box LLM Response Consistency and Recovery
Aug 5, 2026Song ZichenLLM Reliability Corruption Robustness
Sungkyunkwan University
The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
Aug 4, 2026Shashwat Sourav, Aishwarya BalwaniLLM Reliability Reasoning Trace Analysis
Department of Physics, Washington University in St. Louis · Department of Developmental Neurobiology, St. Jude Children’s Research Hospital
ANCHOR-RE: An Agentic Neuro-Symbolic Framework for Grounded Biomedical Relation Extraction
Aug 4, 2026Shufan Ming, Yikun Han, Gibong Hong +2Relation Extraction LLM Grounding
School of Information Sciences, University of Illinois Urbana-Champaign, 501 E Daniel St., Champaign, 61820, IL, USA · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 516 Delaware St SE, Minneapolis, 55455, MN, USA
Chat Debugging: An Exploratory Study of Human-AI Collaboration to Debug Analog Circuits
Aug 3, 2026John Hu, Andrew AshLLM Reliability Human-AI Collaboration
School of Electrical and Computer Engineering Oklahoma State University Stillwater, OK 74078, USA
Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
Aug 3, 2026Xuan Ren, Weiqi Zhai, Tianle Pu +4LLM Answer Verification Reasoning Evaluation
1Alibaba Group · 2Alibaba DAMO Academy
How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Models
Aug 3, 2026Andres Algaba, Francesca Carlon, Lynn Delcon +3LLM Evaluation LLM Interpretability
1Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · 2imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussel, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
Conformity Mitigations in Large Language Models Lie on a Single Resistance-Receptivity Frontier
Aug 3, 2026Zafar Hussain, Kristoffer NielboLLM Evaluation Normative Conformity in Language Models
Aarhus University
EntailLLM: Verifying LLM-Generated Vulnerability Discovery Paths with Domain Knowledge via Logic Programming
Aug 3, 2026Kaustuv Mukherji, Jaikrishna Manojkumar Patil, Colton Payne +4Logic Programming Software Vulnerability Detection
Syracuse University · HRL Laboratories
RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings
Aug 2, 2026Raj Shekhar SinghRetrieval-Augmented Generation Long-Form Document Generation
Indian Institute of Technology, Roorkee Roorkee, Uttarakhand, India
SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Scheduling
Aug 2, 2026Shrenil Shaun Sharma, Avi SharmaCombinatorial Optimization LLM Reliability
Department of Electrical Engineering and Computer Sciences University of California, Berkeley
GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs
Aug 1, 2026Xuechen LiRemote Sensing Image Understanding Retrieval-Augmented Generation
University of Minnesota, Twin Cities
Distilling Reasoning Traces into Advisory Prompts for Software Engineering Tasks
Aug 1, 2026Faizan Faisal, Prem Devanbu, Toufique AhmedReasoning Distillation LLM Prompting
University of California, Davis · IBM
SymboUQ: Symbolic Uncertainty Quantification for Spatial Reasoning in LLMs
Aug 1, 2026Dahai Yu, Lin Jiang, Rongchao Xu +1Uncertainty Quantification LLM Reliability
Florida State University, Tallahassee, Florida

Aug 11, 2026, cs.AIJ/K move · Enter open · S save

When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs

Utkarsh Bahuguna

Scaler School of Technology

LLM Reliability LLM Uncertainty Estimation Majority Voting

Open paper AskPDF arXiv

Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer. On the full GPQA Diamond benchmark (198 graduate-level science questions), majority voting reduces per-problem accuracy on a majority of problems for two instruction-tuned models from different families: 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B, with Qwen the primary demonstration and Llama corroborating the direction from a near-chance baseline. The effect was pre-registered on a 151-problem confirmatory split after being observed on 47 exploratory problems, and all four confirmatory hypotheses passed. A grid oracle that routes each problem to the best N across {1, 2, 4, 8, 16, 32, 64} marks a theoretical upper bound 14 accuracy points above N = 1 for Qwen and 17 for Llama, an oracle bound requiring ground truth rather than a deployable method. No verifier-free gate reaches it: neither a plurality-agreement gate nor a token-entropy gate moves accuracy more than 0.002 from fixed-budget voting at N = 64. The mechanism is direct: confidence does not track correctness on these problems. In the highest-agreement bin the plurality answer is correct about half the time for Qwen, and for Llama that bin is less accurate than its lowest-agreement bin. We pre-register and confirm these findings on small instruction-tuned models; we do not test reasoning-native models, which we flag as the central open question.

Momentum

Related topics

Latest papers 507