LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
Jun 2, 2026Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1Inter-Rater Reliability Multilingual Language Model Evaluation
Microsoft Research
Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Jun 1, 2026Simone Caldarella, Davide Talon, Rahaf Aljundi +2LLM Evaluation Overthinking in Language Models
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection
May 31, 2026Bohan Yang, Yijun Gong, Zhi Zhang +3Language Model Error Detection LLM Reliability
Binjiang Institute of Zhejiang University · Beijing Normal-Hong Kong Baptist University · Great Bay University +1
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
May 30, 2026Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +1LLM Evaluation Language Model Generation Evaluation
Northeastern University, Massachusetts, USA · University of Southern California, California, USA
Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Models
May 30, 2026S M Tahmid Siddiqui, Akib Jawad Ononto, Anoop Singhal +1LLM Hallucination Mitigation Question Answering
The University of Texas at Dallas, Richardson, TX 75080, USA · National Institute of Standards and Technology, Gaithersburg, USA
Sequential statistical inference for Large Language Models: Representation, validity, and monitoring
May 30, 2026Yao XieLanguage Model Calibration LLM Reliability
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology
Score $\times$ Decoder: A Unified View of Unsupervised Inference-Time Scaling for Hallucination Mitigation
May 30, 2026Yun-Chen Cheng, Che-Yu Lin, Cheng-Lin YangLanguage Model Decoding LLM Hallucination Mitigation
CyCraft AI Lab, Taiwan
Capability Self-Assessment: Teaching LLMs to Know Their Limits
May 29, 2026Haoyan Yang, Reza Shirkavand, Yukai Jin +3LLM Evaluation Language Model Self-Assessment
Stony Brook University · University of Maryland · Florida State University
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3Multilingual Language Model Evaluation Language Model Error Detection
Department of Computer Science and Engineering, Islamic University of Technology · Department of Computer Science and Engineering, University of California
LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
May 29, 2026Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij +1Multilingual Language Model Evaluation LLM-as-a-Judge
National Research Council, Canada · IMDA, Singapore
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
May 29, 2026Tom Lucas, Alessio Buscemi, Alfredo Capozucca +2LLM Evaluation AI Accountability
Luxembourg Institute of Science and Technology (LIST), Luxembourg, University of Luxembourg, Luxembourg, and Scriptor Artis, France · Luxembourg Institute of Science and Technology (LIST), Luxembourg · University of Luxembourg, Luxembourg +1
Toxic HallucinAItions: Perturbing Prompts and Tracing LLM Circuits
May 29, 2026Soorya Ram Shimgekar, Agam Goyal, Amruta Parulekar +6Prompt Sensitivity LLM Interpretability
University of Illinois Urbana-Champaign · Nimblemind
The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability
May 28, 2026Mikhail L. Arbuzov, Lee Mosbacker, Sisong Bei +3LLM Reliability
Independent Researcher · Palo Alto Networks
Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs
May 28, 2026David Gros, Adam GleaveLanguage Model Safety Evaluation Adversarial Attacks on LLMs
SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
May 28, 2026Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +1AI for Science LLM Evaluation
University of Maryland, College Park
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
May 28, 2026Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1Multilingual Language Model Evaluation LLM Evaluation
National University of Singapore
Harnessing non-adversarial robustness in large language models
May 28, 2026Qinghua Zhou, Ellina Aleshina, Andrey Lovyagin +6LLM Fine-Tuning Neural Network Robustness
International Joint Laboratory of AI for Industry, QUST, Qingdao, China · King’s College London, London, UK · Applied AI Institute, Moscow, Russia +2
Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
May 27, 2026Gabrielle Kaili-May Liu, Arman CohanConfidence Estimation in Language Models Language Model Calibration
Yale University
Entropy Distribution as a Fingerprint for Hallucinations in Generative Models
May 27, 2026Mattia J. Villani, Pranav Deshpande, Akshay Seshadri +2Language Model Error Detection Language Model Generation Evaluation
Global Technology Applied Research, JPMorganChase, New York, NY 10001, USA
The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
May 27, 2026Eric Onyame, Runtao Zhou, Kowshik Thopalli +2CoT Faithfulness Multilingual Language Model Evaluation
University of Virginia · Lawrence Livermore National Laboratory
A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving
May 26, 2026Sahil KadadekarLLM Inference Language Model Safety Evaluation
Independent Researcher.
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
May 26, 2026Hankyeol Kim, Pilsung KangLLM Evaluation Confidence Estimation in Language Models
Seoul National University
Chain-based Adaptive Reconfiguration Over Lattices for Hallucination Reduction
May 26, 2026Joan Vendrell Gallart, Solmaz Kia, Russell Bent +1LLM Hallucination Mitigation Hallucination in Language Models
Department of Mechanical and Aerospace University of California Irvine Irvine, CA 92617-4322, USA · T-5 Los Alamos National Laboratory Los Alamos, NM 88220, USA · CAI-4 Los Alamos National Laboratory Los Alamos, NM 88220, USA
Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination
May 26, 2026Yedidia Agnimo, Anna Korba, Annabelle Blangero +2LLM Evaluation Hallucination in Language Models
Ekimetrics · Centre Inria de l’Université Grenoble Alpes · CREST, ENSAE +1
Innovation: An Almost Characterization of Hallucination
May 26, 2026Nishant P. Das, Piyush SrivastavaHallucination in Language Models Language Model Calibration
School of Technology and Computer Science, Tata Institute of Fundamental Research, Mumbai, Maharashtra - 400 005, India.
SeDT: Sentence-Transformer Decision-Transformer Conditioning for Multi-Turn Conversation Reliability
May 26, 2026Ramakrishna Vamsi Setti, Jagadeesh Rachapudi, Sachin Chaudhary +2LLM Reliability Language Model Robustness
Independent Researcher · 2Drone Lab, IIT Mandi · 3UPES, Dehradun
The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context
May 26, 2026Zhe Yu, Wenpeng Xing, Yunzhao Wei +4Memorization in Language Models LLM Interpretability
Binjiang Institute of Zhejiang University · Zhejiang University · National Fintech Evaluation Center +2
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
May 26, 2026Yong-eun ChoAI Agent Reliability LLM Agent Evaluation
KailosLab · Seoul, Republic of Korea
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3

Jun 2, 2026, cs.CLJ/K move · Enter open · S save

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

Sourabrata Mukherjee, Hamna Hamna, Kalika Bali, Sunayana Sitaram

Microsoft Research

Inter-Rater Reliability Multilingual Language Model Evaluation LLM Evaluation LLM-as-a-Judge LLM Reliability

Open paper AskPDF arXiv

LLM judges now score most open-ended NLP output, and their mutual agreement is routinely read as evidence that the scores can be trusted. That reading is unsafe: judges may agree because they capture quality, or because they share the same blind spots, and agreement statistics alone cannot tell these apart. We develop a geometric test that can. Treating each judge's scores as a vector, we measure spread, effective rank, the angle to human scores, and the judge-judge, judge-human, and human-human agreement triple for 42 judges on two community-built Indic benchmarks covering four domains and eight languages. Every comparison is reference-matched: a judge and a held-out rater are scored against the same two-rater mean, since an averaged reference otherwise flatters judges by several degrees. On subjective rubrics, judges agree with one another as much as humans do yet reach only 58-66% of human agreement and often concentrate on an axis humans do not weight. On the one rubric with a verifiable answer, most of that gap closes. Ensembles converge on the judges' shared axis rather than the human one, and training widens scores without rotating them. Inter-judge agreement is evidence of human alignment only after this check passes.

Momentum

Related topics

Latest papers 507