LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2LLM Inference ML Reproducibility
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain
Oct 7, 2026Linqi Zhang, Chong Qi, Yan Cheng +4AI-Assisted Decision Making LLM Evaluation
School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalLLM-as-a-Judge Bias in LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy
Oct 7, 2026Jiayu FengLLM Reliability Question Answering
Harvard T.H. Chan School of Public Health
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM Evaluation LLM-Assisted Annotation
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
Oct 6, 2026Jinhyeok Kim, Hye-Young JungLLM Evaluation Multiple-Choice Question Answering
Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Oct 5, 2026Gowthamkumar NandakishoreContrastive Learning LLM Evaluation
Independent researcher
Language models can notice an impossible engineering problem yet still report it as solved
Oct 5, 2026Shaoliang Yang, Jun WangNumerical Reasoning in Language Models Language Model Error Detection
Santa Clara University Santa Clara, CA 95053, USA
Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts
Oct 5, 2026Yanjun Chen, Yongfeng Zhang, Lanjing ZhangLLM Prompting LLM Reliability
Department of Chemical Biology, Ernest Mario School of Pharmacy, Rutgers University, Piscataway, NJ, USA. · East Brunswick High School, East Brunswick, NJ, USA. · Department of Computer Science, Rutgers University, Piscataway, NJ, USA. +3
Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Models
Oct 4, 2026Zhe Yu, Mohan Li, Lei Yu +6LLM Hallucination Mitigation Hallucination in Language Models
Zhejiang University, China · Binjiang Institute of Zhejiang University, China · Guangzhou University, China +5
VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges
Oct 4, 2026Qiushui Xu, Syamil Mohd Razak, Tao Yuan +1LLM-as-a-Judge LLM Fine-Tuning
Penn State University · Amazon
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
When LLMs Sit Above Diagnostic Tools: Unrealized Complementarity in Industrial Fault Diagnosis
Oct 4, 2026Donghwan KimLLM Reliability LLM Tool Use
Independent Researcher
Understanding Errors in LLM-Based Question Answering over Imperfect Tables
Oct 3, 2026Baowen Zhang, Wei Fan, Ruman Wang +1Table QA Language Model Error Detection
University of Wisconsin–Madison · University of Auckland, Auckland, New Zealand · Liaoning Provincial People’s Hospital +1
Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Oct 1, 2026Juan S. SantillanaSupervised Fine-Tuning Tool-Use Evaluation
Independent Researcher
Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
Oct 1, 2026Laura van Weesep, Riccardo Tedoldi, Jens Sjölund +6LLM Auditing LLM-Assisted Annotation
Molecular AI, Discovery Sciences R&D, AstraZeneca · Department of Information Technology, Uppsala University · Robotics, Perception & Learning, KTH Royal Institute of Technology +3
How the Audit Rule Shapes Faithful Factor Explanations in LLMs
Oct 1, 2026Taolin Zhang, Hanyu Wang, Jiuheng Wan +2Counterfactual Explanations Faithfulness of Language Model Explanations
Hefei University of Technology · East China Normal University · Alibaba Cloud Computing
Evaluating LLM-Generated Preference Distributions
Oct 1, 2026Fan Huang, Minsuk Kim, C. Tyler Diggans +1LLM Evaluation Synthetic Data Generation
Indiana University Bloomington Bloomington, IN, USA · Massachusetts Institute of Technology Cambridge, MA, USA · Root Dynamix LLC Sewickley, PA, USA
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Predicting Multi-View Rashomon Representation: Can We Learn Where Models Disagree?
Sep 30, 2026Mingyue Ma, Zongbo Han, Changqing Zhang +1Multi-View Consistency LLM Reliability
Beijing University of Posts and Telecommunications · Tianjin University
Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations
Sep 30, 2026Maximilian von Klinski, Sebastian Lapuschkin, Wojciech Samek +1Deception in Language Models LLM Reliability
Fraunhofer HHI · TU Dublin · TU Berlin +1
Also Small Models Can Reasonably Self-Evaluate Their Confidence
Sep 30, 2026Idil Kapikiran, Thomas Decker, Thomas RunklerConfidence Estimation in Language Models Small Language Models
Technical University of Munich · Siemens AG · LMU Munich +1
Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
Sep 29, 2026Abhinav Rajeev Kumar, Paras ChopraLLM Sycophancy LLM Reliability
Lossfunk
Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Sep 29, 2026Bruno Brocai, Maria BeckerLLM-as-a-Judge Pairwise Preference Evaluation
Heidelberg University German Department
Complexity-Aware Evaluation of LLM Comprehension
Sep 29, 2026Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. AjilaSoftware Engineering LLM Evaluation
Systems and computer Engineering Carleton University Ottawa, Canada
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Sep 29, 2026Zhuo Chen, Hao Zeng, Jiawei Liu +6LLM Evaluation Adversarial Attacks on LLMs
Wuhan University · East China Normal University
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG
Sep 29, 2026Pranav Handa, Ariful AzadRetrieval-Augmented Generation LLM Reliability
Texas A&M University
Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents
Sep 29, 2026Jianchang Su, Yiwei Yang, Wei ZhangAutomated Fact-Checking LLM Reliability
University of Connecticut · University of California, Santa Cruz
How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs
Sep 28, 2026Ali Alfageeh, Rahul Gopinath, Amin AlipourLLM Prompting Few-Shot Prompting
University of Houston · The University of Sydney
A Polyphonic Conception of AI Understanding
Sep 28, 2026Matthieu Queloz, Pierre BeckmannLLM Interpretability LLM Reliability
University of Bern, Department of Philosophy · École Polytechnique Fédérale de Lausanne (EPFL) · Idiap Research Institute +1
Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation
Sep 28, 2026Fahrell Giovanny, Geby Bayuningtyas, Sahrul Mukharom +1Knowledge Conflicts in Language Models Adversarial Attacks on LLMs
Independent Researcher, Tokyo, Japan · Independent Researcher, Jakarta, Indonesia · Department of Informatics, Institut Teknologi Sumatera, Lampung, Indonesia
Semantic Uncertainty Quantification Needs Factual Equivalence
Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language Models Factual Consistency Evaluation
AMIAD, Pˆole Recherche Palaiseau, France
On the Limits of Metacognitive Monitoring in LLMs
Sep 28, 2026Dongqi Han, Yifan Yang, Dongsheng LiLanguage Model Error Detection Confidence Estimation in Language Models
Microsoft Research Asia
Applying Language Models in Clinical Medicine: Recent Trends and Perspectives
Sep 28, 2026Erik AertsLLM Reliability LLM Safety
When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
Sep 28, 2026Yekun Xu, Ante Wang, Jingyi Ren +3Confidence Estimation in Language Models LLM Reliability
College of AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
Toward a Graded Measure of Belief Stability in Large Language Models
Sep 28, 2026Samantha Dies, Branden Fitelson, Tina Eliassi-RadLLM Reliability LLM Uncertainty Estimation
Northeastern University, 360 Huntington Ave, Boston, MA 02115 USA · Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501 USA
Lost with a Map: Conversational State and Behavioral Reliability in Language Models
Sep 27, 2026Atahan Dokme, Larry HeckSpoken Dialogue Systems LLM Reliability
Georgia Institute of Technology
Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
Sep 27, 2026Han Chen, Yingrui LiReasoning Consistency in Language Models LLM Reliability
Independent Researcher
A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Sep 27, 2026Andreas Plesner, Curtis Northcutt, Francisco Guzmán +1Reward Modeling LLM Reliability
ETH Zurich · Handshake AI
Feedback Makes Perfect: A Closed-Loop Framework for NL-to-STL Translation
Sep 27, 2026Bowen Ye, Xiang YinSignal Temporal Logic LLM Reliability
Shanghai Jiao Tong University Shanghai, China
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Sep 27, 2026Krishna Chytanya AyyagariLLM Evaluation LLM-as-a-Judge
Google
Beyond Average Safety: Chance-Constrained LLM Fine-tuning
Sep 24, 2026Taha Entesari, Mahyar FazlyabStochastic Optimization Chance-Constrained Optimization
Department of Electrical and Computer Engineering Johns Hopkins University
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Sep 24, 2026Douglas LeithAI Agent Reliability Data Provenance
Trinity College Dublin Dublin, Ireland
Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs
Sep 23, 2026Haitong Jiang, Chunlin Liu, Yile Wang +1LLM Self-Correction Controllable Text Generation
College of Computer Science and Software Engineering, Shenzhen University
Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Sep 22, 2026Gaoyuan Du, Anam Nawaz Khan, Rex Zhou +4Language Model Decoding LLM Inference
University of Tennessee, Knoxville · University of Chicago · Amazon +1
Calibration as a First-Class Criterion in LLM Evaluation
Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLLM Evaluation Confidence Estimation in Language Models
Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA
On the security and privacy of LLMs in Mobility
Sep 22, 2026Mauro Conti, Lorenzo Perinello, Umberto SalviatiIntelligent Transportation Systems LLM Security
University of Padova Italy · Fondazione Bruno Kessler Italy
Clarification Is Not Correction: LLMs Fail to Let Go
Sep 21, 2026Jianzhe Lin, Xiaolin Li, Fei Wang +3LLM Reliability LLM Context Management
MetaAI
LLJ Cards: Best practices for the Use of LLMs as Judges
Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM Evaluation LLM-as-a-Judge
McGill University · Mila - Quebec AI Institute · Concordia University
ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation
Sep 21, 2026Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang +3VLM Evaluation LLM Evaluation
CSU-JPG, Central South University · National University of Singapore
Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
Sep 20, 2026Akash ChavanConstrained Decoding LLM Evaluation
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
Sep 18, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidHuman-in-the-Loop Evaluation LLM Reliability
Citigrounp, Inc, London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom
An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language Models Language Model Self-Assessment
DreamAI
Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute
Sep 17, 2026Gunwoo Lee, Changmin Sung, Sang-Hwan Gwak +3LLM Evaluation Selective Prediction
Large-scale AI Research Center, Korea Institute of Science and Technology Information (KISTI), 245 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea. · Department of Applied AI, University of Science and Technology (UST), 217 Gajeong-ro, Yuseong-gu, Daejeon, 34113, Republic of Korea.
Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaLLM Evaluation LLM-Assisted Annotation
KIT · Department of Sociology and Computational Sociology, Karlsruhe Institute of Technology (KIT), Douglasstr. 24, 76133 Karlsruhe, Germany · ALMAnaCH, médialab, Sciences Po +2
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction
Sep 17, 2026Theodore O. CochranNumerical Reasoning in Language Models LLM Reliability
AI for Altruism
Playing log(N)-Questions over Wikipedia Abstracts: How Per-Round Errors Compound Under Information Asymmetry
Sep 16, 2026Peter PotashLLM Evaluation Pragmatic Reasoning in Language Models
WaveTLM: Reliable Time-Series Language Modeling through Task Compilation
Sep 16, 2026Jiahui Chen, Bingke Zhu, Hongyu Pan +1LLM Reliability Time-Series Foundation Models
Institute of Automation, Chinese Academy of Sciences Beijing, China

Oct 7, 2026, cs.PFJ/K move · Enter open · S save

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu+1

Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland+1

LLM Inference ML Reproducibility LLM Evaluation LLM Reliability

Open paper AskPDF arXiv

Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.

Momentum

Related topics

Latest papers 507