LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2LLM Inference ML Reproducibility
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain
Oct 7, 2026Linqi Zhang, Chong Qi, Yan Cheng +4AI-Assisted Decision Making LLM Evaluation
School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalLLM-as-a-Judge Bias in LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy
Oct 7, 2026Jiayu FengLLM Reliability Question Answering
Harvard T.H. Chan School of Public Health
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM Evaluation LLM-Assisted Annotation
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
Oct 6, 2026Jinhyeok Kim, Hye-Young JungLLM Evaluation Multiple-Choice Question Answering
Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Oct 5, 2026Gowthamkumar NandakishoreContrastive Learning LLM Evaluation
Independent researcher
Language models can notice an impossible engineering problem yet still report it as solved
Oct 5, 2026Shaoliang Yang, Jun WangNumerical Reasoning in Language Models Language Model Error Detection
Santa Clara University Santa Clara, CA 95053, USA
Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts
Oct 5, 2026Yanjun Chen, Yongfeng Zhang, Lanjing ZhangLLM Prompting LLM Reliability
Department of Chemical Biology, Ernest Mario School of Pharmacy, Rutgers University, Piscataway, NJ, USA. · East Brunswick High School, East Brunswick, NJ, USA. · Department of Computer Science, Rutgers University, Piscataway, NJ, USA. +3
Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Models
Oct 4, 2026Zhe Yu, Mohan Li, Lei Yu +6LLM Hallucination Mitigation Hallucination in Language Models
Zhejiang University, China · Binjiang Institute of Zhejiang University, China · Guangzhou University, China +5
VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges
Oct 4, 2026Qiushui Xu, Syamil Mohd Razak, Tao Yuan +1LLM-as-a-Judge LLM Fine-Tuning
Penn State University · Amazon
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
When LLMs Sit Above Diagnostic Tools: Unrealized Complementarity in Industrial Fault Diagnosis
Oct 4, 2026Donghwan KimLLM Reliability LLM Tool Use
Independent Researcher
Understanding Errors in LLM-Based Question Answering over Imperfect Tables
Oct 3, 2026Baowen Zhang, Wei Fan, Ruman Wang +1Table QA Language Model Error Detection
University of Wisconsin–Madison · University of Auckland, Auckland, New Zealand · Liaoning Provincial People’s Hospital +1
Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Oct 1, 2026Juan S. SantillanaSupervised Fine-Tuning Tool-Use Evaluation
Independent Researcher
Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
Oct 1, 2026Laura van Weesep, Riccardo Tedoldi, Jens Sjölund +6LLM Auditing LLM-Assisted Annotation
Molecular AI, Discovery Sciences R&D, AstraZeneca · Department of Information Technology, Uppsala University · Robotics, Perception & Learning, KTH Royal Institute of Technology +3
How the Audit Rule Shapes Faithful Factor Explanations in LLMs
Oct 1, 2026Taolin Zhang, Hanyu Wang, Jiuheng Wan +2Counterfactual Explanations Faithfulness of Language Model Explanations
Hefei University of Technology · East China Normal University · Alibaba Cloud Computing
Evaluating LLM-Generated Preference Distributions
Oct 1, 2026Fan Huang, Minsuk Kim, C. Tyler Diggans +1LLM Evaluation Synthetic Data Generation
Indiana University Bloomington Bloomington, IN, USA · Massachusetts Institute of Technology Cambridge, MA, USA · Root Dynamix LLC Sewickley, PA, USA
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Predicting Multi-View Rashomon Representation: Can We Learn Where Models Disagree?
Sep 30, 2026Mingyue Ma, Zongbo Han, Changqing Zhang +1Multi-View Consistency LLM Reliability
Beijing University of Posts and Telecommunications · Tianjin University
Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations
Sep 30, 2026Maximilian von Klinski, Sebastian Lapuschkin, Wojciech Samek +1Deception in Language Models LLM Reliability
Fraunhofer HHI · TU Dublin · TU Berlin +1
Also Small Models Can Reasonably Self-Evaluate Their Confidence
Sep 30, 2026Idil Kapikiran, Thomas Decker, Thomas RunklerConfidence Estimation in Language Models Small Language Models
Technical University of Munich · Siemens AG · LMU Munich +1
Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
Sep 29, 2026Abhinav Rajeev Kumar, Paras ChopraLLM Sycophancy LLM Reliability
Lossfunk
Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Sep 29, 2026Bruno Brocai, Maria BeckerLLM-as-a-Judge Pairwise Preference Evaluation
Heidelberg University German Department
Complexity-Aware Evaluation of LLM Comprehension
Sep 29, 2026Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. AjilaSoftware Engineering LLM Evaluation
Systems and computer Engineering Carleton University Ottawa, Canada
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Sep 29, 2026Zhuo Chen, Hao Zeng, Jiawei Liu +6LLM Evaluation Adversarial Attacks on LLMs
Wuhan University · East China Normal University
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG
Sep 29, 2026Pranav Handa, Ariful AzadRetrieval-Augmented Generation LLM Reliability
Texas A&M University
Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents
Sep 29, 2026Jianchang Su, Yiwei Yang, Wei ZhangAutomated Fact-Checking LLM Reliability
University of Connecticut · University of California, Santa Cruz

Oct 7, 2026, cs.PFJ/K move · Enter open · S save

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu+1

Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland+1

LLM Inference ML Reproducibility LLM Evaluation LLM Reliability

Open paper AskPDF arXiv

Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.

Momentum

Related topics

Latest papers 507