LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models
Sep 16, 2026Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin +2LLM Evaluation Cross-Cultural Language Model Evaluation
Artificial Intelligence Laboratory, SKOLKOVO School of Management, Moscow · Artificial Intelligence Laboratory, University of Tyumen · Institute of Business Studies, Russian Presidential Academy of National Economy and Public Administration, Moscow
An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks
Sep 16, 2026Chandimal Adikari, Nandika HerathCode Generation LLM Reliability
Faculty of Engineering and Information Sciences University of Wollongong Wollongong, Australia · Faculty of Engineering, Computing and Science Western Sydney University Parramatta, Australia
The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention
Sep 15, 2026Srijith RavikumarLanguage Model Safety Evaluation Language Model Calibration
Amazon.com LLC
Easy to Catch a Liar, Hard to Clear an Honest One: Language Models Diagnosing a Corrupted Reward Channel from a Verified Record
Sep 15, 2026Arman Nik KhahLLM Auditing Deception in Language Models
The University of Texas at Dallas
An Empirical Study of Counterfactual Self-Explanations in LLMs
Sep 15, 2026Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis +2Counterfactual Explanations Faithfulness of Language Model Explanations
National Technical University of Athens, Athens, Greece · Instituto de Telecomunicações, Lisbon, Portugal
When the Wrong Key Wins: Understanding and Detecting Hallucinations in LLMs
Sep 14, 2026Xuhan Tong, Haoyue Bai, Dawei Zhou +2Language Model Error Detection Hallucination in Language Models
University of Wisconsin–Madison
PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews
Sep 14, 2026Miguel Zabaleta, Baihan LinLLM Auditing AI-Assisted Scientific Research
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Competence-Gated Pooling of Language Models and Priors for Event Forecasting
Sep 14, 2026Aditi Tiwari, Aashrith Bandaru, Heng JiEvent Time Prediction Selective Prediction
University of Illinois Urbana-Champaign Urbana, Illinois, USA
Language Is an Insufficient Substrate for Quantitative Reasoning, and Consequential Domains Need Large Quantitative Models
Sep 14, 2026Reuben Vandeventer, David Imrem, David J. WildNumerical Reasoning in Language Models LLM Reliability
Duo Dimensio LLC · Luddy School of Informatics, Computing and Engineering, Indiana University
Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?
Sep 14, 2026Obinna I. EkekezieCoreference Resolution LLM Answer Verification
Cambridge Health Alliance, Cambridge, MA, United States of America. · Harvard Medical School, Boston, MA, United States of America.
SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling
Sep 14, 2026Junquan Gu, Shibo Cui, Xiangfeng Luo +1LLM Self-Correction Ensemble Learning
Biomedical Reference Generation Remains Unreliable across 26 Large Language Models
Sep 14, 2026Maxim Topaz, Zhihong Zhang, Nir Roguin +3LLM Evaluation Hallucination in Language Models
School of Nursing, Columbia University, New York, NY, USA · Data Science Institute, Columbia University, New York, NY, USA · VNS Health, New York, NY, USA +4
Towards a Deterministic Math Solver for Clinical Language Models
Sep 12, 2026Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange +5Numerical Reasoning in Language Models LLM Reliability
MIT Critical Data · UNC Chapel-Hil · University of Pavia +1
Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Sep 12, 2026Bincheng Gu, Min Gao, Zongwei Wang +3Test-Time Adaptation LLM Reliability
Key Laboratory of Dependable Service Computing in Cyber Physical Society, Chongqing University, China · King’s College London, London, UK · Griffith University, Brisbane, Australia
Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization
Sep 11, 2026Dong-Jae Koh, Huisu Kim, SeongHwan Yoon +4Multi-Agent LLM Systems LLM Reliability
School of Computer Science and Engineering, Kyungpook National University, South Korea
Legible Failures: Detecting and Repairing In-Context Binding Errors
Sep 11, 2026Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. HariLanguage Model Error Detection LLM Reliability
Efficient Computation Inc.
INDRA: A New AI Tool for Exploring Tobacco, Fossil Fuel, and Chemical Industry Archives
Sep 11, 2026Daniel Akselrad, Robert N. ProctorRetrieval-Augmented Generation LLM Reliability
Department of Epidemiology & Population Health, Stanford University, Stanford, CA, USA · Department of History, Stanford University, Stanford, CA, USA
If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs
Sep 9, 2026Xietao Wang-Lin, Anton Isopoussu, Louis MahonLLM Self-Correction Automated Program Repair
University of Warwick · UnlikelyAI
ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
Sep 9, 2026Jianzong Wang, Chuhang Liu, Botao Zhao +6Confidence Estimation in Language Models Language Model Calibration
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · Tsinghua Shenzhen International Graduate School, Shenzhen, China · Hubei University of Technology, Hubei, China
Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal
Sep 9, 2026Yunxiang Mo, Donghao Zhao, Hejia GengEarly Stopping LLM Reliability
HKUST · University of Oxford
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
Sep 9, 2026Eshwar Reddy M, Sourav KarmakarReward Modeling Reward Hacking
AI Engineer, Testsigma University of San Diego · Senior AI Scientist, Intuit India
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Sep 8, 2026Manikandan Ravikiran, Siddharth VohraMultiple-Choice Question Answering LLM Reliability
Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Sep 8, 2026Jennifer Wang, Joachim Baumann, Daniel E. Ho +1LLM Evaluation Language Model Generation Evaluation
Stanford University
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Sep 8, 2026Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah +1LLM Evaluation LLM Auditing
Qatar Computing Research Institute, HBKU
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Sep 7, 2026Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo +10LLM Evaluation LLM Reliability
Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters +7
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
Sep 7, 2026Abrar Shahriar Qurat-Ul-Ain MastoiTool-Augmented Language Model Agents On-Device Language Model Inference
University of the West of England, Bristol,BS16 1QY United Kingdom
A Removal Based Approach to Improve LLM Faithfulness at Test-Time
Sep 7, 2026Qinglan Luo, S M A Nahian, John Guttag +2Faithfulness of Language Model Explanations LLM Auditing
MIT CSAIL · MGH, HMS
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Sep 3, 2026Haoyaun Zhu, Jie ZhangLLM Evaluation LLM-as-a-Judge
School of Electronic and Electrical Engineering, University of Sheffield, Sheffield S10 2TN, UK · R&D Department, Ranplan Wireless Network Design Ltd., Cambridge CB23 3UY, UK · R&D Department, Cambridge AI+ Ltd., Cambridge CB23 3UY, UK

Sep 16, 2026, cs.HCJ/K move · Enter open · S save

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin, Alexander Antipov+1

Artificial Intelligence Laboratory, SKOLKOVO School of Management, Moscow · Artificial Intelligence Laboratory, University of Tyumen · Institute of Business Studies, Russian Presidential Academy of National Economy and Public Administration, Moscow

LLM Evaluation Cross-Cultural Language Model Evaluation LLM Reliability Cultural Bias in Language Models

Open paper AskPDF arXiv

We present a robust benchmark for measuring cultural value orientation in large language models on the two Inglehart-Welzel axes over several domains and roles (hence GYROval - Gridded Yielding of Robust value Orientation), together with the results of administering it to twenty models. Items are binary contrastive scenarios in the sense introduced by CDEval: both options are legitimate courses of action, neither is correct, there is no answer key, and a model's score on an axis is the proportion of its responses falling on the counted pole. Eleven of the twenty models were additionally administered a paired Russian translation of the identical items and a second sampling temperature. The instrument is publicly released in both languages. Stability was assessed by treating the vignette as the unit of analysis, ranking the models within the levels of each perturbation factor, and summarising the agreement between levels by tie-corrected Kendall's \emph{W} against an empirical permutation null.

Momentum

Related topics

Latest papers 507