LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models
Sep 15, 2026Julio C. Amador Diaz LopezLLM Evaluation Confidence Estimation in Language Models
SiftyML, LTD London, United Kingdom
Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Sep 15, 2026Yuan Pu, Yewon Chang, Furong Jia +4LLM Evaluation LLM Auditing
Department of Computer Science, Duke University, 308 Research Drive, Durham, 27708, NC. · Department of Medicine, Duke University, 40 Duke Medicine Circle, Durham, 27710, NC. · Geriatrics and Extended Care, Durham VA Health System, 508 Fulton Street, Durham, 27705, NC. +2
Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening
Sep 15, 2026Qiangju Chen, Yang XiaoLLM Evaluation Language Model Robustness
1Macquarie University · 2The University of Melbourne
Before You Poll with LLMs: A Deliberative Diagnostic Framework
Sep 14, 2026Ahmed Wali, Hassaan TayyabLLM Evaluation Political Bias in Language Models
Lahore University of Management Sciences
Are LLMs Good Financial User Simulators? A Preliminary Study
Sep 14, 2026Jiajie He, Jiangyuan Hong, Dongling Ni +2LLM Evaluation Human Behavior Simulation
1Hithink Research · 2Nanyang Technological University · 3McMaster University
Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation
Sep 14, 2026Sarra Gharsallah, Adele Robaldo, Mariia Tokareva +5LLM Evaluation LLM-as-a-Judge
Data Science Department, EURECOM, Campus SophiaTech, Biot, France
Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku
Sep 14, 2026Livia Oddi, Simone Scardapane, Toru Sugimoto +1LLM Evaluation AI-Generated Text Detection
DIET, Sapienza University of Rome, Rome, Italy · Shibaura Institute of Technology, Tokyo, Japan · DIAG, Sapienza University of Rome, Rome, Italy
Temperature Fragility and the Conditional Benefits of Truncation Sampling
Sep 14, 2026Francesco La RosaLLM Evaluation Language Model Decoding
University of Edinburgh
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation
Sep 14, 2026M. Ali BayramMultilingual Language Model Evaluation LLM Evaluation
Department of Computer Engineering, Yıldız Technical University İstanbul, Turkey
Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain
Sep 14, 2026Motaz Saad, Anna Borrelli, Ivan Gentile +3LLM Evaluation LLM Grounding
University of Salento · University of Naples Federico II · IFAB Foundation
SALUTE: Benchmarking and Adapting LLMs for the Defense Domain
Sep 14, 2026Hyeongcheol Park, Sumin In, Suyeon Myeong +5LLM Evaluation Domain-Adaptive Pretraining
Korea University, Republic of Korea · Sungkyunkwan University, Republic of Korea · Hanwha Systems, Republic of Korea
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge
Sep 14, 2026Shreyas Krishnan, Serina Chang, Abhishek NagarajLLM Evaluation Factual Knowledge in Language Models
University of California, Berkeley · National Bureau of Economic Research (NBER)
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
Sep 14, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3LLM Evaluation
Bosch Research · Sun Yat-sen University
Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage
Sep 14, 2026Foad Namjoo, Remy Ogasawara, Amirali Abdullah +3Data Leakage LLM Evaluation
University of Utah · Thoughtworks · University of Massachusetts Amherst +1
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Sep 14, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48LLM Evaluation Physical Reasoning
Biomedical Reference Generation Remains Unreliable across 26 Large Language Models
Sep 14, 2026Maxim Topaz, Zhihong Zhang, Nir Roguin +3LLM Evaluation Hallucination in Language Models
School of Nursing, Columbia University, New York, NY, USA · Data Science Institute, Columbia University, New York, NY, USA · VNS Health, New York, NY, USA +4
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs
Sep 14, 2026Naihao Deng, Samee Arif, Shuaichen Chang +2LLM Evaluation LLM Alignment
University of Michigan · The Ohio State University · University of Aberdeen +1
A primer on evaluation methods for large language models in healthcare
Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7LLM Evaluation Clinical Language Model Evaluation
Mass General Brigham AI, Mass General Brigham, Boston, MA, USA · Generative AI Laboratory, School of Informatics, University of Edinburgh, Edinburgh, UK · University of Edinburgh Centre for Cardiovascular Science, Edinburgh, UK +5
Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks
Sep 13, 2026Kareem Khattab, Omar Khattab, Mohamed IbrahemLLM Evaluation Text Generation
Entendre Finance
Measuring the Creativity of Frontier LLMs in Automated Research
Sep 12, 2026Yiheng Zhao, Mengzhuo Chen, Chengming Hu +3Creativity Assessment LLM Evaluation
Concordia University, Montreal, Canada · Independent Researcher · Mcgill University, Montreal, Canada +2
IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External Directives
Sep 12, 2026Kainan Zhou, Zhaoyi Li, Janet Sung +2LLM Evaluation Benchmark Design
Google LLC Mountain View, CA, USA
Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures
Sep 12, 2026Victor Mazzotti, Luiz Pereira, Marina Bitencourt dos Santos +4LLM Evaluation
Instituto de Matemática, Estatística e Computação Científica (IMECC) · Instituto de Computação (IC) · Instituto de Estudos da Linguagem (IEL) +1
NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment
Sep 12, 2026Guoqiang Zhang, Kexin Tan, Ming Zhang +12LLM Evaluation LLM Answer Verification
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Graduate School of Engineering, The University of Tokyo, Tokyo, Japan · AtomInnoLab 4Atom Infinite Pte. Ltd., Singapore +1
Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing
Sep 12, 2026Yi LiuLLM Evaluation Framing Effects in Language Models
University of Science and Technology of China
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Sep 11, 2026Yu-Chung HsiaoLLM Evaluation LLM-as-a-Judge
Cisco Systems
The widening evaluation gap in medical large language model research 2023 to 2026
Sep 11, 2026Raad Bin Tareaf, Murad Al-Rajab, Samia LoucifLLM Evaluation Clinical Language Model Evaluation
XU Exponential University of Applied Sciences, Data Science and Artificial Intelligence Cluster, Marlene-Dietrich-Allee 12B, 14482 Potsdam, Germany · College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates · College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates
On the Impact of Anonymization on the Performance of Large Language Models
Sep 10, 2026Tobias Deußer, Max Hahnbück, Lorenz Sparrenberg +3LLM Evaluation Privacy-Preserving Language Models
University of Bonn, Bonn, Germany · 2Fraunhofer IAIS, Sankt Augustin, Germany · 3Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, Germany +1
SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
Sep 10, 2026Qibai Chen, Zeming LiuSoftware Engineering LLM Evaluation
Independent Researcher United States · Brown University United States
Quantifying Logical Consistency in Transformers via Query-Key Alignment
Sep 10, 2026Eduard Tulchinskii, Anastasia Voznyuk, Laida Kushnareva +4Transformer Interpretability LLM Evaluation
Skolkovo Institute of Science and Technology · AI Foundation and Algorithm Lab · Moscow Institute of Physics and Technology +2

Sep 15, 2026, cs.LGJ/K move · Enter open · S save

When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models

Julio C. Amador Diaz Lopez

SiftyML, LTD London, United Kingdom

LLM Evaluation Confidence Estimation in Language Models LLM Uncertainty Estimation

Open paper AskPDF arXiv

Modern predictive systems expose multiple quantities that are commonly interpreted as measures of confidence. However, these quantities can summarize different aspects of the predictive process. This distinction matters when confidence is used to evaluate reliability or inform downstream oversight and control. We investigate whether different confidence readouts are empirically interchangeable in an autoregressive language model by comparing local confidence, defined from the probability of the greedy-selected answer token, with global confidence, defined from modal-answer frequency under repeated sampling. Across MMLU and ARC Challenge, the two signals are weakly correlated and differ substantially in their association with correctness: global confidence is moderately associated with correctness, whereas local confidence shows little association. We further test whether question-level disagreement between the signals is associated with sampling instability. On ARC, larger local--global confidence gaps are associated with higher answer entropy, more distinct sampled answers, and lower modal-answer concentration. The gap--entropy association persists when disagreement and instability are estimated from disjoint stochastic samples, indicating that it is not explained by shared finite-sample variation. The corresponding relationship is substantially weaker on MMLU, where only 4% of questions exhibit sampling instability. These results show that confidence readouts derived from the same predictive system are not empirically interchangeable and that their disagreement can provide a diagnostic of unstable sampling behavior. Confidence should therefore be treated as an explicitly defined measurement rather than as a single intrinsic scalar property of a model, particularly when it is used to inform downstream evaluation, oversight, or control.

Momentum

Related topics

Latest papers 1,631