LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty
Oct 30, 2026Matthias Busch, Marius Tacke, Tamadur Albaraghtheh +5Molecular Property Prediction LLM Evaluation
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +4
Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment
Oct 30, 2026Fan Huang, Hongyu ZhouAutomated Peer Review LLM-as-a-Judge
Indiana University Bloomington, Bloomington, Indiana, USA · School of Information Resource Management, Renmin University of China, Beijing, China · Faculty of Social Sciences, University of Antwerp, Antwerp, Belgium
SERIAL-7: Deriving, Realizing, and Preserving Machine-Verifiable Outputs Across Representations
Oct 30, 2026Youngkyung Seo, MiHyeon Kim, Jisoo Lee +1Structured Output Generation LLM Evaluation
KT Corporation
FrontierMat: Benchmarking Language Models at the Frontier of Materials R&D
Oct 30, 2026Feng Chen, Jianguo Huang, Yushan Xiao +20LLM Evaluation Scientific Reasoning in Language Models
Nanyang Technological University, Singapore · Peking University · Shanghai Jiao Tong University +11
Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali
Oct 29, 2026Sabah Nushra, Md Kishor MorolHallucination in Language Models LLM Evaluation
Islamic University of Technology Dhaka, Bangladesh · Facebook Menlo Park, CA, USA
Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs
Oct 29, 2026Alan Sorani, Ashvni Narayanan, Michael Shalyt +8Mathematical Reasoning Benchmarks LLM Evaluation
Ramanujan Machine Group, Technion, Haifa, Israel. · Department of Mathematics, Computer Science and Statistics, Ghent University, Ghent, Belgium. · Research Institute for Symbolic Computation, Johannes Kepler Universität Linz, Linz, Austria. +1
Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
Oct 29, 2026Peng Xie, Amr AlanwarLLM-as-a-Judge LLM Evaluation
TUM School of Computation, Information and Technology, Department of Computer Engineering Technical University of Munich, 74076 Heilbronn, Germany
Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity
Oct 29, 2026Robert PasternakLLM Evaluation
Leibniz-Centre General Linguistics (ZAS)
What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty
Oct 29, 2026Xiaomeng Xiong, Marcus Walker, Yanlin JiangItem Difficulty Estimation Medical Education
University of Florida, Gainesville, Florida, USA. · National Commission on Certification of Physician Assistants, USA.
SocialArena: Evaluating Social Intelligence in Large Language Models
Oct 29, 2026Zongqi Wang, Rui Wang, Yuchuan Wu +7LLM Evaluation
Alibaba Token Hub, Alibaba Group · Tsinghua University · Alibaba Group
When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks
Oct 29, 2026Carole Lunny, Ian Goodman, Stefano Berrone +12LLM Reliability LLM Evaluation
Cochrane Hypertension Review Group, University of British Columbia.
Blind Spots: Where Small Language Models Fall Behind Frontier Models on Terminal Tasks
Oct 28, 2026Ashima Suvarna, Ruta Desai, Ahmed ElgoharySmall Language Models LLM Evaluation
University of California, Los Angeles · Microsoft Research
TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics
Oct 28, 2026Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad +1LLM Evaluation Language Model Reasoning Evaluation
Department of Mechanical Engineering Boston University Boston, MA 02215 · Department of Mechanical Engineering Division of Systems Engineering Boston University Boston, MA 02215
Cross-Primitive Consistency in Jev
Oct 28, 2026Xu Liu, Guikun Chen, Kanzhi Wu +1LLM Evaluation Consistency Regularization
Zhejiang University · vivo
Benchmarking decision models against generative large language models on medical multiple-choice questions
Oct 28, 2026Oscar Freyer, Daniel Glauert, Moritz Hofmann +3LLM Evaluation Clinical Language Model Evaluation
Else Kröner Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany · Independent researcher, Berlin, Germany · Denkmal.ai, Berlin, Germany +2
Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains
Oct 28, 2026Yang Liu, Ran Zou, Zhengtao Gui +1Human-in-the-Loop Evaluation LLM Evaluation
Department of Statistics and Operations Research, University of North Carolina at Chapel Hill, USA · Department of Statistics, University of California, Irvine, USA · School of Computer Science and Engineering, University of New South Wales, Australia
Keepace.ai: A Technical Report on a Large Language Model for Sports and Health
Oct 28, 2026Keepace. ai Team, :, Bowen Wu +13Medical LLMs LLM Safety Evaluation
Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training
Oct 28, 2026Ulyana Tkachenko, Hui Wen Goh, Guram Gogia +1Automatic Rubric Generation Rubric-Based Evaluation
Handshake AI
Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing
Oct 27, 2026Ilija Subasic, Brett Levert, Andrew Rabinovich +2LLM-as-a-Judge LLM Evaluation
Upwork Inc
Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared
Oct 27, 2026Yifeng Li, Ziang NiLLM Evaluation LLM Reliability
University of Trento · Delft University of Technology
Seq@K: Evaluating Language Models Under Sequential Feedback
Oct 27, 2026Thomson Yen, Gilbert Yang, Naimeng Ye +8LLM Evaluation
Decision, Risk, and Operations Division, Columbia Business School · University of California, Santa Barbara · Northwestern University
OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery
Oct 27, 2026Jiarui Yao, Ruida Wang, Tong ZhangLLM Evaluation Large Language Model-Guided Scientific Discovery
University of Illinois Urbana-Champaign
Do bots feel bad? Assessing empirical evidence for functional valence in LLMs
Oct 27, 2026Iwan WilliamsEmotion Representation in Language Models LLM Evaluation
University of Copenhagen, Center for Philosophy of Artificial Intelligence, Section for Philosophy, Department of Communication
A Quantitative Approach to Asch Conformity in Frontier Large Language Models
Oct 27, 2026Senthil Rajasekaran, Stylianos Loukas VasileiouNormative Conformity in Language Models LLM Evaluation
Université libre de Bruxelles · New Mexico State University
MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge
Oct 26, 2026Jannatul Shefa, Alejandro Salado, Paul Wach +1Requirements Engineering Knowledge Augmentation for Language Models
Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, Virginia, USA · Department of Systems and Industrial Engineering, University of Arizona, Tucson, Arizona, USA
Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss
Oct 26, 2026Oz AmramLanguage Model Auditing LLM Evaluation
Independent
Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates
Oct 26, 2026Zehao Zhao, Shangjian Yin, Rui Liu +6LLM Evaluation Inference-Time Adaptation of Language Models
Meta AI
Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load
Oct 26, 2026Hennadii KutomanovLLM Evaluation Instruction Following
Independent researcher
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM Evaluation LLM Auditing
Texas A&M Transportation Institute, Texas A&M University System · Zachry Department of Civil and Environmental Engineering, Texas A&M University · Department of Civil, Construction and Environmental Engineering, Marquette University +2
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
Oct 26, 2026Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu +7Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
African Institute for Mathematical Sciences (AIMS), Cameroon · University of Nigeria, Nsukka · Brown University +5

Oct 30, 2026J/K move · Enter open · S save

Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty

Matthias Busch, Marius Tacke, Tamadur Albaraghtheh, Sviatlana V. Lamaka+4

Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany+4

Molecular Property Prediction LLM Evaluation LLM Uncertainty Estimation In-Context Learning

Open paper AskPDF arXiv

Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.