LLM Evaluation

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

  1. Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains

    Mar 15, 2026Andrew KatzLLM EvaluationSurprisal Theory

    Department of Engineering Education, Virginia Tech

  2. QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

    Mar 14, 2026Yao Wu, Kangping Yin, Liang Dong +13LLM EvaluationClinical Language Model Evaluation

    Quark Medical Team, Alibaba Group

  3. Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities

    Mar 13, 2026Yurui Zhu, Giovanni Colavizza, Matteo RomanelloDocument Information ExtractionLLM Information Extraction

    Odoma LLC

  4. Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

    Mar 13, 2026Pei-Fu Guo, Ya-An Tsai, Chun-Chia Hsu +6LLM Evaluation

    National Taiwan University · University of California, Los Angeles · Academia Sinica, Taiwan +1

  5. Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study

    Mar 13, 2026Zhiye Jin, Yibai Li, K. D. Joshi +4LLM EvaluationPsychometric Validation

    Nancy · Marywood University · The University of Scranton +3

  6. Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

    Mar 13, 2026Kun Wang, Reinhard HeckelLLM EvaluationInference-Time Language Model Alignment

    School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning

  7. CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

    Mar 13, 2026João Silva, Luís Gomes, António BrancoLLM EvaluationPortuguese Language Models

    University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal

  8. LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

    Mar 12, 2026Himel Ghosh, Nick Elias WernerLanguage Model Bias EvaluationLLM Evaluation

    Technical University of Munich, Germany · Sapienza University of Rome, Italy

  9. When LLM Judge Scores Look Good but Best-of-N Decisions Fail

    Mar 12, 2026Eddie LandesbergLLM EvaluationLLM-as-a-Judge

  10. Agreement Between Large Language Models, Human Reviewers, and Authors in Evaluating STROBE Checklists for Observational Studies in Rheumatology

    Mar 12, 2026Emre Bilgin, Ebru Ozturk, Meera Shah +6LLM EvaluationHealthcare

    Sakarya University, Faculty of Medicine, Department of Internal Medicine, Division of Rheumatology, Sakarya, Turkiye · Topiwala NaBonal Medical College and B.Y.L. Nair Hospital, Department of Rheumatology, , Mumbai, India · St Luke’s Medical Center (Global City), Philippines +9

  11. Language Model Teams as Distributed Systems

    Mar 12, 2026Elizabeth Mieczkowski, Katherine M. Collins, Ilia Sucholutsky +2Multi-Agent LLM SystemsMulti-LLM Collaboration

    Princeton University · Massachusetts Institute of Technology · University of Cambridge +1

  12. Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions

    Mar 12, 2026Tae-Eun SongLLM Self-CorrectionLLM Evaluation

    Daejeon Jungang Cheonggua Co., Ltd.

  13. To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times

    Mar 12, 2026Thomas Hikaru Clark, Carlos Arriaga, Javier Conde +2LLM Evaluation

    Massachusetts Institute of Technology · Universidad Politécnica de Madrid

  14. BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

    Mar 12, 2026Ilias AarabZero-Shot Text ClassificationLLM Evaluation

    European Central Bank, Frankfurt am Main, Germany

  15. Large Language Models for Biomedical Article Classification

    Mar 12, 2026Jakub Proboszcz, Paweł CichoszFew-Shot Text ClassificationLarge Language Model-Based Text Classification

    Institute of Computer Science, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland

  16. SemBench: A Universal Semantic Framework for LLM Evaluation

    Mar 12, 2026Mikel Zubillaga, Naiara Perez, Oscar Sainz +1LLM EvaluationMultilingual Language Model Evaluation

    HiTZ Center - Ixa, University of the Basque Country UPV/EHU

  17. Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation

    Mar 11, 2026Minghe Shen, Ananth Balashankar, Adam Fisch +2LLM ReliabilityLLM Evaluation

    University College London · Google DeepMind

  18. Temporal Text Classification with Large Language Models

    Mar 11, 2026Nishat Raihan, Marcos ZampieriText ClassificationLarge Language Model-Based Text Classification

    George Mason University, Fairfax, VA, USA

  19. AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

    Mar 11, 2026Yibai Li, Xiaolin Lin, Zhenghui Sha +2LLM EvaluationPsychometric Validation

    The University of Scranton · California State University, Sacramento · The University of Texas at Austin +1

  20. Ranking Reasoning LLMs under Test-Time Scaling

    Mar 11, 2026Mohsen Hariri, Michael Hinczewski, Jing Ma +1LLM EvaluationTest-Time Scaling

    Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA · Department of Physics, Case Western Reserve University, Cleveland, OH, USA

  21. Making Bielik LLM Reason (Better): A Field Report

    Mar 11, 2026Adam Trybus, Bartosz Bartnicki, Remigiusz KinasLLM ReasoningLanguage Modeling

    Institute of Philosophy, Jagiellonian University, 52 Grodzka St., Kraków 31-044, Poland · Bielik.ai (Speakleash Foundation), 33/4 Niemczańska St., Wrocław 50-561, Poland

  22. PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

    Mar 11, 2026Minki Hong, Eunsoo Lee, Sohyun Park +1LLM PromptingLLM Evaluation

    Department of Computer Science and Artificial Intelligence, Dongguk University, South Korea

  23. Brevity Constraints Reverse Performance Hierarchies in Language Models

    Mar 11, 2026MD Azizul HakimLLM EvaluationLanguage Model Scaling

    Department of Computer Science Bangladesh Sweden Polytechnic Institute Chattogram, Bangladesh

  24. Evaluating Digital Inclusiveness of Digital Agri-Food Tools Using Large Language Models: A Comparative Analysis Between Human and AI-Based Evaluations

    Mar 11, 2026Githma Pewinya, Carolina Martins, Garcia MariangelLLM ReliabilityLLM Evaluation

    Colombo, Sri Lanka · International Water Management Institute

  25. Large language models can disambiguate opioid slang on social media

    Mar 11, 2026Kristy A. Carpenter, Issah A. Samori, Mathew V. Kiang +4Word Sense DisambiguationSocial Media Analysis

    Departments of 1Biomedical Data Science, Stanford University, Stanford, CA 94305, USA · Departments of 2Bioengineering, Stanford University, Stanford, CA 94305, USA · Departments of 3Epidemiology and Population Health, Stanford University, Stanford, CA 94305, USA +5

  26. Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

    Mar 11, 2026Tim Schopf, Michael FärberAutomated EvaluationLLM-as-a-Judge

    TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan

  27. Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

    Mar 10, 2026Keita BroadwaterLLM EvaluationLanguage Model Safety Evaluation

    Independent Researcher, San Jose, USA

  28. CREATE: Testing LLMs for Associative Creativity

    Mar 10, 2026Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman +2Creativity AssessmentComputational Creativity

    New York University · The University of Texas at Austin

  29. One-Eval: An Agentic System for Automated and Traceable LLM Evaluation

    Mar 10, 2026Chengyu Shen, Yanheng Hou, Minghui Pan +8Automated EvaluationLLM Evaluation

    Peking University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +1

  30. EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

    Mar 10, 2026Aman Sharma, Paras ChopraLLM EvaluationLLM Code Generation

    Lossfunk