LLM Evaluation

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

  1. SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era

    Mar 17, 2026Han Jang, Junhyeok Lee, Kyu Sung ChoiLLM EvaluationScientific Communication

    Seoul National University · Seoul National University Hospital · Seoul National University College of Medicine

  2. Assessing the Pedagogical Readiness of Large Language Models as AI Tutors in Low-Resource Contexts: A Case Study of Nepal's K-10 Curriculum

    Mar 17, 2026Pratyush Acharya, Prasansha Bharati, Yokibha Chapagain +2AI in EducationLLM Evaluation

  3. VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?

    Mar 16, 2026Srijan Bansal, Jiao Fangkai, Yilun Zhou +3LLM Self-CorrectionLanguage Model Error Detection

    Salesforce AI Research · Nanyang Technological University · A*STAR

  4. Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN

    Mar 16, 2026Ritajit Dey, Iadh Ounis, Graham McDonald +1Knowledge Conflicts in Language ModelsTemporal Reasoning in Language Models

    University of Glasgow, Glasgow, UK · University of Strathclyde, Glasgow, UK

  5. When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making

    Mar 16, 2026Nazia RiasatLLM EvaluationLLM Reliability

    North Dakota State University

  6. Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks

    Mar 16, 2026Timo FreieslebenLLM Evaluation

    Munich Center for Mathematical Philosophy (MCMP) LMU Munich

  7. Pretraining and Benchmarking Modern Encoders for Latvian

    Mar 16, 2026Arturs ZnotinsLanguage Model PretrainingLow-Resource Language Processing

    Institute of Mathematics and Computer Science, University of Latvia, 29 Raina bulv., Riga, LV-1459, Latvia

  8. Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

    Mar 16, 2026Jaehyeok Lee, Xiaoyuan Yi, Jing Yao +4LLM EvaluationLLM Alignment

    Sungkyunkwan University · Microsoft Research Asia · Singapore University of Technology and Design

  9. LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

    Mar 16, 2026Jon-Paul CacioliLLM EvaluationLanguage Model Calibration

    Independent Researcher, Melbourne, Australia

  10. BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

    Mar 16, 2026Yuzhe TangCommonsense ReasoningLanguage Model Reasoning Evaluation

    Georgia Institute of Technology

  11. CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

    Mar 15, 2026Junhang Cheng, Fang Liu, Jia Li +3Code GenerationLLM Code Generation

    Beihang University, Beijing, China · Wuhan University, Wuhan, China

  12. An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs

    Mar 15, 2026Qian Zhu, Xinnan Guo, Jingjing Huo +5Domain Adaptation for LLMsLLM Alignment

    Ant Group

  13. Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains

    Mar 15, 2026Andrew KatzLLM EvaluationSurprisal Theory

    Department of Engineering Education, Virginia Tech

  14. QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models

    Mar 14, 2026Yao Wu, Kangping Yin, Liang Dong +13LLM EvaluationClinical Language Model Evaluation

    Quark Medical Team, Alibaba Group

  15. Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities

    Mar 13, 2026Yurui Zhu, Giovanni Colavizza, Matteo RomanelloDocument Information ExtractionLLM Information Extraction

    Odoma LLC

  16. Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

    Mar 13, 2026Pei-Fu Guo, Ya-An Tsai, Chun-Chia Hsu +6LLM Evaluation

    National Taiwan University · University of California, Los Angeles · Academia Sinica, Taiwan +1

  17. Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study

    Mar 13, 2026Zhiye Jin, Yibai Li, K. D. Joshi +4LLM EvaluationPsychometric Validation

    Nancy · Marywood University · The University of Scranton +3

  18. Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

    Mar 13, 2026Kun Wang, Reinhard HeckelLLM EvaluationInference-Time Language Model Alignment

    School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning

  19. CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

    Mar 13, 2026João Silva, Luís Gomes, António BrancoLLM EvaluationPortuguese Language Models

    University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal

  20. LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

    Mar 12, 2026Himel Ghosh, Nick Elias WernerLanguage Model Bias EvaluationLLM Evaluation

    Technical University of Munich, Germany · Sapienza University of Rome, Italy

  21. When LLM Judge Scores Look Good but Best-of-N Decisions Fail

    Mar 12, 2026Eddie LandesbergLLM EvaluationLLM-as-a-Judge

  22. Agreement Between Large Language Models, Human Reviewers, and Authors in Evaluating STROBE Checklists for Observational Studies in Rheumatology

    Mar 12, 2026Emre Bilgin, Ebru Ozturk, Meera Shah +6LLM EvaluationHealthcare

    Sakarya University, Faculty of Medicine, Department of Internal Medicine, Division of Rheumatology, Sakarya, Turkiye · Topiwala NaBonal Medical College and B.Y.L. Nair Hospital, Department of Rheumatology, , Mumbai, India · St Luke’s Medical Center (Global City), Philippines +9

  23. Language Model Teams as Distributed Systems

    Mar 12, 2026Elizabeth Mieczkowski, Katherine M. Collins, Ilia Sucholutsky +2Multi-Agent LLM SystemsMulti-LLM Collaboration

    Princeton University · Massachusetts Institute of Technology · University of Cambridge +1

  24. Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions

    Mar 12, 2026Tae-Eun SongLLM Self-CorrectionLLM Evaluation

    Daejeon Jungang Cheonggua Co., Ltd.

  25. To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times

    Mar 12, 2026Thomas Hikaru Clark, Carlos Arriaga, Javier Conde +2LLM Evaluation

    Massachusetts Institute of Technology · Universidad Politécnica de Madrid

  26. BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

    Mar 12, 2026Ilias AarabZero-Shot Text ClassificationLLM Evaluation

    European Central Bank, Frankfurt am Main, Germany

  27. Large Language Models for Biomedical Article Classification

    Mar 12, 2026Jakub Proboszcz, Paweł CichoszFew-Shot Text ClassificationLarge Language Model-Based Text Classification

    Institute of Computer Science, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland

  28. SemBench: A Universal Semantic Framework for LLM Evaluation

    Mar 12, 2026Mikel Zubillaga, Naiara Perez, Oscar Sainz +1LLM EvaluationMultilingual Language Model Evaluation

    HiTZ Center - Ixa, University of the Basque Country UPV/EHU

  29. Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation

    Mar 11, 2026Minghe Shen, Ananth Balashankar, Adam Fisch +2LLM ReliabilityLLM Evaluation

    University College London · Google DeepMind

  30. Temporal Text Classification with Large Language Models

    Mar 11, 2026Nishat Raihan, Marcos ZampieriText ClassificationLarge Language Model-Based Text Classification

    George Mason University, Fairfax, VA, USA