LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Large Language Model Selection with Limited Annotations
Oct 10, 2025Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models
Oct 8, 2025Donggyu Lee, Hyeok Yun, Meeyoung Cha +3Causal Reasoning in Language Models LLM Evaluation
Graduate School of Data Science, KAIST · College of Business, KAIST · Data Science for Humanity Group, MPI-SP +2
Psychological Imagination Networks Show Cross-Population Centrality and Clustering Alignment in Humans That Large Language Models Fail to Replicate
Oct 5, 2025Saurabh Ranjan, Brian OdegaardLLM Evaluation
University of Florida
When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models
Oct 5, 2025Dustin Wright, Sarah Masud, Jared Moore +5LLM Evaluation Cultural Bias in Language Models
University of Copenhagen · Stanford University · University of Colorado Boulder +1
Uncovering the Computational Ingredients of Human-Like Representations in LLMs
Oct 1, 2025Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1LLM Evaluation LLM Fine-Tuning
University of Wisconsin–Madison · Stanford University
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Reward Modeling LLM Evaluation
KAIST · TH Nürnberg · Microsoft Research Asia
Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management
Sep 30, 2025Runze Zhang, Xiaowei Zhang, Mingyang ZhaoLLM Evaluation Human Behavior Simulation
Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation
Sep 30, 2025Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck +57Mathematical Reasoning Benchmarks LLM Evaluation
Evidence for Limited Metacognition in LLMs
Sep 25, 2025Christopher AckermanLLM Evaluation LLM Uncertainty Estimation
Measuring Combinatorial Creativity in Large Language Models and Humans
Sep 25, 2025Samuel Schapiro, Akshay K. Jagadish, Alexi Gladstone +11Computational Creativity Creativity Assessment
Knowledge Lab, University of Chicago · AI Innovation Institute, Stony Brook University · Princeton AI Lab +6
Predicting LLM Reasoning Performance with Small Proxy Model
Sep 25, 2025Woosung Koh, Juyoung Suk, Sungjun Han +2Language Model Pretraining LLM Evaluation
Trillion Labs · KAIST AI · ▽Trillion Labs, ♠KAIST AI
Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Sep 25, 2025Jungsoo Park, Ethan Mendes, Gabriel Stanovsky +1Language Model Performance Prediction LLM Evaluation
Georgia Institute of Technology, Atlanta, Georgia · The Hebrew University of Jerusalem, Jerusalem, Israel
Polarity Detection of Sustainable Development Goals in News Text
Sep 24, 2025Andrea Cadeddu, Alessandro Chessa, Vincenzo De Leo +5LLM Evaluation Sentiment Analysis
Linkalab s.r.l., Viale Elmas, 142, Cagliari, 09122, Italy · Department of Mathematics and Computer Science, University of Cagliari, via Ospedale2026 72, Cagliari, 09124, Italy · Knowledge Media Institute, The Open University, Walton Hall, Kents Hill, Milton Keynes, MK76AA, United Kingdom
The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
Sep 22, 2025Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou +5LLM Evaluation LLM Auditing
CUHK · JHU · CMU +3
Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
Sep 22, 2025Juyeon Yoon, Somin Kim, Robert Feldt +1LLM Evaluation LLM Reliability
Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Chalmers University of Technology, Gothenburg, Sweden
Large Language Model probabilities cannot reliably distinguish between possible and impossible language
Sep 18, 2025Evelina Leivada, Raquel Montero, Paolo Morosi +5Syntactic Generalization in Language Models LLM Evaluation
Departament de Filologia Catalana, Centre de Lingüística Teòrica, Universitat Autònoma de Barcelona, Barcelona 08193, Spain · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, 08010, Spain · Similarweb, Spain +1
Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Sep 10, 2025Marianna Nezhurina, Harsh Raj, Victor May +7LLM Training LLM Evaluation
LAION · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ) · Open-Ψ (Open-Sci) Collective +5
LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue
Sep 2, 2025Katharine Kowalyshyn, Matthias ScheutzLLM Evaluation Theory of Mind
Department of Computer Science Tufts University
Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM Evaluation Language Model Generation Evaluation
University of Illinois Chicago · Adobe Research
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference Learning LLM Evaluation
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules
Aug 14, 2025Nasim Shirvani-Mahdavi, Chengkai LiLLM Evaluation Explainability Evaluation
University of Texas at Arlington, Arlington, Texas 76019, USA
Limits of LLM Text Detectors in Education
Aug 11, 2025Lukas Gehring, Benjamin Paa\ssenLLM Evaluation AI-Generated Text Detection
Faculty of Technology, Bielefeld University
Post-training for Efficient Communication via Convention Formation
Aug 8, 2025Yilun Hua, Evan Wang, Yoav ArtziLLM Evaluation LLM Post-Training
Department of Computer Science and Cornell Tech Cornell University
How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective
Aug 2, 2025Ma'ayan Armony, Albert Meroño-Peñuela, Gerard CanalLLM Planning LLM Evaluation
Department of Informatics, King’s College London
Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models
Jul 23, 2025Tianlin Li, Yunxiang Wei, Zhiming Li +5LLM Evaluation Code Language Models
Nanyang Technological University Singapore · Zhejiang University China · Peng Cheng Laboratory China +2
IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research
Jul 21, 2025Yuanhao Shen, Daniel Xavier de Sousa, Ricardo Marçal +2LLM Evaluation Large Language Model-Guided Scientific Discovery
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute, Queen’s University, Canada · Instituto Federal de Goiás, Anápolis, Brazil · National Research Council Canada
FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
Jul 14, 2025Tao Feng, Haozhen Zhang, Zijie Lei +5LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · Nanyang Technological University · Meta Monetization AI +2

Oct 10, 2025J/K move · Enter open · S save

Large Language Model Selection with Limited Annotations

Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler+1

TU Delft · ETH Zurich

Model Selection LLM Evaluation Active Learning Information Gain

Open paper AskPDF arXiv

Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.