LLM Evaluation

LLM: Large Language Model

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,539 papers144 papers in the last four weeks, down 56% on the four weeks before. 1.1% of all new papers.

  1. Idea2Plan: Exploring AI-Powered Research Planning

    Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM EvaluationLLM-as-a-Judge

  2. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

    Oct 27, 2025Liwei Jiang, Yuanjun Chai, Margaret Li +7Open-Ended GenerationLLM Evaluation

  3. Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

    Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM EvaluationPsychometric Validation

  4. User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios

    Oct 23, 2025Xiaoyuan Wu, Roshni Kaushik, Wenkai Li +2LLM EvaluationLLM-as-a-Judge

  5. CS-54k: A Paper-Grounded Corpus and Benchmark for AI Research QA

    Oct 23, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +12Document QAKnowledge-Intensive QA

  6. CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

    Oct 23, 2025Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9Creativity AssessmentLLM Evaluation

  7. KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

    Oct 21, 2025Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6Multilingual Language Model EvaluationLLM Evaluation

  8. BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

    Oct 18, 2025Tian Xia, Tianrun Gao, Wenhao Deng +4LLM EvaluationLLM Grounding

  9. LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

    Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM EvaluationCompetitive Programming

  10. StatEval: A Comprehensive Benchmark for Large Language Models in Statistics

    Oct 10, 2025Yuchen Lu, Run Yang, Yichen Zhang +6LLM EvaluationReasoning Evaluation

  11. Large Language Model Selection with Limited Annotations

    Oct 10, 2025Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model SelectionLLM Evaluation

  12. EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

    Oct 8, 2025Donggyu Lee, Hyeok Yun, Meeyoung Cha +3Causal Reasoning in Language ModelsLLM Evaluation

  13. What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models

    Oct 5, 2025Dustin Wright, Sarah Masud, Jared Moore +5LLM EvaluationCultural Bias in Language Models

  14. Uncovering the Computational Ingredients of Human-Like Representations in LLMs

    Oct 1, 2025Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1LLM EvaluationLLM Fine-Tuning

  15. Rethinking Reward Models for Multi-Domain Test-Time Scaling

    Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Reward ModelingLLM Evaluation

  16. Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management

    Sep 30, 2025Runze Zhang, Xiaowei Zhang, Mingyang ZhaoLLM EvaluationHuman Behavior Simulation

  17. Feedback Forensics: A Toolkit to Measure AI Personality

    Sep 30, 2025Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier +1LLM EvaluationPersonality Modeling in Language Models

  18. Measuring Combinatorial Creativity in Large Language Models and Humans

    Sep 25, 2025Samuel Schapiro, Akshay K. Jagadish, Alexi Gladstone +11Computational CreativityCreativity Assessment

  19. Predicting LLM Reasoning Performance with Small Proxy Model

    Sep 25, 2025Woosung Koh, Juyoung Suk, Sungjun Han +2Language Model PretrainingLLM Evaluation

  20. Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions

    Sep 25, 2025Jungsoo Park, Ethan Mendes, Gabriel Stanovsky +1Language Model Performance PredictionLLM Evaluation

  21. Polarity Detection of Sustainable Development Goals in News Text

    Sep 24, 2025Andrea Cadeddu, Alessandro Chessa, Vincenzo De Leo +5LLM EvaluationSentiment Analysis

  22. The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

    Sep 22, 2025Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou +5LLM EvaluationLLM Auditing

  23. Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

    Sep 22, 2025Juyeon Yoon, Somin Kim, Robert Feldt +1LLM EvaluationLLM Reliability