LLM Evaluation

LLM: Large Language Model

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,539 papers144 papers in the last four weeks, down 56% on the four weeks before. 1.1% of all new papers.

  1. Large Language Model probabilities cannot reliably distinguish between possible and impossible language

    Sep 18, 2025Evelina Leivada, Raquel Montero, Paolo Morosi +5Syntactic Generalization in Language ModelsLLM Evaluation

  2. Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison

    Sep 10, 2025Marianna Nezhurina, Harsh Raj, Victor May +7LLM TrainingLLM Evaluation

  3. LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue

    Sep 2, 2025Katharine Kowalyshyn, Matthias ScheutzLLM EvaluationTheory of Mind

  4. Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

    Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM EvaluationLanguage Model Generation Evaluation

  5. Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

    Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference LearningLLM Evaluation

  6. Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules

    Aug 14, 2025Nasim Shirvani-Mahdavi, Chengkai LiLLM EvaluationExplainability Evaluation

  7. Limits of LLM Text Detectors in Education

    Aug 11, 2025Lukas Gehring, Benjamin Paa\ssenLLM EvaluationAI-Generated Text Detection

  8. Post-training for Efficient Communication via Convention Formation

    Aug 8, 2025Yilun Hua, Evan Wang, Yoav ArtziLLM EvaluationLLM Post-Training

  9. How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

    Aug 2, 2025Ma'ayan Armony, Albert Meroño-Peñuela, Gerard CanalLLM PlanningLLM Evaluation

  10. Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

    Jul 23, 2025Tianlin Li, Yunxiang Wei, Zhiming Li +5LLM EvaluationCode Language Models

  11. Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction

    Jul 22, 2025Tianyun Zhong, Guozhao Mo, Yanjiang Liu +9Document Information ExtractionLLM Evaluation

  12. IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research

    Jul 21, 2025Yuanhao Shen, Daniel Xavier de Sousa, Ricardo Marçal +2LLM EvaluationLarge Language Model-Guided Scientific Discovery

  13. FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

    Jul 14, 2025Tao Feng, Haozhen Zhang, Zijie Lei +5LLM EvaluationLLM Routing

  14. Detoxify: A framework for abusive text transformation using LLMs

    Jul 14, 2025Rohitash Chandra, Jiyong Choi, Jayesh SonawaneLLM EvaluationText Detoxification

  15. A Mathematical Theory of Discursive Networks

    Jul 9, 2025Juan B. GutiérrezLLM ReliabilityMulti-Agent LLM Systems

  16. LLM-Based Social Simulations Require a Boundary

    Jun 24, 2025Zengqing Wu, Run Peng, Takayuki Ito +2LLM EvaluationHuman Behavior Simulation

  17. Measuring Intent Comprehension in LLMs

    Jun 19, 2025Nadav Kunievsky, James A. EvansPrompt SensitivityLLM Evaluation

  18. LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

    Jun 4, 2025Neeva Oza, Ishaan Govil, Parul Gupta +3LLM EvaluationLanguage Model Robustness

  19. C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models

    Jun 2, 2025Ping Wu, Guobin Shen, Dongcheng Zhao +6Value AlignmentLLM Alignment

  20. Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

    May 31, 2025Eunsu Kim, Haneul Yoo, Guijin Son +3LLM Evaluation

  21. SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments

    May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM EvaluationAI Agent Benchmarks

  22. PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

    May 29, 2025Atharva Naik, Prakam, Yash Mathur +6Program SynthesisInductive Inference

  23. Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

    May 27, 2025Baran Barbarestani, Isa Maks, Piek VossenContent ModerationToxicity Detection

  24. But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

    May 23, 2025Leon Eshuijs, Archie Chaudhury, Alan McBeth +1LLM EvaluationLLM-as-a-Judge

  25. R3: Robust Rubric-Agnostic Reward Models

    May 19, 2025David Anugraha, Zilu Tang, Lester James V. Miranda +5Reward ModelingLLM Evaluation

  26. Enriching Patent Claim Generation with European Patent Dataset

    May 18, 2025Lekang Jiang, Chengzu Li, Stephan GoetzText GenerationLLM Evaluation

  27. Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?

    May 11, 2025Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu +1LLM EvaluationQuantitative Finance