LLM Evaluation

LLM: Large Language Model

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,539 papers144 papers in the last four weeks, down 56% on the four weeks before. 1.1% of all new papers.

  1. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

    Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer InferenceLLM Evaluation

  2. Evaluating LLMs on Chinese Topic Constructions: A Research Proposal Inspired by Tian et al. (2024)

    Apr 21, 2025Xiaodong YangLLM EvaluationLanguage Model Probing

  3. DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

    Apr 18, 2025Enhao Huang, Pengyu Sun, Shuxun Wang +13LLM EvaluationSoftware Vulnerability Detection

  4. SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

    Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM EvaluationLanguage Model Generation Evaluation

  5. Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

    Mar 26, 2025Kaiyue Feng, Yilun Zhao, Yixin Liu +4LLM EvaluationPhysical Reasoning

  6. No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

    Mar 7, 2025Michael Krumdick, Charles Lovering, Varshini Reddy +2LLM EvaluationLLM-as-a-Judge

  7. MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents

    Mar 3, 2025Kunlun Zhu, Hongyi Du, Zhaochen Hong +8Multi-Agent CollaborationAI Agent Benchmarks

  8. AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence

    Feb 28, 2025Rilton Franzone, Magnus Sesodia, Alina Petrova +7Legal NLPLLM-Assisted Annotation

  9. Large Language Models for Cryptocurrency Transaction Analysis: A Bitcoin Case Study

    Jan 30, 2025Yuchen Lei, Yuexin Xiang, Tsz Hon Yuen +3LLM EvaluationLLMs for Cybersecurity

  10. Humanity's Last Exam

    Jan 24, 2025Long Phan, Alice Gatti, Ziwen Han +1178LLM EvaluationLanguage Model Reasoning Evaluation

  11. Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions

    Nov 15, 2024Yutao Hou, Yajing Luo, Zhiwen Ruan +4Multi-Hop QALLM Evaluation

  12. Growing a Tail: Increasing Output Diversity in Large Language Models

    Nov 5, 2024Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1LLM EvaluationLLM Prompting

  13. Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

    Sep 1, 2024Jasper Dekoninck, Maximilian Baader, Martin VechevLLM Evaluation

  14. MoralBench: Moral Evaluation of LLMs

    Jun 6, 2024Jianchao Ji, Yutong Chen, Mingyu Jin +3Moral Reasoning in Language ModelsLLM Safety Benchmarks

  15. CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

    Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM EvaluationCausal Discovery

  16. LLMRec: Benchmarking Large Language Models on Recommendation Task

    Aug 23, 2023Junling Liu, Chao Liu, Peilin Zhou +8Large Language Model-Based RecommendationRecommender Systems