LLM Evaluation

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

  1. Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

    Mar 24, 2026Siddhant Kulkarni, Yukta KulkarniMulti-Agent LLM SystemsLLM Agent Orchestration

    Department of Computer Science and Engineering, New York University, New York, NY 10012

  2. Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length

    Mar 23, 2026Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-ColladosLLM EvaluationLong-Context Language Model Evaluation

    School of Computer Science and Informatics, Cardiff University

  3. Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning

    Mar 23, 2026Domonkos VargaLLM AuditingLLM Evaluation

  4. Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks

    Mar 23, 2026Yiliang Song, Hongjun An, Jiangan Chen +4LLM EvaluationBenchmark Contamination

    Institute of Artificial Intelligence (TeleAI), China Telecom · Guangxi Normal University · Northwestern Polytechnical University

  5. PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Prompts

    Mar 22, 2026Neeladri Bhuiya, Shib Sankar Dasgupta, Andrew McCallum +1LLM EvaluationRepresentation Learning

    CICS, University of Massachusetts Amherst, USA · A10 Networks · Amazon AWS AI

  6. Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

    Mar 22, 2026Jinghan Cao, Yu Ma, Xinjin Li +2Small Language ModelsEfficient Language Model Inference

    San Francisco State University - Department of Computer Science 1600 Holloway Ave, San Francisco - USA · Carnegie Mellon University - Department of Computer Science 5000 Forbes Ave, Pittsburgh - USA · Columbia University - Department of Computer Science 116th and Broadway, New York - USA +2

  7. COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding

    Mar 22, 2026Xiaozhe Li, Tianyi Lyu, Siyi Yang +6LLM Evaluation

    Tongji University · CurrentsAI Research · Stanford University

  8. Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study

    Mar 21, 2026Jenny Gao, Yongfeng Zhang, Mary L Disis +1LLM ReliabilityBiomedical IR

    College of Arts and Science, New York University, New York, NY · College of Arts and Science, New York University, New York, NY 10003 · Department of Computer Sciences, School of Arts & Sciences, Rutgers University, Piscataway, NJ +9

  9. RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation

    Mar 21, 2026Kaustubh D. Dhole, Eugene AgichteinAutomatic Rubric GenerationLLM Evaluation

    Department of Computer Science, Emory University, Atlanta, GA, USA

  10. Can ChatGPT Really Understand Modern Chinese Poetry?

    Mar 21, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation

    NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau

  11. BenchBench: Benchmarking Automated Benchmark Generation

    Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark DesignSynthetic Benchmark Generation

    Nanyang Technological University Singapore, Singapore

  12. RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

    Mar 21, 2026Kaiyuan Li, Jing-Cheng Pang, Yang YuLanguage Model Reasoning EvaluationRL for Language Model Reasoning

    National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China

  13. Large Language Models for Missing Data Imputation: Understanding Behavior, Hallucination Effects, and Control Mechanisms

    Mar 20, 2026Arthur Dantas Mangussi, Ricardo Cardoso Pereira, Ana Carolina Lorena +1Incomplete Data ImputationLLM Evaluation

    Computer Science Division, Aeronautics Institute of Technology, Praça Marechal Eduardo Gomes, 50, São José dos Campos, 12228-900, São Paulo, Brazil · Science and Technology Institute, Federal University of São Paulo, Avenue Cesare Monsueto Giulio Lattes, 1201, São José dos Campos, 12231-280, São Paulo, Brazil · University of Coimbra, CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Polo II, Pinhal de Marrocos, Coimbra 3030-290, Portugal

  14. Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study

    Mar 20, 2026Mohammed Rakibul HasanLLM ReliabilityLLM Evaluation

    Department of Electrical & Computer Engineering, North South University, Dhaka, Bangladesh

  15. Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable

    Mar 20, 2026Rounak Saha, Gurusha Juneja, Dayita Chaudhuri +3LLM EvaluationAI-Generated Text Detection

    ♠Indian Institute of Science · ♢University of California, Santa Barbara · ♡Carnegie Mellon University

  16. When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal

    Mar 20, 2026Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien +5Audio-Language Model EvaluationLLM Evaluation

    National Taiwan University, Taiwan

  17. Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation

    Mar 20, 2026Richard J. YoungCoT FaithfulnessLLM Evaluation

    University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

  18. Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews

    Mar 20, 2026Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo +13Systematic Review AutomationLLM Evaluation

    University of Oxford · Imperial College London · University of Nottingham +2

  19. EVE: A Domain-Specific LLM Framework for Earth Intelligence

    Mar 20, 2026Àlex R. Atrio, Antonio Lopez, Jino Rohit +6Earth ScienceLLM Evaluation

    Pi School · Mistral AI · Translated +1

  20. SEAR: Schema-Based Evaluation and Routing for LLM Gateways

    Mar 20, 2026Zecheng Zhang, Han Zheng, Yue XuLLM RoutingLLM Evaluation

    Strukto.AI, USA · Infron.AI, USA

  21. TextReasoningBench: Does Reasoning Really Improve Text Classification in Large Language Models?

    Mar 20, 2026Xinyu Guo, Yazhou Zhang, Jing QinLarge Language Model-Based Text ClassificationLLM Evaluation

    School of Computer Science and Technology, Tianjin University, China · School of Nursing, The Hong Kong Polytechnic University, HongKong

  22. ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

    Mar 19, 2026Tianlong Wang, Pinqiao Wang, Weili Shi +1LLM EvaluationLLM Planning

    University of Virginia

  23. Correct Chains, Wrong Answers: Dissociating Reasoning from Output in LLM Logic

    Mar 19, 2026Abinav Rao, Sujan Rachuri, Nikhil VemuriLLM ReasoningLLM Evaluation

  24. FinTradeBench: A Financial Reasoning Benchmark for LLMs

    Mar 19, 2026Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan +2Financial Reasoning in Language ModelsFinancial Reasoning Benchmarks

    University of Central Florida

  25. How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

    Mar 19, 2026Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13Audio UnderstandingAudio-Language Model Evaluation

    National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan

  26. Evaluating Counterfactual Strategic Reasoning in Large Language Models

    Mar 19, 2026Dimitrios Georgousis, Maria Lymperaiou, Angeliki Dimitriou +2Strategic Reasoning Risks in Language ModelsLLM Evaluation

    National Technical University of Athens

  27. Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans

    Mar 19, 2026Christian Di Maio, Tommaso Guidi, Luigi Quarantiello +4LLM EvaluationHuman-AI Interaction

    DIISM, University of Siena, Italy · Department of Computer Science, University of Pisa, Italy · LUISS University, Rome, Italy

  28. ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs

    Mar 19, 2026Wanjia Zhao, Ludwig Schmidt, Yejin Choi +3LLM EvaluationAI Agent Benchmarks

    Stanford University · Microsoft Research

  29. Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions

    Mar 19, 2026Xiaoyi LiLanguage Model Post-TrainingLLM Alignment

  30. GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Norms

    Mar 19, 2026Masayuki Kawarada, Kodai Watanabe, Soichiro MurakamiLLM Decision-MakingLLM Evaluation

    CyberAgent