LLM Evaluation

LLM: Large Language Model

Overview

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

CardsList

2,539 papers144 papers in the last four weeks, down 56% on the four weeks before. 1.1% of all new papers.

  1. Energy estimates of open-weight large language models across stated food weights: a dose-response evaluation

    Nov 19, 2026Donato Riccio, Fabrizio Maturo, Iulia BrezeanuLLM EvaluationOpen-Weight Language Models

  2. Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study

    Nov 19, 2026Seungho Song, Donggyu Lee, Jaehan Kim +2LLM Hallucination DetectionLLM Evaluation

  3. A four-level executable validation protocol for evaluating LLM-generated topology specifications of physics-based HVAC digital twins

    Nov 19, 2026Younghwan Joo, Jeonghoon Han, Sanghyun Oh +2Digital TwinsLLM Evaluation

  4. When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Québec French

    Nov 18, 2026Eeham Khan, Richard Khoury, Leila KosseimContinued PretrainingDomain-Adaptive Pretraining

  5. Benchmarking LLM abilities on postdoc-level natural science problems

    Nov 18, 2026Alina Avanesyan, Alexey Boldyrev, Liqun Deng +10Scientific Agent BenchmarksLLM Evaluation

  6. Reliability is a user-specific property of large language models

    Nov 18, 2026Xingyu ZhaoLLM ReliabilityLLM Evaluation

  7. FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

    Nov 17, 2026Yi-Cheng Lai, Hen-Hsen HuangKnowledge Conflicts in Language ModelsLLM Evaluation

  8. Do LLMs Understand Network Traces? A Comparative Study of Reasoning over Raw and Filtered Traffic Data

    Nov 17, 2026Irene Gosálvez-White, Roberto Magán-Carrión, Carlota de la Vega-Soriano +1LLM EvaluationLanguage Model Reasoning Evaluation

  9. SOTA or Luck? The Winner's Curse in LLM Leaderboards

    Nov 17, 2026Sikun Xu, Raphael Thomadsen, Dennis J. Zhang +1LLM EvaluationSelection Bias

  10. Knowing Which, Not How Much: Language Models Order Party Positions They Cannot Place

    Nov 16, 2026Yousef Attia, Jarosław A. ChudziakLLM Evaluation

  11. Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments

    Nov 16, 2026Bonam Mingole, Davis Yadav, Hui Min Lee +2Generative AI in EducationEducational Assessment

  12. Rubric-Based AES with LLM Ensembles: Scoring of Student Responses in Critical Online Reasoning Using Ensembles of LLMs

    Nov 16, 2026Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher +2Automated Essay ScoringEducational Assessment

  13. Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B

    Nov 16, 2026Hao Wang, Ting HuangLLM Agent HarnessesAI Agent Benchmarks

  14. TRIAGE-CPG: A Cross-Model Study of Budgeted Static Malware Triage with Large Language Models

    Nov 16, 2026Paul-Andrei Sava, Alexander Küchler, Daniel Kowatsch +4Malware AnalysisLLMs for Cybersecurity

  15. Benchmarking Large Language Models for Linguistic Annotation

    Nov 16, 2026Ondřej Tichý, Anna Marklová, Barbora Bulantová +2LLM EvaluationLanguage Model Calibration

  16. No-Harm-Jury A Jury-Based Multi-Dimensional Harm Evaluation Framework for Medical AI Responses

    Nov 16, 2026Antonio Puertas Gallardo, Mario CeresaLanguage Model Safety EvaluationAI Safety Evaluation

  17. Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures

    Nov 16, 2026Assaf Hallak, Yftah Ziser, Gal ChechikLarge Language Model-Based RecommendationLLM Evaluation

  18. IndustrialKnowEval: A Benchmark for Evaluating Multi-Level Industrial Knowledge in LLMs

    Nov 16, 2026Haiteng Wang, Haoze Li, Jingwei Guo +3LLM EvaluationKnowledge-Intensive QA

  19. The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)

    Nov 15, 2026Bahaeddin TürkoğluOpen-Weight Language ModelsML Reproducibility

  20. WeightBench: Evaluating Language Models as Weight-Space Programmers

    Nov 13, 2026Aayush Marasini, Barsat Khadka, Nick RahimiWeight-Space ProgrammingLLM Evaluation

  21. Evaluating Large Language Models for Scientific Paper Search: Retrieval Success and Inference Latency on Mac Studio and NVIDIA A100

    Nov 13, 2026Frederic Andriantseheno Rakoto, Yannis Bendi-Ouis, Xavier HinautLLM EvaluationScientific Document Retrieval

  22. BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

    Nov 13, 2026Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu +1LLM EvaluationLanguage Model Pretraining