LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,443 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
Apr 10, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezEnergy-Aware LLM Inference LLM Inference Efficiency
Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland
CONDESION-BENCH: Conditional Decision-Making of Large Language Models in Compositional Action Space
Apr 10, 2026Yeonjun Hwang, Sungyong Park, Minju Kim +2LLM Decision-Making LLM Evaluation
Department of Artificial Intelligence, Yonsei University
Quantisation Reshapes the Metacognitive Geometry of Language Models
Apr 10, 2026Jon-Paul CacioliLanguage Model Self-Assessment Metacognition in Language Models
Independent Researcher, Melbourne, Australia
TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice
Apr 10, 2026Gang Hu, Yating Chen, Haiyan Ding +5LLM Evaluation
Yunnan University, Yunnan, China. · Jianghan University, Wuhan, China. · Nanjing Audit University, Nanjing, China. +1
Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models
Apr 9, 2026Yousra Fettach, Guillaume Bied, Hannu Toivonen +1LLM Alignment LLM Evaluation
Ghent University, Belgium · University of Helsinki, Finland
LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs
Apr 9, 2026Paolo Gajo, Domenic Rosati, Hassan Sajjad +1Relation Extraction LLM Information Extraction
University of Bologna · Dalhousie University
QoS-QoE Translation with Large Language Model
Apr 9, 2026Yingjie Yu, Mingyuan Wu, Ahmadreza Eslaminia +3LLM Evaluation
University of Illinois Urbana-Champaign, Urbana, Illinois, USA
Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces
Apr 9, 2026Jiawei Chen, Ruoxi Xu, Boxi Cao +12Human Behavior Simulation User Simulation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology
TEMPER: Testing Emotional Perturbation in Quantitative Reasoning
Apr 9, 2026Atahan Dokme, Benjamin Reichman, Larry HeckFraming Effects in Language Models LLM Evaluation
Georgia Institute of Technology
Too long; didn't solve
Apr 8, 2026Lucía M. Cabrera, Isaac Saxton-Knight, Jocelyn D'ArcyMathematical Reasoning Benchmarks LLM Mathematical Reasoning
Instituto Balseiro · Poindexter Labs
CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data
Apr 8, 2026Mohamed Ehab, Ali Hamdi, Khaled ShabanClass-Imbalanced Learning Language Model Ensembles
Faculty of computer science, October University for Modern Science & Arts, Giza,Egypt. · Department of Computer Science and Engineering, Qatar University, Doha, Qatar.
Syntax Is Easy, Semantics Is Hard: Evaluating LLMs for LTL Translation
Apr 8, 2026Priscilla Kyei Danso, Mohammad Saqib Hasan, Niranjan Balasubramanian +1Linear Temporal Logic LLM Evaluation
Stony Brook University, Stony Brook, USA
EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration
Apr 8, 2026Jianfei Wu, Zhichun Wang, Zhensheng Wang +1Geospatial QA Geospatial Reasoning Benchmarks
School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · Beijing Key Laboratory of Artificial Intelligence for Education, Beijing 100875, China · Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education, Beijing 100875, China +1
Unified Deployment-Aware Evaluation of Open Reasoning Language Models
Apr 8, 2026Md Motaleb Hossen Manik, Ge WangLLM Evaluation Language Model Reasoning Evaluation
Department of Computer Science, Rensselaer Polytechnic Institute · Department of Biomedical Engineering, Rensselaer Polytechnic Institute
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
Apr 8, 2026José Pombal, Ricardo Rei, André F. T. MartinsBias in LLM-as-a-Judge LLM Evaluation
Sword Health · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa +2
The AI Skills Shift: Mapping Skill Obsolescence, Emergence, and Transition Pathways in the LLM Era
Apr 8, 2026Rudra Jadhav, Janhavi DanveAI and Labor LLM Evaluation
Department of Computer Science, Savitribai Phule Pune University, Pune, India
What Does a Sharing Question Add? Auditing LLM Survey Scores for Misinformation
Apr 8, 2026Zonghuan Xu, Xiang Zheng, Yutao Wu +1LLM Evaluation LLM Auditing
Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · City University of Hong Kong, Hong Kong SAR, China · Deakin University, Australia
Beyond Accuracy: Diagnosing Algebraic Reasoning Failures in LLMs Across Nine Complexity Dimensions
Apr 8, 2026Parth Patil, Dhruv Kumar, Yash Sinha +1Working Memory in Language Models LLM Mathematical Reasoning
Department of Computer Science & Information Systems BITS Pilani · School of Computer Engineering, KIIT Bhubaneshwar
TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks
Apr 8, 2026Xiangyu Wang, Jin Wu, Haoran Shi +3Multi-LLM Collaboration Multi-Agent LLM Systems
Department of Educational Psychology, East China Normal University · Lab of Artificial Intelligence for Education, East China Normal University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University +1
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
Apr 8, 2026Minzhu Tu, Shiyu Ni, Keping BiLLM-as-a-Judge LLM Evaluation
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · Beijing University of Post and Telecommunications +1
Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach
Apr 8, 2026Xiaoyou Qin, Zhihong Li, Xiaoxiao ChengLarge Language Model-Based Social Simulation User Modeling
School of Journalism, Fudan University · College of Media and International Culture, Zhejiang University
LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
Apr 8, 2026Igor Ivanov, David Demitri AfricaLLM Evaluation LLM Safety Evaluation
Meridian Cambridge
Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks
Apr 7, 2026Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo +5LLM Evaluation Synthetic Benchmark Generation
Institute of Mathematics and Computer Science, University of São Paulo, Brazil · HUG Labs, Brazil · Institute of Informatics, Federal University of Goiás, Brazil +1
Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
Apr 7, 2026Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1LLM Evaluation Long-Context Language Model Evaluation
Information Science Cornell University · School of Communication and Culture Aarhus University
The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models
Apr 7, 2026Xiaojie Gu, Ziying Huang, Weicong Hong +3Knowledge Editing LLM Reliability
Independent Researcher · Cornell Tech · The Ohio State University +1
FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
Apr 7, 2026Michael Krumdick, Varshini Reddy, Shivani Chaudhary +8Computer-Use Agent Benchmarks LLM Evaluation
Kensho Technologies · S&P Global · MIT
Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts
Apr 7, 2026Fatih UenalLLM Evaluation LLM Reliability
Department of Computer Science, University of Colorado Boulder
What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces
Apr 7, 2026Birger MoëllComputational Literary Studies LLM-as-a-Judge
Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden
Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation
Apr 7, 2026Abdullah Mazhar, Het Riteshkumar Shah, Aseem Srivastava +2Clinical Language Model Evaluation LLM Evaluation
IIIT Delhi · MBZUAI · Wysa
PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?
Apr 7, 2026Yusen Hou, Weicai Long, Haitao Hu +3Genomics LLM Evaluation
Hong Kong University of Science and Technology (Guangzhou)

Apr 10, 2026J/K move · Enter open · S save

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

Energy-Aware LLM Inference LLM Inference Efficiency Energy-Efficient ML Heterogeneous Computing LLM Evaluation

Open paper AskPDF arXiv

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.