LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
May 1, 2026Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer +2LLM Evaluation Language Model Scaling
Leiden University - The Netherlands · TNO - The Netherlands · TU Delft - The Netherlands
Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
May 1, 2026Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger +4Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich · INSAIT · Sofia University "St. Kliment Ohridski"
Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference
May 1, 2026Yuxuan Gao, Megan Wang, Yi Ling YuLLM Evaluation Energy-Efficient ML
University of Pennsylvania · OpenMesh AI · Columbia University
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3LLM Evaluation Automated Grading
DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology
Investigating LLM's Problem Solving Capability -- a Study on Statics Questions
Apr 30, 2026Tanner Culleton, Hung-Fu ChangLLM Evaluation Physical Reasoning
1R. B, Annis School of Engineering, University of Indianapolis, Indianapolis, IN, 46227, USA
TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering
Apr 30, 2026An-Yang Ji, Jun-Peng Jiang, De-Chuan Zhan +1Table QA LLM Evaluation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
Apr 30, 2026Garvin KruthofLLM Evaluation LLM Alignment
Technical University of Munich
Taming the Centaur(s) with LAPITHS: a framework for a theoretically grounded interpretation of AI performances
Apr 30, 2026Matteo Da Pelo, Alessio Donvito, Claudio Frongia +2LLM Evaluation Computational Cognitive Modeling
University of Cagliari, Italy · University of Bari Aldo Moro, Italy · University of Salerno, CIIT Lab, Italy
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Apr 30, 2026Sihong Wu, Owen Jiang, Yilun Zhao +5LLM Evaluation Scholarly Peer Review
Yale University · New York University · TCS Research
Test Before You Deploy: Governing Updates in the LLM Supply Chain
Apr 30, 2026Mohd Sameen Chishti, Damilare Peter Oyinloye, Jingyue LiLLM Evaluation LLM Auditing
Department of Computer Science NTNU Trondheim, Norway
RuC: HDL-Agnostic Rule Completion Benchmark Generation
Apr 30, 2026Arnau Ayguadé Domingo, Miquel Alberti-Binimelis, Cristian Gutierrez-Gomez +5LLM Evaluation LLM Prompting
Barcelona Supercomputing Center · Universitat Polit`ecnica de Catalunya
Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions
Apr 30, 2026Zhuoran Pan, Yue Li, Zhi Guan +2Financial Services LLM Evaluation
School of Computer Science Peking University Beijing, PA 100871 · Taiyuan University of Technology Taiyuan, Shanxi, PA 030024 · Peking University Beijing, PA 100871
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM Evaluation Language Model Generation Evaluation
SAP · Stanford University
Exploring Applications of Transfer-State Large Language Models: Cognitive Profiling and Socratic AI Tutoring
Apr 30, 2026Minori NoguchiLLM Evaluation Intelligent Tutoring Systems
Hitachi High-Tech Corporation
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
Apr 30, 2026Renwei Meng, Bowen Zhang, Jian Wang +4LLM Evaluation LLM Auditing
Anhui University, Hefei, China
A Reproducibility Study of LLM-Based Query Reformulation
Apr 30, 2026Amin Bigdeli, Radin Hamidi Rad, Hai Son Le +4LLM Evaluation LLM Prompting
University of Waterloo Waterloo, Ontario, Canada · Mila – Quebec AI Institute Montreal, Quebec, Canada · Toronto Metropolitan University Toronto, Ontario, Canada +2
Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation
Apr 30, 2026Jon-Paul CacioliLLM Evaluation
Independent Researcher, Melbourne, Australia
Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR
Apr 30, 2026Sidi Chang, Peiying Zhu, Yuxiao Chen +1LLM Evaluation Rubric-Based Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI Labs, San Francisco, USA
LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps
Apr 30, 2026Keito Inoshita, Xiaokang Zhou, Akira Kawai +1LLM Evaluation Annotator Disagreement
Faculty of Business and Commerce, Kansai University, Osaka, Japan · Faculty of Business Data Science, Kansai University, Osaka, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan +2
REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)
Apr 30, 2026Jun Yeon Won, Xin Jin, Shiqing Ma +1LLM Evaluation Software Reverse Engineering
Ohio State University Columbus, OH, United States · University of Massachusetts Amherst Amherst, MA, United States
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Apr 29, 2026Jon-Paul CacioliLLM Evaluation Language Model Generation Evaluation
Independent Researcher, Melbourne, Australia
When Your LLM Reaches End-of-Life: A Framework for Confident Model Migration in Production Systems
Apr 29, 2026Emma Casey, David Roberts, David Sim +1LLM Evaluation
Verint Systems Inc, Melville, NY, USA
CL-bench Life: Can Language Models Learn from Real-Life Context?
Apr 29, 2026Shihan Dou, Yujiong Shen, Chenhao Huang +35LLM Evaluation LLM Grounding
Hunyuan Team, Tencent · Fudan University
Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics
Apr 29, 2026Jatin Bhusal, Nancy Mahatha, Aayush Acharya +1LLM Evaluation Educational Assessment
Research and Incubation Center (RAIN), Sunway College Kathmandu
Do Larger Models Really Win in Drug Discovery? A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction
Apr 29, 2026Jinjiang Guo, Sheng DingLLM Evaluation Molecular Property Prediction
Global Health Drug Discovery Institute, Beijing 100084, China · School of Pharmaceutical Sciences, Tsinghua University, Beijing 100084, China
Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization
Apr 29, 2026Yash Ganpat SawantLLM Evaluation LLM Personalization
Independent AI Researcher
Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Apr 29, 2026Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika YahataSoftware Engineering LLM Evaluation
AIBL · CESAR School · CMCC +1
A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection
Apr 29, 2026Genan Dai, Zini Chen, Yi Yang +1Multi-Agent LLM Systems LLM Evaluation
School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
Apr 29, 2026Han Gong, Zhen Zhou, Yunyang Shi +4LLM Evaluation Intelligent Transportation Systems
School of Transportation Southeast University Nanjing, Jiangsu 211189 · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, Jiangsu 214122 · Department of Civil and Environmental Engineering Hong Kong Polytechnic University Kowloon, Hong Kong

May 1, 2026J/K move · Enter open · S save

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer, Catholijn M. Jonker+1

Leiden University - The Netherlands · TNO - The Netherlands · TU Delft - The Netherlands

LLM Evaluation Language Model Scaling Information Retrieval RAG Evaluation Human-AI Collaboration

Open paper AskPDF arXiv

Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human-AI workflows has stayed behind. This work evaluates a chatbot-style assistant based on Retrieval-Augmented Generation (RAG) in a realistic multi-turn information-seeking scenario inspired by workplace settings where compliance with local legislation and secure handling of sensitive data are often key. Specifically, we examine the performance of humans (N=112) assisted by RAG-assistants compared to LLM-only or LLM+RAG baselines. In this setting, we investigate how underlying model size (3B, 8B, and 70B) shapes the human-AI collaborative dynamic and how it influences perceived usability and satisfaction. Results show that the performance gain of human-AI collaboration over the model-only baselines is significant, irrespective of model size, suggesting that hybrid systems are beneficial in information-seeking scenarios. Interestingly, however, perceived usability and satisfaction among participants showed little difference across model sizes. This demonstrates a nuanced trade-off between model size, performance, and user perception. Our work highlights the added value of evaluating AI applications in actual multi-turn interactions with human users, looking at usability and satisfaction besides accuracy, rather than focusing on benchmark performance only.