LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,443 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

Do Emotions in Prompts Matter? Effects of Emotional Framing on Large Language Models
Apr 2, 2026Minda Zhao, Yutong Yang, Chufei Peng +5LLM Prompting Framing Effects in Language Models
Harvard University. · Bryn Mawr College.
GaelEval: Benchmarking LLM Performance for Scottish Gaelic
Apr 2, 2026Peter Devine, William Lamb, Beatrice Alex +5Multilingual Language Model Evaluation LLM Evaluation
University of Edinburgh · Lancaster University · University of Cardiff +2
LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
Apr 2, 2026Linyang He, Qiyao Yu, Hanze Dong +5Mathematical Reasoning Benchmarks LLM Evaluation
Columbia University · Microsoft Research · University of Amsterdam
What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
Apr 2, 2026Delip Rao, Chris Callison-BurchClaim Verification LLM Evaluation
University of Pennsylvania, Philadelphia, PA, USA
Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study
Apr 2, 2026Gabby Sanchez, Sneha Oommen, Cassandra T. Britto +3LLM Evaluation Classification
RMIT University, Melbourne, Australia
Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements
Apr 2, 2026Yifei Wang, Tianlin Li, Xiaohan Zhang +4LLM Reliability LLM Evaluation
Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore +1
Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation
Apr 1, 2026Hexuan Wang, Jingyu Zhang, Benjamin Van Durme +1LLM Evaluation LLM Grounding
Department of Computer Science, Johns Hopkins University
Cost-Efficient Estimation of General Abilities Across Benchmarks
Apr 1, 2026Michael Krumdick, Adam Wiemerslage, Seth Ebner +2LLM Evaluation Item Response Theory
Kensho Technologies · MIT
Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines
Apr 1, 2026Jingjie Ning, Xueqi Li, Chengyu YuMulti-LLM Collaboration LLM Evaluation
School of Computer Science, Carnegie Mellon University
Fast and Accurate Probing of In-Training LLMs' Downstream Performances
Apr 1, 2026Zhichen Liu, Tianle Lun, Zhibin Wen +7LLM Evaluation Language Model Probing
Southern University of Science and Technology · Huawei
Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models
Apr 1, 2026Hongjian Zou, Yidan Wang, Qi Ding +2LLM Training LLM Evaluation
Vivo AI Lab, Shenzhen, China · Hong Kong University of Science and Technology, Hong Kong, China
Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects
Apr 1, 2026Dumitru Verşebeniuc, Martijn Elands, Sara Falahatkar +4Educational QA Question Answering
Department of Advanced Computing Sciences, Maastricht University, Maastricht 6200 MD, The Netherlands
On Challenges in Evaluation Methodology: A Review
Apr 1, 2026Ruchira Dhar, Qiwei Peng, Anders SøgaardLLM Evaluation
Department of Computer Science, University of Copenhagen
More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation Automated Data Annotation
Home Team Science and Technology Agency
Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMs
Apr 1, 2026Xingyang Yu, Yinghuan Zhang, Yufei Zhang +1Scientific Reasoning in Language Models LLM Evaluation
Department of physics, Virginia Tech · Independent Researcher · Department of Computer Science and Engineering, Michigan State University
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
Mar 31, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation LLM Evaluation
Carnegie Mellon University Africa Kigali, Rwanda
Measuring (some aspects of) the metacognition of AI
Mar 31, 2026Richard Servajean, Philippe ServajeanLLM Evaluation LLM Uncertainty Estimation
Center for Brain Science, RIKEN, Saitama, Japan. · National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan. · Department of Psychology, Paul-Val´ery University, Montpellier, France.
Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
Mar 31, 2026Zoë Prins, Samuele Punzo, Frank Wildenburg +2LLM Interpretability LLM Evaluation
Graduate School of Informatics, University of Amsterdam · College of Informatics, University of Amsterdam · Department of Medical Informatics, Amsterdam University Medical Center +1
GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification
Mar 31, 2026Iordanis Fostiropoulos, Muhammad Rafay Azhar, Abdalaziz Sawwan +8LLM Evaluation User Modeling
Meta Recommendation Systems (MRS)
The Model Says Walk: Measuring whether LLMs Condition on Hidden Constraints
Mar 30, 2026Yubo Li, Lu Zhang, Tianchong Jiang +2Constraint-Guided Reasoning in Language Models Implicit Reasoning in Language Models
Carnegie Mellon University · Independent Researcher
ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts
Mar 30, 2026Rongtian YeChart Understanding Visual Reasoning
Department of Computer Science, Aalto University
MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
Mar 30, 2026Han Wang, Yifan Sun, Brian Ko +8CoT Faithfulness LLM Evaluation
University of Illinois Urbana-Champaign · University of Washington · University of California San Diego
Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
Mar 30, 2026Urja Khurana, Michiel van der Meer, Enrico Liscio +2LLM Evaluation
Department of Intelligent Systems, Delft University of Technology · Leiden Institute of Advanced Computer Science (LIACS), Leiden University · Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam
The Necessity of Setting Temperature in LLM-as-a-Judge
Mar 30, 2026Lujun Li, Lama Sleem, Yangjie Xu +4LLM-as-a-Judge LLM Evaluation
University of Luxembourg · ETH Zürich
Evaluating LLMs for Answering Student Questions in Introductory Programming Courses
Mar 30, 2026Thomas Van Mullem, Bart Mesuere, Peter DawyndtAI in Education Generative AI in Education
Large Language Models for Analyzing Enterprise Architecture Debt in Unstructured Documentation
Mar 29, 2026Christin Pagels, Simon Hacks, Rob Henk BemthuisSoftware Engineering LLM Evaluation
Stockholm University, Stockholm, Sweden · University of Twente, Enschede, The Netherlands
Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?
Mar 29, 2026Yuxuan Gu, Lunjun Liu, Xiaocheng Feng +4LLM Evaluation
Harbin Institute of Technology.
LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis
Mar 29, 2026Yen-Hsun Huang, Yu-Shiou LinSystematic Review Automation AI-Assisted Scientific Research
Department of Education, Taipei Veterans General Hospital, Taipei, Taiwan · Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan
LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications
Mar 28, 2026Alexandre Cristovão MaioranoLLM Evaluation RAG Evaluation
Lumytics
WMIP: A Working-Memory-Inspired Programmable Probe of Cumulative State Tracking in Large Language Models
Mar 28, 2026Dengzhe Hou, Lingyu Jiang, Deng Li +3Working Memory in Language Models LLM Evaluation
Tohoku University · Lappeenranta-Lahti University of Technology LUT · Texas A&M University +1

Apr 2, 2026J/K move · Enter open · S save

Do Emotions in Prompts Matter? Effects of Emotional Framing on Large Language Models

Minda Zhao, Yutong Yang, Chufei Peng, Rachel Gonsalves+4

Harvard University. · Bryn Mawr College.

LLM Prompting Framing Effects in Language Models LLM Evaluation

Open paper AskPDF arXiv

Emotional tone is pervasive in human communication, yet its influence on large language model (LLM) behaviour remains unclear. Here, we examine how first-person emotional framing in user-side queries affect LLM performance across six benchmark domains, including mathematical reasoning, medical question answering, reading comprehension, commonsense reasoning and social inference. Across models and tasks, static emotional prefixes usually produce only small changes in accuracy, suggesting that affective phrasing is typically a mild perturbation rather than a reliable general-purpose intervention. This stability is not uniform: effects are more variable in socially grounded tasks, where emotional context more plausibly interacts with interpersonal reasoning. Additional analyses show that stronger emotional wording induces only modest extra change, and that human-written prefixes reproduce the same qualitative pattern as LLM-generated ones. We then introduce EmotionRL, an adaptive emotional prompting framework that selects emotional framing adaptively for each query. Although no single emotion is consistently beneficial, adaptive selection yields more reliable gains than fixed emotional prompting. Together, these findings show that emotional tone is neither a dominant driver of LLM performance nor irrelevant noise, but a weak and input-dependent signal that can be exploited through adaptive control.