LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

Position: The Term "Machine Unlearning" Is Overused in LLMs
May 8, 2026Sangyeon Yoon, Yeachan Jun, Albert NoLLM Evaluation Machine Unlearning
Department of Artificial Intelligence, Yonsei University, Seoul, Korea.
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
May 8, 2026Moaath Alshaikh, Tasneem Alshaher, Ricardo Vieira +7Software Engineering LLM Evaluation
Federal University of Bahia · State University of Feira de Santana · Federal University of Sergipe +1
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
May 8, 2026Saloni Garg, Amit SagtaniLLM Evaluation LLM-as-a-Judge
Independent Researcher San Francisco, California, USA · San Francisco State University San Francisco, California, USA
From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs
May 8, 2026Hanmeng Liu, Shichao Weng, Xiulai Liu +3Logical Reasoning LLM Evaluation
Hainan University, Haikou, China · Fudan University, Shanghai, China
FAME: Forecasting Academic Impact via Continuous-Time Manifold Evolution
May 8, 2026Jianrong Ding, Jianyuan Zhong, Zhengyan Shi +1AI for Science LLM Evaluation
Department of Computer Science and Engineering The Chinese University of Hong Kong
The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
May 8, 2026Zekai Tong, Ruiyao Xu, Aryan Shrivastava +2LLM Evaluation LLM Reliability
University of Chicago · 2Northwestern University
MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries
May 8, 2026Zixuan Xie, Xinyu Liu, Shangtong ZhangLLM Evaluation LLM Agent Evaluation
University of Virginia
The Position Curse: LLMs Struggle to Locate the Last Few Items in a List
May 8, 2026Zhanqi Zhang, Hua-Dong Xiong, Robert C. Wilson +3LLM Evaluation LLM Fine-Tuning
UC San Diego · Georgia Tech · New York University
SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models
May 8, 2026George Boateng, Naafi Ibrahim, Samuel John +8Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich, Switzerland · Kwame AI Inc., U.S. · Ashesi University, Ghana +1
An Interpretable and Scalable Framework for Evaluating Large Language Models
May 7, 2026Xinhao Qu, Qiang Heng, Hao Zeng +1LLM Evaluation Item Response Theory
Department of Statistics, University of California, Riverside · School of Statistics and Data Science, Southeast University · Department of Statistics and Data Science, Southern University of Science and Technology
ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
May 7, 2026Inder Preet, Shuxin Lin, Dhaval PatelLLM Evaluation Language Model Generation Evaluation
IBM, Dublin · IBM, New York
MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media
May 7, 2026Souvik Pramanik, S. M. Riaz Rahman Antu, Shak Mohammad Abyad +2LLM Evaluation LLM-Assisted Annotation
North South University, Dhaka, Bangladesh
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Can LLMs Take Retrieved Information with a Grain of Salt?
May 7, 2026Behzad Shayegh, Mohamed Osama Ahmed, Fred Tung +1Retrieval-Augmented Generation LLM Evaluation
RBC Borealis
How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
May 7, 2026Chun Zheng, Lianlong Wu, Bingqian Li +2LLM Evaluation LLM Reasoning
University of Science and Technology of China · University of Oxford
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
May 7, 2026Sunnie S. Y. Kim, Margit Bowler, Leon A GatysLLM Evaluation LLM Prompting
Apple
IntentGrasp: A Comprehensive Benchmark for Intent Understanding
May 7, 2026Yuwei Yin, Chuyuan Li, Giuseppe CareniniLLM Evaluation Intent Classification
Department of Computer Science, University of British Columbia
Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
May 7, 2026Jai Moondra, Ayela Chughtai, Bhargavi Lanka +1Multilingual Language Model Evaluation LLM Evaluation
Carnegie Mellon University · MIT Sloan School of Management
When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
May 7, 2026Sushant Gautam, Finn Schwall, Annika Willoch Olstad +6LLM Safety Benchmarks LLM Evaluation
1Simula Metropolitan Center for Digital Engineering · 2Oslo Metropolitan University · 4Simula Research Laboratory +2
Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
May 7, 2026Hailey Onweller, Elias Lumer, Austin Huber +3LLM Evaluation Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Spend Where It Counts: Dynamic Budget Allocation for Multi-Turn LLM Evaluation
May 7, 2026Shai Feldman, Yaniv RomanoLLM Evaluation Language Model Safety Evaluation
Department of Computer Science, Technion IIT, Israel · Department of Electrical and Computer Engineering, Technion IIT, Israel
Ex Ante Evaluation of AI-Induced Idea Diversity Collapse
May 7, 2026Nafis Saami Azad, Raiyan Abdul BatenLLM Evaluation Computational Creativity
Bellini College of Artificial Intelligence, Cybersecurity, and Computing University of South Florida Tampa, FL–33620, USA
Towards Emotion Consistency Analysis of Large Language Models in Emotional Conversational Contexts
May 7, 2026Sneha Oram, Ojaswita Bhushan, Tejomay Kishor Padole +2LLM Evaluation Language Model Robustness
Indian Institute of Technology Bombay, India
SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
May 7, 2026Jamelle Watson-Daniels, Himaghna Bhattacharjee, Skyler Wang +11LLM Evaluation Rubric-Based Evaluation
1Meta · 2McGill University · 3Handshake AI +1
Rethinking Vacuity for OOD Detection in Evidential Deep Learning
May 7, 2026Claire McNamaraLLM Evaluation Evidential Deep Learning
∗Accenture Labs · †Trinity College Dublin.
Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonLLM Safety Benchmarks LLM Evaluation
University of Bath
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLLM Evaluation Language Model Generation Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence
Teaching LLMs Program Semantics via Symbolic Execution Traces
May 7, 2026Jonas Bayer, Stefan Zetzsche, Olivier Bouissou +3LLM Evaluation Continued Pretraining
University of Cambridge · Amazon Web Services
Visual Fingerprints for LLM Generation Comparison
May 7, 2026Amal Alnouri, Andreas Hinterreiter, Christina Humer +2LLM Evaluation Language Model Fingerprinting
Johannes Kepler University Linz · ETH Zürich · Independent Researcher

May 8, 2026J/K move · Enter open · S save

Position: The Term "Machine Unlearning" Is Overused in LLMs

Sangyeon Yoon, Yeachan Jun, Albert No

Department of Artificial Intelligence, Yonsei University, Seoul, Korea.

LLM Evaluation Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

Large language models increasingly face demands to "forget" training data, knowledge, or behaviors due to regulatory deletion obligations, copyright/licensing disputes, and safety or product-policy requirements. This position paper argues that machine unlearning is overused as a term in LLM research and should be reserved for dataset-defined deletion: removing the training influence of a precisely specified forget set such that the resulting model is approximately indistinguishable from retraining without that data. We contend that many tasks currently labeled "unlearning" (e.g., refusal for harmful requests, entity/knowledge removal, or targeted suppression) pursue different, often policy-dependent objectives and therefore require different terminology and baselines (e.g., alignment, suppression, editing, obfuscation). We further argue that this confusion is not cosmetic: because papers make different implicit guarantees under the same label, metrics and benchmarks are frequently reused outside their intended scope, rewarding surface-level non-disclosure (e.g., low ROUGE/forget accuracy) even when retraining-equivalence is not tested and derived capabilities remain. We conclude by calling for stricter terminology tied to explicit guarantees and reference models, and for evaluations that match the claimed objective.