LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Apr 20, 2026Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid +2LLM Evaluation Benchmark Contamination
Oracle AI
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Apr 20, 2026Rongyuan Tan, Jue Zhang, Zhuozhao Li +3LLM Evaluation Feature Attribution
1Southern University of Science and Technology, China · *Work done during an internship at Microsoft. · 2Microsoft
MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
Apr 20, 2026Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav +2Mental Health Counseling LLM Evaluation
Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · School of Computing and Information Systems, Singapore Management University, Singapore
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
Apr 20, 2026Jon-Paul CacioliLLM Evaluation Selective Prediction
Independent Researcher, Melbourne, Australia
Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
Apr 20, 2026Jon-Paul CacioliLLM Evaluation Psychometric Validation
Independent Researcher, Melbourne, Australia
Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance
Apr 19, 2026Parker Seegmiller, Sarah Masud PreumLLM Evaluation Instruction Following
Department of Computer Science Dartmouth College Hanover, NH, USA
KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models
Apr 19, 2026Xiao Zhang, Qianru Meng, Yongjian Chen +2LLM Evaluation Compositional Reasoning
University of Groningen · LIACS, Leiden University
Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier
Apr 19, 2026Jazmia HenryReward Hacking RL for Language Models
University of Oxford
Evaluating Hallucinations in Domain-Adapted Large Language Models
Apr 19, 2026Sanchita Porwal, Sai Prasath S, Xingjian Bi +1LLM Evaluation Domain Adaptation
College of Computing Georgia Institute of Technology Atlanta, GA 30318, USA
ArgBench: Benchmarking LLMs on Computational Argumentation Tasks
Apr 19, 2026Yamen Ajjour, Carlotta Quensel, Nedim Lipka +1LLM Evaluation Argument Mining
Leibniz University Hannover · Adobe Research · Leibniz University Hannover, L3S
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Apr 19, 2026Miaosen Chai, Wang Bill Zhu, Shangshang Wang +5LLM Evaluation Automated Program Repair
University of Southern California · Microsoft · University of Wisconsin–Madison +1
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Apr 19, 2026Jingyi Ren, Ante Wang, Yunghwei Lai +5LLM Evaluation Selective Prediction
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
From Legal Text to Executable Decision Models: Evaluating Structured Representations for Legal Decision Model Generation
Apr 18, 2026David GrausLLM Evaluation Legal Document Analysis
University of Amsterdam · Amsterdam, The Netherlands
SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction
Apr 18, 2026Hangxiao Zhu, Yuyu Zhang, Ping Nie +1LLM Evaluation
Texas A&M University · Verdent AI · University of Waterloo
Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning
Apr 18, 2026Isabella Luong, Joyee Chen, Arturs Kanepajs +5LLM Evaluation Moral Reasoning in Language Models
SPAR · Compassion Aligned Machine Learning · NUS +2
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text
Apr 18, 2026Refael Shaked Greenfeld, Reut TsarfatyCoreference Resolution LLM Evaluation
Bar-Ilan University
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM Evaluation Language Model Generation Evaluation
NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE
Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation
Apr 18, 2026Michelle Star, Andrew Aquilina, Yu-Ru LinLLM Evaluation Emotional Support Conversation
School of Computing and Information, University of Pittsburgh
PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
Apr 18, 2026Yuhe Wu, Guangyu Wang, Yuran Chen +6LLM Evaluation Hallucination in Language Models
HKUST(GZ) · NYUSH · DUFE +2
Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models
Apr 18, 2026Yuchen Pan, Soung Chang LiewLLM Evaluation Financial Reasoning in Language Models
Department of Information Engineering, The Chinese University of Hong Kong
Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms
Apr 18, 2026Sree Bhattacharyya, Manas Mehta, Leona Chen +4LLM Evaluation Cultural Bias in Language Models
The Pennsylvania State University · Duke University · Northeastern University +1
Machine individuality: Separating genuine idiosyncrasy from response bias in large language models
Apr 18, 2026Valentin Kriegmair, Dirk U. WulffLLM Evaluation Personality Modeling in Language Models
Center for Adaptive Rationality, Max Planck Institute for Human Development, 14195 Berlin, Germany · Center for Cognitive and Decision Sciences, Department of Psychology, University of Basel, 4055 Basel, Switzerland
Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
Apr 17, 2026Yang Liu, Hongming Li, Melissa Xiaohui Qin +2LLM Evaluation
University of Science and Technology Beijing · 2State Key Laboratory of General Artificial Intelligence, BIGAI
Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design
Apr 17, 2026Shriram Chennakesavalu, Kirill Shmilovich, Hayley Weir +5LLM Evaluation Molecular Property Prediction
Prescient Design, Genentech, South San Francisco, CA, USA
No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
Apr 17, 2026Hitesh Mehta, Arjit Saxena, Garima Chhikara +1Multilingual Language Model Evaluation LLM Evaluation
Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India
From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
Apr 17, 2026Van-Truong LeLLM Evaluation Legal Document Analysis
University of Science, VNUHCM Vietnam National University Hanoi Open University Ho Chi Minh, Vietnam · Vietnam National University · Hanoi Open University
SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation
Apr 17, 2026Deshan Sumanathilaka, Nicholas Micallef, Julian Hough +1LLM Evaluation Few-Shot Prompting
Department of Computer Science, Swansea University, Wales, UK
Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models
Apr 17, 2026Reham Alharbi, Valentina Tamma, Terry R. Payne +1LLM Evaluation
Taibah University, Madinah, Saudi Arabia · University of Liverpool, Liverpool, UK
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Apr 17, 2026Jiacheng Shen, Masato Hagiwara, Milad Alizadeh +9LLM Evaluation Question Answering
2NYU Center for Data Science; NYU Shanghai · 1Earth Species Project · 3NYU Shanghai Center for Data Science; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai

Apr 20, 2026J/K move · Enter open · S save

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Graham Horwood+1

Oracle AI

LLM Evaluation Benchmark Contamination Text-to-SQL Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL datasets-Spider, SParC, CoSQL, and the newer BIRD benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall's tau with bootstrap confidence intervals. By aligning these robustness trends with benchmark release dates, we observe a clear temporal gradient: older benchmarks such as Spider exhibit the strongest negative values and thus the highest likelihood of training leakage, whereas the more recent BIRD dataset shows minimal sensitivity and appears largely uncontaminated. Together, these findings highlight the importance of temporally contextualized, syntactic-probing evaluation for trustworthy NL2SQL benchmarking.