LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
May 24, 2026Jinyan Su, Claire CardieLLM Evaluation Clarification Question Generation
1Cornell University
AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
May 24, 2026Michael Hardy, Anka Reuel, Lijin Zhang +6LLM Evaluation Benchmark Design
Stanford University, Stanford, CA, USA · BroadMetrics, Edison, NJ, USA
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
May 24, 2026Minwei Kong, Chonghe Jiang, Ao Qu +24LLM Evaluation Large Language Model-Guided Optimization
Singapore-MIT Alliance for Research and Technology · Massachusetts Institute of Technology · Northeastern University +10
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
May 24, 2026Russell Yang, Ruishi Chen, Pierce Kelaita +6Human Preference Evaluation LLM Evaluation
Stanford University · Snorkel AI · Harvey
By Their Fruits You Will Know Them: Comparing Formalizations of Law by the Decisions They Encode
May 24, 2026Julius Vernie, Matthias GrabmairLLM Evaluation Differential Testing
Technical University of Munich (TUM) · Technical University of Munich
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
May 24, 2026Yoav Gur-Arieh, Ana Marasović, Mor GevaCoT Faithfulness LLM Evaluation
Tel Aviv University · University of Utah
Large Language Model Selection with Limited Annotations
May 24, 2026Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich
Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges
May 24, 2026Xianglin Yang, Bryan Hooi, Gelei Deng +2LLM Evaluation LLM-as-a-Judge
School of Computing, National University of Singapore, Singapore · Nanyang Technological University, Singapore
Repeated Sequences Reveal Gaps between Large Language Models and Natural Language
May 24, 2026Kumiko Tanaka-IshiiLLM Evaluation Language Model Generation Evaluation
Waseda University, Japan.
Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models
May 23, 2026Feng Chen, Justin Tauscher, Changye Li +7LLM Evaluation Multi-Agent LLMs
Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, USA · Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA · Department of Psychology, Louisiana State University, Baton Rouge, LA, USA +1
TS-Skill: A Benchmark for Evaluating Analytical Skills in Time-Series Question Answering
May 23, 2026Liying Han, Kang Yang, Oliver Wang +9LLM Evaluation Time Series Reasoning
University of California, Los Angeles · Samsung Research America · Carnegie Mellon University +2
Emotional intelligence in large language models is fragmented across perception, cognition, and interaction
May 23, 2026Minghao Lv, Lu Chen, Enchang Zhang +6LLM Evaluation LLM Alignment
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China. · MoE Key Lab of Artificial Intelligence. · Jiangsu Key Lab of Language Computing, China. +1
When Mean CE Fails: Median CE Can Better Track Language Model Quality
May 23, 2026Hao Guo, Simon Dennis, Rivaan Patil +1LLM Evaluation LLM Training
1i14 · University of Melbourne · University of California, Santa Cruz
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
May 23, 2026Ali Şenol, Garima Agrawal, Huan LiuLLM Evaluation Language Model Robustness
Department of Computer Engineering, Tarsus University, Takba¸s M. Kartaltepe S., Tarsus 33400, Mersin, Türkiye · School of Computing and Augmented Intelligence (SCAI), Arizona State University (ASU), Tempe, AZ 85281, USA · HumaConn AI Consulting, Queen Creek, AZ 85140, USA
AstroMind: A High-Fidelity Benchmark for Spacecraft Behavior Reasoning Based on Large Language Models
May 23, 2026Hao Liu, Siyuan Yang, Qinglei Hu +1LLM Evaluation
Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · KTH Royal Institute of Technology, Stockholm, Sweden · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China +1
EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy
May 23, 2026Jinzhao Li, Yinuo Chen, Dongxu Piao +9Spatial Reasoning Benchmarks LLM Evaluation
College of AI, Tsinghua University · 2ByteDance · 3State Key Laboratory of General Artificial Intelligence, BIGAI
LLMTabBench: Evaluating LLMs on Binary Tabular Classification From Zero to Few Shots
May 23, 2026Daria Grushina, Kseniia Kuvshinova, Alina Kostromina +3Zero-Shot Learning LLM Evaluation
Sb AI Lab, HSE University, NES · NES · Russia +4
How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description
May 23, 2026Abraham Camelo-Guerrero, Jairo Diaz-RodriguezLLM Evaluation Clustering
School of Information Technology York University Toronto, Ontario M3J 1P3 · Department of Mathematics and Statistics York University Toronto, Ontario M3J 1P3
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
May 23, 2026Noel ThomasLLM Evaluation Language Model Robustness
Mohamed bin Zayed University of Artificial Intelligence · Abu Dhabi, UAE
Enhancing Reliability in LLM-Based Secure Code Generation
May 22, 2026Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1LLM Evaluation CoT Reasoning
the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA
LLMs Show No Signs Of Individuated Metacognition
May 22, 2026M. Moran, Mark WhitingLLM Evaluation Confidence Estimation in Language Models
Pareto AI
DRInQ: Evaluating Conversational Implicature with Controlled Context Variation
May 22, 2026Hirona Jacqueline Arai, Xiang RenLLM Evaluation Pragmatic Reasoning in Language Models
University of Southern California
How Well Do Models Follow Their Constitutions?
May 22, 2026Arya Jakkli, Senthooran Rajamanoharan, Neel NandaLLM Evaluation Language Model Safety Evaluation
CentraleSup´elec · †Work conducted during the ML Alignment & Theory Scholars (MATS) program. · Google DeepMind
Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
May 22, 2026Ashok Chandrasekar, Jason KrambergerLLM Evaluation LLM Serving
Google
Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation
May 22, 2026Mariam Barakat, Ekaterina KochmarLLM Evaluation LLM Grounding
Mohamed bin Zayed University of Artificial Intelligence, UAE
Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline
May 22, 2026Will Jack, Noah Lehman, Keller Maloney +1Prompt Sensitivity LLM Evaluation
Unusual
NLG Evaluation: Past, Present, Future
May 22, 2026Ehud ReiterLLM Evaluation Language Model Generation Evaluation
Dept of Computing Science University of Aberdeen Aberdeen, UK
How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
May 22, 2026Björn Nieth, Marianna Gracheva, Michaela Mahlberg +2LLM Evaluation Language Model Generation Evaluation
Department Artificial Intelligence in Biomedical Engineering (AIBE), FAU Erlangen-Nürnberg, Germany · Department of Digital Humanities and Social Studies (DHSS), FAU Erlangen-Nürnberg, Germany · University of Birmingham, United Kingdom +2
Naturalistic measure of social norms alignment
May 22, 2026Yevhen Kostiuk, Kenneth Enevoldsen, Peter Bjerregaard Vahlstrup +2Social Dilemmas LLM Evaluation
Aarhus University
Instance-Optimal Estimation with Multiple LLM Judges on a Budget
May 22, 2026Junghyun Lee, Sanghwa Kim, Yassir Jedra +2Parameter Estimation LLM Evaluation
KAIST AI · ICL EEE · KTH EECS

May 24, 2026, cs.CLJ/K move · Enter open · S save

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

Jinyan Su, Claire Cardie

1Cornell University

LLM Evaluation Clarification Question Generation LLM Decision-Making

Open paper AskPDF arXiv

User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user's intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions.

Momentum

Related topics

Latest papers 1,643