LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Capability Self-Assessment: Teaching LLMs to Know Their Limits
May 29, 2026Haoyan Yang, Reza Shirkavand, Yukai Jin +3LLM Evaluation Language Model Self-Assessment
Stony Brook University · University of Maryland · Florida State University
Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions
May 29, 2026Wesley Scivetti, Ethan Wilcox, Nathan Schneider +2LLM Evaluation Open-Weight Language Models
Georgetown University · The University of Texas at Austin · Leipzig University & ScaDS.AI Dresden/Leipzig
Preference-Aware Rubric Learning for Personalized Evaluation
May 29, 2026Yilun Qiu, Xiaoyan Zhao, Yang Zhang +7LLM Evaluation LLM Alignment
National University of Singapore · Xiaohongshu Inc. · The University of Tokyo
Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task
May 29, 2026Bart Evelo, Meaghan Fowlie, Denis PapernoLLM Evaluation LLM Grounding
RealityTest: How People Probe AI Identity and Whether Models Disclose It
May 29, 2026Anna Gausen, Sarenne Wallbridge, Bessie O'Dell +2LLM Evaluation Language Model Safety Evaluation
AI Security Institute, London, UK
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
May 29, 2026Tom Lucas, Alessio Buscemi, Alfredo Capozucca +2LLM Evaluation AI Accountability
Luxembourg Institute of Science and Technology (LIST), Luxembourg, University of Luxembourg, Luxembourg, and Scriptor Artis, France · Luxembourg Institute of Science and Technology (LIST), Luxembourg · University of Luxembourg, Luxembourg +1
Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory
May 29, 2026Han Zhang, Zihao Tang, Xin Yu +8LLM Evaluation Memory-Augmented Language Models
Center for Applied Statistics, Renmin University of China · School of Statistics, Renmin University of China · Microsoft
How Much Do LLMs Know About Chinese Zero Pronouns?
May 29, 2026Yifei Li, Guanyi Chen, Tingting HeLLM Evaluation
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, National Language Resources Monitoring and Research Center for Network Media, School of Computer Science, Central China Normal University
Office Comprehension Benchmark
May 29, 2026Firoz Shaik, Mateus Picanço Lima Gomes, Tanvir Aumi +17Document Understanding LLM Evaluation
Microsoft · † Work done while at Microsoft.
Pairwise Reference Alignment as a Model-Level Ordinal Observable
May 29, 2026Mujing LiHuman Preference Evaluation LLM Evaluation
Independent Researcher
Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs
May 29, 2026Weijia Zhang, Ruiqi Chen, Yunze Xiao +1LLM Evaluation Moral Reasoning in Language Models
University of Illinois Urbana-Champaign · University of Michigan · Carnegie Mellon University +1
Bounded Behavioral Indistinguishability for Black-Box LLM Distillation
May 28, 2026Munawar HasanLLM Evaluation Language Model Distillation
Michigan Technological University, Houghton, MI, USA
SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
May 28, 2026Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +1AI for Science LLM Evaluation
University of Maryland, College Park
Resolution Diagnostics for Paired LLM Evaluation
May 28, 2026Anany KotawalaLLM Evaluation Pairwise Comparison
Princeton University, Princeton, NJ, USA.
ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
May 28, 2026A. J. Lew, Y. Cao, M. J. BuehlerLLM Evaluation Scientific Reasoning
Unreasonable Labs · Unreasonable Labs Mountain View, CA 94043, USA
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
May 28, 2026Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1Multilingual Language Model Evaluation LLM Evaluation
National University of Singapore
Protocol for evaluating ChatGPT in biomedical association generation and verification using a RAG-enabled, cross-model majority voting workflow
May 28, 2026Ahmed Abdeen Hamed, Luis M. RochaLLM Evaluation LLM Answer Verification
Department of Biochemistry, University of Nebraska-Lincoln, Lincoln, NE 68588, USA · School of Systems Science & Industrial Engineering, Binghamton University, Binghamton, NY 13902, USA · Universidade Catόlica Portuguesa, Catόlica Biomedical Research Centre, Lisbon, Portugal
SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?
May 28, 2026Jiamin Chen, Yidi Wu, Qiexiang Wang +6LLM Evaluation LLM-as-a-Judge
1ByteDance Inc. · 2City University of Hong Kong
Teaching Values to Machines: Simulating Human-Like Behavior in LLMs
May 28, 2026Asaf Yehudai, Naama Rozen, Ariel GeraLLM Evaluation Human Behavior Simulation
The Hebrew University of Jerusalem · Tel-Aviv University · IBM Research
Latent Performance Profiling of Large Language Models
May 28, 2026Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya +7LLM Evaluation LLM Interpretability
Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi India · Yardi School of Artiﬁcial Intelligence, Indian Institute of Technology Delhi, New Delhi India · Hewlett Packard Enterprise, India +5
Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs
May 28, 2026Mahjabin Nahar, Nafis Irtiza Tripto, Aiping Xiong +2LLM Evaluation Language Model Bias Evaluation
The Pennsylvania State University
ExCAM: Explainable Cultural Awareness Metrics
May 28, 2026Christoph Leiter, Haiyue Song, Hour Kaing +4LLM Evaluation Language Model Generation Evaluation
University of Mannheim, Germany · National Institute of Information and Communications Technology · University of Technology Nuremberg, Germany
NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models
May 28, 2026Anany KotawalaData Leakage LLM Evaluation
Princeton University, Princeton, NJ, USA.
Social Reasoning in Machines: Investigating Collective Truth-Seeking Dynamics in Large Language Model Debate
May 28, 2026Tom PecherLLM Evaluation Hallucination in Language Models
Department of Computer Science University of Bath
PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing
May 28, 2026Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł +2LLM Evaluation Scholarly Peer Review
Department of Artificial Intelligence Wrocław University of Science and Technology
Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
May 28, 2026Guneet KohliLLM Evaluation LLM-as-a-Judge
Apple
ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation
May 28, 2026Yutong Yang, Chenxi Miao, Weikang Li +1LLM Evaluation Personality Modeling in Language Models
Peking University · Baidu Inc
Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
May 28, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Legal NLP
SecondLayer
NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs
May 28, 2026Yunjin Qi, Zhaojun Jiang, Xuan Wu +10LLM Evaluation
Department of Psychology and Behavioral Sciences, Zhejiang University · College of Artificial Intelligence, Zhejiang University · 3Human Machine Interaction Lab, Huawei Technologies Co., Ltd. +1
Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
May 28, 2026Wajdi Zaghouani, Kholoud K. Aldous, Yicheng GaoMultilingual Language Model Evaluation LLM Evaluation
Northwestern University in Qatar

May 29, 2026, cs.AIJ/K move · Enter open · S save

Capability Self-Assessment: Teaching LLMs to Know Their Limits

Haoyan Yang, Reza Shirkavand, Yukai Jin, Jiawei Zhou+2

Stony Brook University · University of Maryland · Florida State University

LLM Evaluation Language Model Self-Assessment LLM Reliability Metacognition in Language Models

Open paper AskPDF arXiv

The ability to recognize one's own limitations and decide whether to solve a problem or delegate is fundamental for reliable intelligent systems. Yet we show that modern large language models systematically lack this ability: across diverse model families and scales, they overestimate their competence and attempt queries they cannot solve. We refer to this ability as Capability Self-Assessment (CSA) and formulate it as a policy-learning problem, aiming to improve self-assessment while preserving the model's original capabilities. Our results show that reinforcement learning teaches CSA effectively, significantly outperforming supervised fine-tuning while preserving original capabilities. In contrast, supervised fine-tuning severely degrades the capabilities the model is meant to assess. Moreover, learned self-assessment behavior generalizes well out of distribution, suggesting that CSA is a transferable model trait. Finally, CSA is practically useful: it improves local-cloud decision making at inference time and provides a signal for targeted data selection during training.

Momentum

Related topics

Latest papers 1,643