LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield
It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories
Sep 3, 2026Yigit Utku BulutLLM Evaluation Language Model Probing
Johannes Kepler University Linz
To What Extent Do Large Language Models Understand Bangla Idioms?
Sep 3, 2026Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi +1LLM Evaluation Pragmatic Reasoning in Language Models
Technical University Dortmund · Research Center Trustworthy Data Science and Security · BRAC University +1
Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory
Sep 3, 2026Divyesh Bommana, Mohammad Saim, Tianyu JiangLLM Evaluation Classification
University of Cincinnati
FrameBench:A Language Understanding Benchmark Based on Frame Semantics
Sep 3, 2026Chihiro Yano, Ryohei SasanoMultilingual Language Model Evaluation LLM Evaluation
Graduate School of Informatics, Nagoya University
What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
Sep 3, 2026Daisuke KikutaLLM Inference Efficiency LLM Evaluation
NTT, Inc.
The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis
Sep 2, 2026Ahmed Asaad, Amr Mohamed, Yang Zhang +1LLM Evaluation LLM Auditing
Durham University Business School · MBZUAI & Ecole Polytechnique · Ecole Polytechnique +1
SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
Sep 2, 2026Michael J. BommaritoLLM Evaluation Library and Information Science
Portions of this work were prepared with assistance from large language models. The author is solely responsible for all content, including any errors or omissions.
User Feedback Provides a Unique Signal that LLMs Can not Detect
Sep 2, 2026Shachar Don-Yehiya, Leshem Choshen, Omri AbendLLM Evaluation RL from Human Feedback
The Hebrew University of Jerusalem · IBM Research · MIT +1
How Output Format Confounds Data Quality and Capability in Instruction Tuning
Sep 2, 2026Chengguang Gan, Hanjun Wei, Yunhao Liang +3LLM Evaluation Instruction Tuning
1Independent Researcher · University of Chinese Academy of Sciences · 3Pusan National University +2
Benchmarking Language Models for Statistical Problem Formulation
Sep 2, 2026Chen Wang, Junzhe Zhao, Xin Cong +2LLM Evaluation
Department of Statistics and Data Science, Tsinghua University
Accurate in space, unreliable in time: how LLMs represent national cultural change
Sep 1, 2026Yalda Daryani, Miranda Bogen, Madeleine I. G. DaeppLLM Evaluation Cultural Evolution
Department of Psychology, University of Southern California · Center for Democracy and Technology
VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages
Sep 1, 2026Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda +4Multilingual Language Model Evaluation LLM Evaluation
Georgia Institute of Technology · University of Texas at Austin
CHARM: Character Hallucination for Multicultural Role Play Benchmark
Sep 1, 2026Sunkyung Han, Nahyeon Park, Gaeun Seo +2LLM Evaluation Large Language Model-Based Role-Play Simulation
Sungkyunkwan University, Suwon, South Korea · Adobe Research, CA, USA
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Sep 1, 2026Muhammed Saeed, Simon RazniewskiLLM Evaluation LLM Auditing
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
Does task decomposition improve automatic NLG evaluation?
Sep 1, 2026Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1LLM Evaluation LLM-as-a-Judge
Amazon, Barcelona, Spain · Amazon, Berlin, Germany
EDRAC: Benchmarking Arabic Dialect Reading Comprehension
Sep 1, 2026Noor Abo Mokh, Kirill Chirkunov, Teresa Lynn +15LLM Evaluation Arabic NLP
Mohamed bin Zayed University of Artificial Intelligence · New York University Abu Dhabi · IBM Research AI +1
Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches
Sep 1, 2026Marco Simnacher, Georg Keilbar, Benjamin König +2LLM Evaluation Privacy Leakage in Language Models
Chair of Statistics Humboldt-Universität zu Berlin Spandauer Str. 1, 10178 Berlin, Germany · Chair Digital Health & Machine Learning Hasso-Plattner-Institut for Digital Engineering Prof.-Dr.-Helmert-Straße 2-3, 14482 Potsdam, Germany
VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences
Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9LLM Evaluation LLM Personalization
1Monash University · 2Singapore Management University · University of Liverpool +1
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM Evaluation Large Language Model-Based Recommendation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Towards a Reliable and Practical Eval Pipeline
Sep 1, 2026Emma Thuong Nguyen, Abhishek GhoseLLM Evaluation LLM-as-a-Judge
Salesforce
Can Large Language Models Forecast What Researchers Study Next?
Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2LLM Evaluation Forecasting Benchmarks
University of Illinois Urbana-Champaign
Value Over Language Model: Detecting Original Contribution in Writing
Sep 1, 2026Vibhhu Sharma, Thorsten Joachims, Sarah DeanLLM Evaluation AI-Generated Text Detection
Cornell University
Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMs
Sep 1, 2026Jainil Dharmil ShahLLM Quantization LLM Evaluation
Elmore Family School of Electrical and Computer Engineering · Purdue University, West Lafayette, Indiana, USA
SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task
Sep 1, 2026Qiming Bao, Neşet Özkan Tan, Siyuan Wang +1Document Understanding LLM Evaluation
University of Auckland · New Zealand · The Chinese University of Hong Kong +1
Predicting Program Exit Code with LLMs and Programming Language Semantics
Sep 1, 2026Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa +2LLM Evaluation Code Language Models
The University of Texas at Austin · Cisco Research
Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random
Sep 1, 2026Cris HuynhLLM Evaluation Pragmatic Reasoning in Language Models
Independent researcher
Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
Sep 1, 2026Jingshen Zhang, Shaoyang Xu, Wenxuan ZhangLLM Evaluation Cultural Alignment
School of Computer Science and Technology, Tianjin University · 2iNLP Lab, Singapore University of Technology and Design
Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?
Aug 31, 2026Qiaoyuan Zheng, Yiqu YangLLM Evaluation Pairwise Ranking
ETH Zurich Zurich, Switzerland
Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries
Aug 31, 2026Phuong Anh Nguyen, Jill Noorily, Matthew Flathers +7LLM Evaluation LLM Auditing
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02446, USA · Psychiatry and Mental Health Department, School of Medicine, Pontificia Universidad Javeriana, Bogotá 110231, Colombia · Tufts University School of Medicine, Tufts University, Boston, MA 02111, USA

Sep 3, 2026, cs.CLJ/K move · Enter open · S save

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

Danting Zhang, Bei Peng, Robert Loftin

1Independent Researcher · University of Sheffield

LLM Safety Benchmarks LLM Evaluation Content Moderation

Open paper AskPDF arXiv

Large language models (LLMs) demonstrate strong performance on standard content moderation benchmarks. However, these benchmarks often aggregate multiple moderation criteria into a single label, making it unclear whether models can disentangle them and reliably apply each criterion when making decisions. To study whether LLMs exhibit criterion-conditioned behaviour, we introduce Diagnostic Evaluation of COntent (DECO), a criterion-independent factorisation of content that enables controlled, criterion-level evaluation. We also introduce pairwise evaluation to compare model outputs across different criteria for the same input. Across four moderation datasets and four LLMs, we find that strong benchmark performance can hide substantial failures at the criterion level. Models struggle most when correct decisions depend not on overall harmfulness, but on the specific aspect of the content that the criterion requires them to assess. Our results highlight a key limitation of current content moderation benchmarks: strong performance on aggregated labels does not provide sufficient evidence that LLMs can reliably evaluate content with respect to individual moderation criteria. These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.

Momentum

Related topics

Latest papers 1,631