LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
Jun 17, 2026Han Chen, Ming Li, Chenguang Wang +4LLM Evaluation Educational Assessment
MBZUAI · University of Maryland · Virginia Tech
TW-LegalBench: Measuring Taiwanese Legal Understanding
Jun 17, 2026Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4LLM Evaluation Legal Judgment Prediction
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA
DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
Jun 17, 2026Patrick Cooper, Alvaro VelasquezLLM Evaluation Abductive Reasoning
University of Colorado Boulder
Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports
Jun 16, 2026Ahmed Ryan, Saad Sakib Noor, Md Erfan +3LLM Evaluation Cyber Threat Intelligence
The University of Alabama · The University of Dhaka
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Jun 16, 2026Michèle FinckLLM Evaluation Legal Reasoning
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen.
How Inference Compute Shapes Frontier LLM Evaluation
Jun 16, 2026Jessica McFadyen, Ole Jorgensen, Harry Coppock +2LLM Evaluation Language Model Generation Evaluation
AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA
DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue
Jun 16, 2026Guillermo Gil de Avalle, Laura Maruster, Shaina Raza +1LLM Evaluation LLM Grounding
University of Groningen, PO Box 72, 9700 AB Groningen, The Netherlands · Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON, Canada
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
Jun 16, 2026Rui Wen, Lu Sun, Jiayang Liu +3Open-Ended Generation LLM Evaluation
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2
Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations
Jun 16, 2026Mika Mäntylä, Patricia Matsubara, Katia Romero Felizardo +5LLM Evaluation Evidence Selection
University of Helsinki, Finland · UFMS, Brazil · UTFPR – Federal University of Technology - Paraná, Brazil +3
Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
Jun 16, 2026Ryo Fukuda, Takatomo Kano, Siddhant Arora +7LLM Evaluation Multimodal Large Language Models
NTT, Inc., Japan · Language Technologies Institute, Carnegie Mellon University, USA
The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes
Jun 16, 2026Marina Mancoridis, Zoë HitzigLLM Evaluation Reasoning Consistency in Language Models
Massachusetts Institute of Technology · Harvard Society of Fellows
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Jun 16, 2026Jeffrey G. Wang, Jason Wang, Marvin Li +1LLM Evaluation Membership Inference Attacks
Harvard University · Harvard Business School
NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Jun 16, 2026Logan Mann, Abdur Rahman, Mohammad Saifullah +2LLM World Models Multilingual Language Model Evaluation
University of California, Santa Barbara · Pocket FM
Do Large Language Models Always Tell The Same Stories?
Jun 15, 2026Thennal DK, Hans Ole HatzelLLM Evaluation Diverse Text Generation
Language Technology Group University of Hamburg
Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
Jun 15, 2026Yanan LongLLM Evaluation Benchmark Auditing
StickFlux Labs
Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods
Jun 15, 2026Francesca Carlon, Brecht Verbeken, Vincent Ginis +1LLM Evaluation AI-Assisted Scientific Research
1Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussels, Belgium · 2imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives
Jun 15, 2026Luca ZhouLLM Evaluation Language Model Safety Evaluation
Sapienza University of Rome
Evaluating LLM Personalization via Semantic Constraint Verification
Jun 15, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM Evaluation LLM Personalization
University of New South Wales · 2Mohamed bin Zayed University of Artificial Intelligence · 3The Technology Innovation Institute +1
Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors
Jun 15, 2026Junyi Yao, Zihao Zheng, Baichuan LiLLM Evaluation Intelligent Tutoring Systems
Washington University in St. Louis, St. Louis, MO, USA · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX, USA
GRACE: Step-Level Benchmark for Faithful Reasoning over Context
Jun 15, 2026Hoang Pham, Dong Le, Anh Tuan LuuCoT Faithfulness Contextual Faithfulness
Nanyang Technological University · VinUniversity
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Jun 14, 2026Nafiseh Nikeghbal, Amir Hossein Kargaran, Shaghayegh Kolli +1LLM Evaluation LLM Reliability
1Technical University of Munich · Munich Center for Machine Learning · 1Technical University of Munich 2 Munich Center for Machine Learning +2
SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity
Jun 14, 2026Yifan Mo, Xiao Fu, Yue Su +4AI for Science LLM Evaluation
Vrije Universiteit Amsterdam, Amsterdam, Netherlands · Wageningen University & Research, Netherlands
A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
Jun 14, 2026Weixiao Zhou, Gengyao Li, Xianfu Cheng +3LLM Evaluation Long-Context Language Model Evaluation
CCSE, Beihang University · Fanyu AI Laboratory · MAIS, CASIA
FinBalance: A Multi-Document Accounting Reconciliation Benchmark
Jun 14, 2026Sasank Tumpati, Devansh Agarwal, Ayush Kedia +6Financial Services LLM Evaluation
BITS Pilani · KIIT Bhubaneswar · University of Oxford
Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes
Jun 14, 2026Costa GeorgantasLLM Evaluation LLM Reliability
David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM
Jun 14, 2026Hans Weytjens, Ingo WeberLLM Evaluation Predictive Process Monitoring
Technical University of Munich, School of CIT, Germany · Fraunhofer Gesellschaft, Munich, Germany
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Mitsubishi Heavy Industries, Ltd., Research & Innovation Center, Heat Transfer Research Department, Takasago, Hyogo 676-8686, Japan
Is Code Better Than Language for Algorithmic Reasoning
Jun 14, 2026Terry Tong, Yu Feng, Surbhi Goel +1LLM Evaluation LLM Tool Use
University of Pennsylvania · 2Oracle AI
EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management
Jun 14, 2026Rongzhi Zhu, Xiang Huang, Yuchuan Wu +8RL for Language Models LLM Evaluation
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Qwen-Character Team , Alibaba Group

Jun 17, 2026, cs.CLJ/K move · Enter open · S save

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

Han Chen, Ming Li, Chenguang Wang, Yijun Liang+3

MBZUAI · University of Maryland · Virginia Tech

LLM Evaluation Educational Assessment Psychometric Validation

Open paper AskPDF arXiv

Item discrimination is a fundamental psychometric property of educational assessment, which measures whether an item meaningfully distinguishes students with higher proficiency from students with lower proficiency. While various existing works have explored whether large language models (LLMs) can estimate item difficulty, it remains unclear whether they can capture item discrimination. In this work, we evaluate 42 proprietary and open-weight LLMs in zero-shot settings using two complementary approaches: direct discrimination prediction, where models explicitly estimate an item's discrimination value from its content, and response-based Classical Test Theory (CTT) calibration, where LLM answers are treated as synthetic student responses to compute discrimination scores. Our results show that direct prediction yields weak alignment with human-calibrated discrimination: the best-performing model reaches only a Spearman correlation of 0.152. Response-based CTT calibration provides a stronger but still limited signal, with the all-persona synthetic respondent pool reaching a Spearman correlation of 0.241. These findings highlight item discrimination as an open challenge for LLM-based psychometric evaluation: current LLMs contain non-random discrimination-relevant signal, but they do not yet reliably capture how assessment items distinguish human students.

Momentum

Related topics

Latest papers 1,631