LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers
May 13, 2026Samuel Schapiro, Alexi Gladstone, Jonah Black +1Creativity Assessment LLM Evaluation
University of Illinois Urbana-Champaign
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
May 13, 2026Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen +7Legal NLP LLM Evaluation
Visual Analysis and Perception Lab, Aalborg University · Pioneer Center for AI, Denmark · Center of Excellence for Global Mobility Law, University of Copenhagen +1
PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users
May 13, 2026Hannah Rose Kirk, Liu Leqi, Fanzhi Zeng +4LLM Evaluation LLM Sycophancy
University of Oxford · UK AI Security Institute · University of Texas at Austin +2
LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information
May 13, 2026Stef van BuurenLLM Evaluation Confidence Estimation in Language Models
1. TNO - Netherlands Organization for Applied Scientific Research, Leiden · 2. Dept. of Methodology and Statistics, University of Utrecht
TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting
May 13, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation RL for Language Model Reasoning
Department of Statistic and Data Science Northwestern University
The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
May 13, 2026Adil AminLLM Evaluation Forecasting Benchmarks
ZEHEN Labs
Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators
May 12, 2026Heejin Do, Shashank Sonkar, Mrinmaya SachanLLM Evaluation LLM Sycophancy
ETH Zürich, ETH AI Center · University of Central Florida · ETH Zürich
DisaBench: A Participatory Evaluation Framework for Disability Harms in Language Models
May 12, 2026Eugenia Kim, Ioana Tanase, Christina MallonLLM Safety Benchmarks LLM Evaluation
Microsoft
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
May 12, 2026Zenghui Zhou, Man Li, Xiaoke Fang +3LLM Evaluation Automated Software Testing
School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Haidian District, Beijing, 100191, China
Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models
May 12, 2026Mohammed Saidul Islam, Negin Baghbanzadeh, Farnaz Kohankhaki +5LLM Evaluation Benchmark Design
1Vector Institute · 2York University
Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring
May 12, 2026Jamshid Mozafari, Bhawna Piryani, Adam JatowtLLM Evaluation
University of Innsbruck, Innsbruck, Austria
Scalable Token-Level Hallucination Detection in Large Language Models
May 12, 2026Rui Min, Tianyu Pang, Chao Du +2LLM Evaluation Hallucination in Language Models
Sea AI Lab · Hong Kong University of Science and Technology · 1Sea AI Lab 2Hong Kong University of Science and Technology +2
MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
May 12, 2026Rezarta Islamaj, Robert Leaman, Joey Chan +13Multi-Hop QA LLM Evaluation
National Library of Medicine, Division of Intramural Research, Bethesda, MD, US · University of Illinois at Urbana-Champaign, Department of Computer Science, Urbana, IL, US · University of Michigan Medical School, Ann Arbor, Michigan, US
PreScam: A Benchmark for Predicting Scam Progression from Early Conversations
May 12, 2026Weixiang Sun, Shang Ma, Yiyang Li +5LLM Evaluation Cybersecurity
University of Notre Dame · Arizona State University
MolDeTox: Evaluating Language Model's Stepwise Fragment Editing for Molecular Detoxification
May 12, 2026Jueon Park, Wonjune Jang, Jiwoo Lee +2Molecular Optimization LLM Evaluation
Korea University · Myongji University · AIGEN Sciences
Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach
May 12, 2026Andrea Morandi, Mahesh ViswanathanLLM Evaluation Bayesian Inference
Cisco Systems, Inc.
PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior
May 12, 2026James Flemings, Murali AnnavaramLLM Evaluation User Modeling
University of Southern California
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
May 12, 2026Xiaoyuan Li, Yuzhe Wang, Moxin Li +6LLM Evaluation Benchmark Construction
University of Science and Technology of China · 2Alibaba Group · 3National University of Singapore
On the Limitations of Large Language Models for Conceptual Database Modeling
May 12, 2026Arthur F. Siqueira, Carlos D. S. Nogueira, Eduarda Farias +2LLM Evaluation LLM Prompting
1Systems and Computing Department Federal University of Campina Grande (UFCG)2026 Campina Grande – PB – Brazil
On Predicting the Post-training Potential of Pre-trained LLMs
May 12, 2026Xiaoyuan Li, Yubo Ma, Kexin Yang +5LLM Evaluation Language Model Post-Training
University of Science and Technology of China · Alibaba Group · National University of Singapore
How Well Do Large Language Models Capture Human Personality?
May 12, 2026Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2LLM Evaluation Representation Collapse
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo
Procedural-skill SFT across capacity tiers: A W-Shaped pre-SFT Trajectory and Regime-Asymmetric Mechanism on 0.8B-4B Qwen3.5 Models
May 12, 2026Igor StrozziSupervised Fine-Tuning LLM Evaluation
Applied Mathematics Department Federal University of Rio de Janeiro
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemLLM Evaluation Multimodal Large Language Models
School of Computing, Macquarie University, Sydney, Australia
PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head
May 12, 2026Chieh-Yen Lin, Shao-Hua SunLLM Quantization LLM Evaluation
Appier AI Research · National Taiwan University
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
Large Language Models for Causal Relations Extraction in Social Media: A Validation Framework for Disaster Intelligence
May 12, 2026Ujun Jeong, Saketh Vishnubhatla, Bohan Jiang +3Relation Extraction LLM Evaluation
Arizona State University, Tempe, Arizona, USA · DEVCOM Army Research Laboratory, Adelphi, Maryland, USA
Predicting Psychological Well-Being from Spontaneous Speech using LLMs
May 11, 2026Erfan Loweimi, Sofia de la Fuente Garcia, Saturnino LuzLLM Evaluation Speech Processing
Cisco, UK · University of Edinburgh, Edinburgh, UK · Centre for Medical Informatics (CMI), Usher Institute, University of Edinburgh
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1LLM Evaluation LLM Reliability
Carnegie Mellon University
Benchmarking LLM-Based Static Analysis for Secure Smart Contract Development: Reliability, Limitations, and Potential Hybrid Solutions
May 11, 2026Stefan-Claudiu Susan, Andrei Arusoaie, Dorel LucanuLLM Evaluation Software Vulnerability Detection
Department of Computer Science Alexandru Ioan Cuza University of Ias,i Ias,i, Romania
Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs
May 11, 2026Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya +6LLM Evaluation Language Model Decoding
♠University of Southern California · ♣Capital One

May 13, 2026, cs.AIJ/K move · Enter open · S save

Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers

Samuel Schapiro, Alexi Gladstone, Jonah Black, Heng Ji

University of Illinois Urbana-Champaign

Creativity Assessment LLM Evaluation Computational Creativity

Open paper AskPDF arXiv

Measuring the creativity of large language models (LLMs) is essential for designing methods that can improve creativity and for enhancing our scientific understanding of this ability. To accomplish this, it has become common in recent years to administer tests of human creativity to LLMs. Although these tests provide a convenient and fully automated way to score "creativity," their validity as measures of machine creativity has not been established, and these tests already have limited validity as predictors of human creativity. To address this problem, we conduct the first large-scale, systematic study assessing the effectiveness of human creativity tests for predicting the creative achievement of LLMs across three target constructs: creative writing, divergent thinking, and scientific ideation. We find that the Divergent Association Task (DAT) and the Conditional DAT are the best predictors of creative writing and divergent thinking, respectively, but that test effectiveness varies significantly by construct, and no single test predicts all constructs well. Moreover, contrary to popular belief, no existing test reliably predicts scientific ideation ability. Motivated by this problem, we introduce the Divergent Remote Association Test (DRAT), a vocabulary-space test that assesses both convergent and divergent thinking in a single instrument. The DRAT is the first and only creativity test for LLMs that is a significant predictor of scientific ideation ability, demonstrating robustness across major design choices. Furthermore, the performance gain of the DRAT is not recoverable from any linear combination of the Divergent Association Task and the Remote Associates Test, indicating that assessing divergent and convergent thinking in the same test is essential to reliably predicting scientific ideation ability.

Momentum

Related topics

Latest papers 1,643