Large Language Model Evaluation

Follow this topic Ask about this topic

Recent momentum

-66%

14 papers in the last 28 days · 0.4% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Large Language Model Evaluation.

Period ending 2026-09-07

18 new papers

A weekly snapshot of new work published in Large Language Model Evaluation.

236 papers

Latest in Large Language Model Evaluation

Open your feed →

A primer on evaluation methods for large language models in healthcare
Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7Large Language Model Evaluation
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Sep 12, 2026Koutian Wu, Junjie Zhou, Ergan Shang +4Artificial Intelligence Benchmarks Large Language Model Evaluation
Positional task conditioning for scalable defect detection across product families in large product catalogs
Sep 9, 2026Soham Satyadharma, Gabriel Roccabruna, Suleiman A. KhanDefect Detection Products
Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach
Sep 7, 2026Janek Groß, Elena Zentgraf, Jens HeidrichLarge Language Model Evaluation Llm-Driven
FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models
Sep 3, 2026Yalun Wu, Junfeng Fang, Jiawei Wang +6Large Language Model Evaluation Evaluation Protocols
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinContent Moderation Large Language Model Evaluation
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2Large Language Model Safety Large Language Model Evaluation
SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
Sep 2, 2026Michael J. BommaritoLarge Language Model Evaluation Term Frequency-Inverse Document Frequency
A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
Sep 2, 2026Yikai Zhao, Saurabh Pandey, Pradeep Kumar MisraClarification Large Language Model Evaluation
CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
Sep 1, 2026Siyuan Li, Xinxin Song, Chen Ruinian +5Rubric-Based Reinforcement Learning Rubrics
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Aug 31, 2026Adrians Skapars, Edoardo ManinoLarge Language Model Safety Algorithm Auditing
Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators
Aug 31, 2026Xinyue Zhao, Ruiyi Zhang, Liqin Ye +3Large Language Model Evaluation Economies
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4Rubrics Large Language Model Evaluation
A Survey on Rubric-Guided Reinforcement Learning for Language Models
Aug 27, 2026Zifei Shan, Fangning ShaoRubric-Based Reinforcement Learning Rubrics
Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory
Aug 13, 2026Junkai Zhou, Shiting Guan, Zhaoyi ZhangArtificial Intelligence Companions Large Language Model Evaluation
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLarge Language Model Evaluation Reasoning Benchmark
From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation
Aug 11, 2026Alireza S. Ziabari, Kat Ellis, Colleen Chan +1Recommendation Quality Large Language Model Evaluation
EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection
Aug 11, 2026Hongrui Bao, Hangyu Rong, Zhuoshang Wang +2Machine-Generated Text Detection Large Language Model Evaluation
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLarge Language Model Bias Large Language Model Evaluation
Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes
Aug 10, 2026Alec Harris, Kasey Corra, Archie Chaudhury +1Large Language Model Evaluation Natural Language Feedback
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Large Language Model Evaluation Luxembourgish
Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
Aug 10, 2026Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde +2Large Language Model Safety Large Language Model Evaluation
PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling
Aug 9, 2026Dongjie Xu, Julius, Hanchi Dong +6Large Language Model Evaluation Diagnostic Benchmark
Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
Aug 7, 2026Afreen Alam, Evgenija Popchanovska, Ana Gjorgjevikj +4Artificial Intelligence Risk Large Language Model Safety
Methodologies for Improving the Quality of AI Tutoring in K-12 Education
Aug 7, 2026Tushar Udeshi, Anna Khazenzon, Kabir Khan +5Large Language Model Evaluation Methodology
Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models
Aug 7, 2026Rens Anderson, Tessa Verhoef, Amirhossein ZohrehvandCreativity Large Language Model Evaluation
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLarge Language Model Bias Large Language Model Evaluation
Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation
Aug 5, 2026Bo Xue, Zhi Hong, Jiayi Li +3Cost-Aware Large Language Model Evaluation
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3Text-To-Sql Large Language Model Benchmarks
Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design
Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4Llm-Driven Large Language Model Evaluation
Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform
Aug 4, 2026Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov +3Workflow Automation and Optimization Enterprise Systems
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Aug 4, 2026Bhavin Jawade, Cameron R. WolfeMachine Translation Quality Large Language Model Evaluation
MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
Aug 3, 2026Saman Sarker Joy, Niloy FarhanLarge Language Model Evaluation Sycophancy
MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
Aug 3, 2026Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1Large Language Model Evaluation Multi-Dimensional Evaluation
EduZone: A Framework for Evaluating LLM Safety for K-12 Students and Teachers
Aug 3, 2026Junyeong Park, Jieun Han, Haneul Yoo +3Large Language Model Safety Large Language Model Evaluation
Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks
Aug 2, 2026Khondoker Ittehadul IslamLarge Language Model Evaluation Qwen3
ModelEquivBench: Certifying Multi-Relational Evaluation of LLM-Generated Optimization Models
Jul 31, 2026Penglin Zhu, Jungang XuLarge Language Model Evaluation Equivalence
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Jul 31, 2026Mengting Chen, Yanshu Sun, Wanting Liang +5Rubrics Large Language Model Evaluation
Benchmarks Are Not Validation: A System-Level View of Financial LLM Applications
Jul 30, 2026Burak Payzun, İrem Demirtaş, Simona Scala +2Large Language Model Evaluation Generative Artificial Intelligence
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Jul 30, 2026Bertil Braun, Martin ForellLarge Language Model Evaluation Pragmatic Competence
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
Jul 30, 2026Weining ZhangJudges Large Language Model Evaluation
AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation
Jul 28, 2026Jia Liu, Veena Krishnaraj, Kateryna Vovk +18Artificial Intelligence Reviewers Artificial Intelligence Scientists
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
Jul 24, 2026Junda Zhao, Shurui Zhou, Eldan CohenTest Generation Large Language Model Evaluation
Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Jul 24, 2026Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao +1Creativity Large Language Model Evaluation
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management
Jul 24, 2026Yi Zhang, Hongyang Wang, Zheng Hao Leong +11Large Language Model Evaluation Waste
Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
Jul 24, 2026Yunting Song, Matthew Watson, Peter Grabowski +1Large Language Model Evaluation Large Language Model Alignment
Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable
Jul 23, 2026Prerit AhujaFinance Benchmarks Large Language Model Reliability
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Jul 23, 2026Hiroki TambaLarge Language Model Evaluation Bias Detection and Mitigation
Auditing Evidence Use in Medical LLM Diagnosis
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenDiagnostic Benchmark Incomplete Evidence
Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups
Jul 22, 2026Haran Shani-Narkiss, Michael Fire, Oren TsurLarge Language Model Bias Framing
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Jul 22, 2026Beining Wang, Weihang Su, Hongtao Tian +8Rubrics Large Language Model Evaluation
Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network
Jul 20, 2026Pilar Sánchez-Gijón, Susana Valdez, Sofía Calvo Del Barrio +3Multilingual Benchmark Multilingual
Human Grounded Evaluation of Large Language Models for Optical Network Automation
Jul 20, 2026Kiarash Rezaei, Omran Ayoub, Paolo Monti +1Large Language Model Evaluation Software-Defined Networking
Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila
Jul 20, 2026Supryadi, Irfan, Julianti +4Human Values Large Language Model Evaluation
JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
Jul 20, 2026Qingjia Huang, Jingyu Zhang, Jianguo Wu +6Large Language Model Jailbreaks Jailbreaks
Efficient Sequential Evaluation of Large Language Models
Jul 19, 2026Chia-Yu Hsu, Shubhanshu ShekharLarge Language Model Evaluation Confidence Calibration
A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
Jul 19, 2026Mohtashim KhanLarge Language Model Evaluation Preference Pairs
CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
Jul 17, 2026Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru +3Large Language Model Evaluation Rubrics
AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets
Jul 17, 2026Ming Chen, Pranav PaiGeospatial Large Language Model Evaluation
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Jul 16, 2026Yijiang Li, Huiqi Zou, Bingyang Wang +1Large Language Model Evaluation Large Language Model Hallucination

Sep 13, 2026 · cs.CLJ/K move · Enter open · S save

A primer on evaluation methods for large language models in healthcare

Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries+6

Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical context evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage. Clinical context evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.