Large Language Model Evaluation

Ask about this topic

Momentum

28 papers in the last four weeks, up 47% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 287

All topics
CardsList

  1. Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

    Oct 7, 2026Daniel Robert Kling Alexander, Catherine Louise KlingLarge Language Model Evaluation

    University of Michigan School of Information · Tisch University Professor, SC Johnson College of Business, Cornell University

  2. LLM Persuasion Is in the Eye of the Evaluation

    Oct 7, 2026Kamile Dementaviciute, Julija Vaitonyte, Tijl De BiePersuasionLarge Language Model Evaluation

    Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics

  3. The Missing Minimal Pair: Stereotype Evaluation in LLMs

    Oct 6, 2026Nataliya Stepanova, Ivan Titov, Emily Allaway +1Large Language Model BiasLarge Language Model Evaluation

    University of Edinburgh · University of Amsterdam

  4. Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods

    Oct 6, 2026Haotian Yang, Huikang Jiang, Yucheng Wu +4Linear Activation SteeringLarge Language Model Evaluation

    State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Columbia University +2

  5. Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints

    Oct 5, 2026Shai Feldman, Yaniv RomanoToken Budget AllocationLarge Language Model Evaluation

    Department of Computer Science, Technion IIT, Israel · Department of Electrical and Computer Engineering, Technion IIT, Israel

  6. Evaluating Whether GPT-6 Astra Performs Unsanctioned Supply-Chain Attacks

    Sep 29, 2026Alexandra Souly, Kai Fronsdal, Abby D'Cruz +2OpenaiLarge Language Model Evaluation

    UK AI Security Institute · UK AISI Research Affiliate

  7. A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses

    Sep 29, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoRubricsTask-Specific Rubrics

    DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London, London, UK · Neurological Institute, Cleveland Clinic London, London, UK · King’s College London, London, UK +1

  8. Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

    Sep 29, 2026Jiayuxuan Yang, Jie M. Zhang, Yiling Lou +1RubricsLarge Language Model Evaluation

    Tsinghua University · King’s College London · University of Illinois Urbana-Champaign

  9. Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers

    Sep 29, 2026Zhuo Chen, Hao Zeng, Jiawei Liu +6Peer ReviewFuzzing

    Wuhan University · East China Normal University

  10. JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

    Sep 29, 2026Qi Cao, Kangning Liu, Xuan Kan +10Large Language Model JudgesJudgement

    Meta · University of California San Diego · The University of Hong Kong +1

  11. Reliability Testing of Medical Model Performance under Distributed Deployment

    Sep 29, 2026Yifei Wang, Xiaohan Zhang, Youtao Ding +4Model EvaluationPre-Deployment Safety Assessments

    Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore +1

  12. Training LLMs to Verbalize Evaluation Awareness

    Sep 28, 2026Usman Anwar, Sahar Abdelnabi, David KruegerVerbalizationLarge Language Model Evaluation

    University of Cambridge · ELLIS Institute Tübingen MPI-IS, Tübingen AI Center · Mila University of Montreal

  13. OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models

    Sep 28, 2026Liner Xiang, Wenbo Zhang, Hengrui CaiLarge Language Model Evaluation

    Department of Statistics, University of California, Irvine

  14. How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?

    Sep 28, 2026Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +2Large Language Model EvaluationFeedback

    Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · National Institute of Informatics +2

  15. SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

    Sep 28, 2026Jingyun Jia, Antoine Remond-Tiedrez, Aaron Alvarez +3Large Language Model EvaluationModel Discovery

    University of Wisconsin–Madison, Intelligible · Intelligible · University of Cincinnati

  16. Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards

    Sep 27, 2026Krishna Chytanya AyyagariVerdictLeaderboard

    Google

  17. How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

    Sep 24, 2026Dipankar SarkarLarge Language Model EvaluationReproducibility

    Skelf Research

  18. Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populations

    Sep 24, 2026Yeeun Chae, Yewon Choi, Seunghyun Lee +1Large Language Model EvaluationSurvey

    Yonsei University · NAVER · Seoul National University Hospital

  19. Calibration as a First-Class Criterion in LLM Evaluation

    Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLarge Language Model EvaluationOverconfidence

    Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA

  20. Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices

    Sep 22, 2026Qian Xie, Yueli He, Nairen CaoLarge Language Model EvaluationRecommendation

    Cornell University · Columbia University · New York University +1

  21. Benchmarking LLM Compliance with China AI Generated Content Regulations

    Sep 17, 2026Chenrui Cui, Hongye Fang, Lisha Song +3Large Language Model EvaluationStrong Large Language Model

    School of Computer Science and Technology, Tongji University, Shanghai, China · Law School of Tongji University, Shanghai, China

  22. Reproducibility is not construct validity: LLM measurement of institutionally situated communication

    Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaInter-Annotator AgreementLarge Language Model Evaluation

    KIT · Department of Sociology and Computational Sociology, Karlsruhe Institute of Technology (KIT), Douglasstr. 24, 76133 Karlsruhe, Germany · ALMAnaCH, médialab, Sciences Po +2

  23. Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

    Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Large Language Model EvaluationReward Signal

    Core contributors · Equal senior contribution

  24. DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions

    Sep 16, 2026Rem Hida, Masahiro Kaneko, Daisuke Oba +2Large Language Model BiasMulti-Turn Interactions

    Institute of Science Tokyo · MBZUAI · Third Intelligence +1

  25. Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models

    Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2Large Language Model EvaluationStroke

    Department of Neurosurgery, National Hospital Organization Kagoshima Medical Center, Kagoshima, Japan · Department of Neurology, Imamura General Hospital, Kagoshima, Japan · Iwamoto Neurosurgery, Kanoya, Kagoshima, Japan +1

  26. PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

    Sep 14, 2026Miguel Zabaleta, Baihan LinSystematic ReviewPeer Review

    Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

  27. Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

    Sep 14, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3Large Language Model BenchmarksLarge Language Model Evaluation

    Bosch Research · Sun Yat-sen University

  28. A primer on evaluation methods for large language models in healthcare

    Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7Large Language Model EvaluationHealthcare

    Mass General Brigham AI, Mass General Brigham, Boston, MA, USA · Generative AI Laboratory, School of Informatics, University of Edinburgh, Edinburgh, UK · University of Edinburgh Centre for Cardiovascular Science, Edinburgh, UK +5

  29. The widening evaluation gap in medical large language model research 2023 to 2026

    Sep 11, 2026Raad Bin Tareaf, Murad Al-Rajab, Samia LoucifSystematic ReviewLarge Language Model Evaluation

    XU Exponential University of Applied Sciences, Data Science and Artificial Intelligence Cluster, Marlene-Dietrich-Allee 12B, 14482 Potsdam, Germany · College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates · College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates

  30. Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

    Sep 10, 2026Koutian Wu, Junjie Zhou, Ergan Shang +6Artificial Intelligence BenchmarksLarge Language Model Benchmarks

    Earth-Space-AI · Tacite AI · Hangzhou Dianzi University +6

  31. The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

    Sep 8, 2026Siddharth Vohra, Manikandan RavikiranLarge Language Model BiasModel Auditing

    Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA · Indian Institute of Technology Mandi, India

  32. Evaluation of Contextual Understanding in Large Language Models

    Sep 8, 2026Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4Large Language Model EvaluationSimilarity and Metrics

    Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA.

  33. ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

    Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungChatbot ArenaLarge Language Model Evaluation

    Vulcan Research, AIFT Singapore

  34. FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models

    Sep 3, 2026Yalun Wu, Junfeng Fang, Jiawei Wang +6Large Language Model EvaluationEvaluation Protocols

    NExT++ Lab, School of Computing, National University of Singapore · 3Xiamen University · University of Manchester +4

  35. Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

    Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinContent ModerationLarge Language Model Evaluation

    1Independent Researcher · University of Sheffield

  36. IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

    Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2Large Language Model SafetyLarge Language Model Evaluation

    Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India

  37. SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

    Sep 2, 2026Michael J. BommaritoLarge Language Model EvaluationTerm Frequency-Inverse Document Frequency

    Portions of this work were prepared with assistance from large language models. The author is solely responsible for all content, including any errors or omissions.

  38. A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

    Sep 2, 2026Yikai Zhao, Saurabh Pandey, Pradeep Kumar MisraClarificationLarge Language Model Evaluation

    Amazon Dallas, Texas, USA · Amazon Seattle, Washington, USA

  39. CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

    Sep 1, 2026Siyuan Li, Xinxin Song, Chen Ruinian +5Large Language Model Reinforcement LearningLarge Language Model Evaluation

    Department of Automation, Tsinghua University · Meituan

  40. BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing

    Aug 31, 2026Adrians Skapars, Edoardo ManinoLarge Language Model EvaluationModel Auditing

    University of Manchester, United Kingdom

  41. Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators

    Aug 31, 2026Xinyue Zhao, Ruiyi Zhang, Liqin Ye +3Large Language Model ForecastingLarge Language Model Evaluation

    Georgia Institute of Technology · University of California San Diego

  42. GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

    Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4RubricsLarge Language Model Evaluation

    Department of Computer Science and Technology, Tsinghua University · National University of Singapore

  43. Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance

    Aug 21, 2026Alexander Thomas, Hubert P. H. Shum, Darren Nellis +4Large Language Model EvaluationMaritime Navigation

    NCB Hazcheck Limited, UK · Durham University, UK

  44. Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory

    Aug 13, 2026Junkai Zhou, Shiting Guan, Zhaoyi ZhangEmotional Support Conversation TaskLarge Language Model Evaluation

  45. Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

    Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLarge Language Model EvaluationReasoning Benchmark

    Federal University of Santa Catarina (UFSC) Brazil

  46. From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation

    Aug 11, 2026Alireza S. Ziabari, Kat Ellis, Colleen Chan +1Sequential RecommendationLarge Language Model Evaluation

    Netflix Los Angeles, USA · Netflix Los Gatos, USA

  47. EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection

    Aug 11, 2026Hongrui Bao, Hangyu Rong, Zhuoshang Wang +2Machine-Generated Text DetectionLarge Language Model Evaluation

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

  48. Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

    Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLarge Language Model BiasLarge Language Model Evaluation

    McGill University · University Canada West · Marketing Management +1

  49. Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

    Aug 11, 2026Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie +1Large Language Model EvaluationModel Judgments

    The University of Queensland · Brisbane, Australia

  50. Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

    Aug 10, 2026Alec Harris, Kasey Corra, Archie Chaudhury +1Large Language Model EvaluationLarge Language Model Alignment

    AI Safety Initiative at Georgia Tech · University of Chicago · Independent

  51. From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

    Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Large Language Model EvaluationLuxembourgish

    City of Amsterdam · Socially Intelligent Artificial Systems Group, University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

  52. Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness

    Aug 10, 2026Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde +2Large Language Model SafetyLarge Language Model Evaluation

    University of the Ryukyus · Technion · Infornation Processing and Telecommunications Center (IPTC) Universidad Polécnica de Madrid

  53. PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

    Aug 9, 2026Dongjie Xu, Julius, Hanchi Dong +6Large Language Model EvaluationDiagnostic Benchmark

    Soochow University · Tencent Inc. · Beijing University of Posts and Telecommunications +4

  54. Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations

    Aug 8, 2026Michael Levit, Josh Ledgard, Haoyu Dong +5Synthetic DataLarge Language Model Evaluation

    Microsoft Corporation

  55. MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

    Aug 8, 2026Yilian Shi, Yueming Lyu, Haoxiang Tan +5Large Language Model SafetyMultimodal Large Language Models

    Nanjing University

  56. Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools

    Aug 7, 2026Afreen Alam, Evgenija Popchanovska, Ana Gjorgjevikj +4Artificial Intelligence RiskLarge Language Model Evaluation

    Department of Administrative Sciences, Metropolitan College, Boston University, Boston, MA 02215, USA · Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University, Skopje 1000, North Macedonia · Department of Computer Science, Metropolitan College, Boston University, Boston, MA 02215, USA

  57. Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models

    Aug 7, 2026Rens Anderson, Tessa Verhoef, Amirhossein ZohrehvandCreativityLarge Language Model Evaluation

    LIACS Leiden University Einsteinweg 55 2333 CC Leiden The Netherlands

  58. Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation

    Aug 5, 2026Bo Xue, Zhi Hong, Jiayi Li +3Cost-AwareLarge Language Model Evaluation

    City University of Hong Kong · South China University of Technology · Zhejiang University

  59. Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

    Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3Text-To-SqlLarge Language Model Benchmarks

    1East China Normal University · 2Hasso Plattner Institute/University of Potsdam

  60. Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

    Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4Large Language Model WorkflowsLarge Language Model Evaluation

    Westlake University · Zhejiang University · Nanjing University of Science and Technology +1