LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

The Path Not Taken: Duality in Reasoning about Program Execution
Apr 22, 2026Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker +1LLM Evaluation Code Language Models
Department of Computer Science, University of Central Florida
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
Apr 21, 2026Zeyu Tang, Sang T. Truong, Deonna Owens +4LLM Evaluation Social Bias in Language Models
Stanford University
The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text
Apr 21, 2026Andrew Hong, Jason Potteiger, Luis E. ZapataLLM Evaluation LLM Prompting
Dimension Labs
Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models
Apr 21, 2026Kihyuk LeeLLM Evaluation Clinical Language Model Evaluation
Data Convergence Team, Office of Hospital Information, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews
Apr 21, 2026Bowen Li, Haochen Ma, Yuxin Wang +5LLM Evaluation Rubric-Based Evaluation
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas
Apr 21, 2026Jon-Paul CacioliLLM Evaluation
Independent Researcher · Melbourne, Australia
Are Large Language Models Economically Viable for Industry Deployment?
Apr 21, 2026Abdullah Mohammad, Sushant Kumar Ray, Pushkar Arora +5LLM Inference Efficiency LLM Evaluation
DSEU-Okhla, New Delhi, India · University of Delhi, New Delhi, India · 6, 8Macquarie University, Sydney, Australia +1
Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation
Apr 21, 2026Eoghan Cunningham, Derek Greene, James Cross +1LLM Evaluation Factual Consistency Evaluation
School of Computer Science, University College Dublin · School of Politics and International Relations, University College Dublin · Department of Informatics, King’s College London
Scoring-Rule Sensitivity in LLM Evaluation: Evidence from Indian Financial Regulatory Text
Apr 21, 2026Rajveer Singh PallLLM Evaluation Financial QA
Gyan Ganga Institute of Technology and Sciences, Jabalpur, India
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Apr 21, 2026Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22Multilingual Language Model Evaluation LLM Evaluation
Alibaba Group · Beijing Language and Culture University · LMU Munich +6
Verbal tics in frontier language models: A critical review of current releases, research evidence, and public discussion
Apr 21, 2026Shuai Wu, Xue Li, Zhijun Wang +4LLM Evaluation LLM Sycophancy
Do Emotions Influence Moral Judgment in Large Language Models?
Apr 21, 2026Mohammad Saim, Tianyu JiangLLM Evaluation Moral Reasoning in Language Models
University of Cincinnati
HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing
Apr 21, 2026Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo +7LLM Evaluation LLM-as-a-Judge
Department of Computer Science and Technology, Tsinghua University
DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning
Apr 21, 2026Ahmed G. A. H Ahmed, C. Okan SakarLLM Evaluation LLM Reasoning with Graphs
Innosol / Bahçe¸sehir University · Department of Computer Engineering, Bahcesehir University, Turkey
Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest
Apr 21, 2026Ramtin Davoudi, Kartik Thakkar, Nazanin Donyapour +2Social Media Analysis LLM Evaluation
Utah State University · Independent Researcher · Vanderbilt University
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
Apr 21, 2026Cristina Garbacea, Heran Wang, Chenhao TanLLM Evaluation User Preference Modeling
University of Chicago
LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification
Apr 20, 2026Pedro Barbosa de Carvalho NetoLegal NLP LLM Evaluation
Teresina, PI, Brazil
Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models
Apr 20, 2026Seyedali Mohammadi, Manas Gaur, Francis FerraroLLM Evaluation Scientific Reasoning in Language Models
University of Maryland, Baltimore County, MD, USA
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
Apr 20, 2026Terry LeitchLLM Quantization Dynamical Systems
Ruxton AI
FUSE: Ensembling Verifiers with Zero Labeled Data
Apr 20, 2026Joonhyuk Lee, Virginia Ma, Sarah Zhao +4Neural Network Verification LLM Evaluation
Stanford University · Google
Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols
Apr 20, 2026Fernando ReitichLLM Evaluation LLM Auditing
Imens, LLC · Weston, Florida, USA
STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs
Apr 20, 2026Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2LLM Evaluation Reasoning Evaluation
IBM Research
Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
Apr 20, 2026Ran Zhang, Steffen Eger, Arda Tezcan +3Creativity Assessment LLM Evaluation
University of Mannheim, Data and Web Science Group · University of Technology Nuremberg (UTN), Department Engineering · 5Natural Language Learning and Generation (NLLG) Lab +2
Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attacks
Apr 20, 2026Jin Zhao, Marta Knežević, Tanja KäserLLM Evaluation Intelligent Tutoring Systems
†Work done during exchange at EPFL.
BEACON: Behavioral Entropy Aggregation for Cross-Model Hallucination Detection in Large Language Models
Apr 20, 2026Naveen Bera, Pulijala Sai Nikhila, Kondaguduru Abhiram +6LLM Evaluation Hallucination Detection
LLM Lens
How Creative Are Large Language Models in Generating Molecules?
Apr 20, 2026Wen Tao, Yiwei Wang, Peng Zhou +6Creativity Assessment LLM Evaluation
Nanyang Technological University · University of California, Merced · Hunan University +1
Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
Apr 20, 2026Xinyao Zhang, Nicole Sonne Heckmann, Manuela Del Castillo Suero +2LLM Evaluation Healthcare
Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark.
ltzGLUE: Luxembourgish General Language Understanding Evaluation
Apr 20, 2026Alistair Plum, Felicia Körner, Anne-Marie Lutgen +8LLM Evaluation Text Classification
University of Luxembourg, Luxembourg · LMU Munich, Germany · Munich Center for Machine Learning, Germany +2
From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?
Apr 20, 2026Hasan Amin, Harry Yizhou Tian, Xiaoni Duan +3LLM Evaluation LLM-Assisted Annotation
Purdue University · Washington University in St. Louis
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Apr 20, 2026Taylor Lundy, Narun K. Raman, Kevin Leyton-BrownLLM Evaluation Bayesian Optimization
Department of Computer Science University of British Columbia Vancouver, BC

Apr 22, 2026J/K move · Enter open · S save

The Path Not Taken: Duality in Reasoning about Program Execution

Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally

Department of Computer Science, University of Central Florida

LLM Evaluation Code Language Models

Open paper AskPDF arXiv

Large language models (LLMs) have shown remarkable capabilities across diverse coding tasks. However, their adoption requires a true understanding of program execution rather than relying on surface-level patterns. Existing benchmarks primarily focus on predicting program properties tied to specific inputs (e.g., code coverage, program outputs). As a result, they provide a narrow view of dynamic code reasoning and are prone to data contamination. We argue that understanding program execution requires evaluating its inherent duality through two complementary reasoning tasks: (i) predicting a program's observed behavior for a given input, and (ii) inferring how the input must be mutated toward a specific behavioral objective. Both tasks jointly probe a model's causal understanding of execution flow. We instantiate this duality in DexBench, a benchmark comprising 445 paired instances, and evaluate 13 LLMs. Our results demonstrate that dual-path reasoning provides a robust and discriminative proxy for dynamic code understanding.