LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought
Mar 18, 2026Zichen Xie, Wenxi WangFormal Verification LLM Evaluation
University of Virginia.
GRAFITE: Generative Regression Analysis Framework for Issue Tracking and Evaluation
Mar 18, 2026Ja Young Lee, Mírian Silva, Mohamed Nasr +6LLM Evaluation Data Contamination in Language Models
IBM Research - AI
Toward Measuring Structural Drift in LLM Communication Loops
Mar 18, 2026Wael Hafez, Amir Nazeri, Chenan WeiLLM Evaluation
Semarx Research LLC Alexandria, VA, USA
Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark
Mar 18, 2026Yao Wang, Xin Liu, Zhuochen Liu +5LLM Evaluation
Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Japan · Intelligent Platforms Research Institute, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan · Digital Science Center, University of Innsbruck, Innsbruck, Austria +2
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Mar 18, 2026Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6Automated Peer Review Scholarly Peer Review
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
Mar 17, 2026Yasi Zhang, Tianyu Chen, Mingyuan Zhou +3RL for Language Models Reinforcement Learning
University of California, Los Angeles · The University of Texas at Austin · Google Research
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
Mar 17, 2026Ryo Kamoi, Ameya Godbole, Binglin Zhou +5LLM Evaluation Human Behavior Simulation
Penn State University · University of Southern California · Microsoft Corporation
Evaluating Ill-Defined Tasks in Large Language Models
Mar 17, 2026Yi Zhou, Basel ShbitaLLM Evaluation Instruction Following
IBM Research, San Jose, CA
LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Mar 17, 2026Lifu Tu, Rongguang Wang, Tao Sheng +2Text-to-SQL Language Model Robustness
Oracle AI
Me, Myself, and $π$ : Evaluating and Explaining LLM Introspection
Mar 17, 2026Atharv Naphade, Samarth Bhargav, Sean Lim +1Language Model Introspection Self-Reflection in Language Model Agents
Carnegie Mellon University
Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models
Mar 17, 2026Mohamed Adel, Bashar Alhafni, Nizar HabashLLM Evaluation Arabic NLP
Computational Approaches to Modeling Language Lab 1New York University Abu Dhabi · 2Mohamed bin Zayed University of Artificial Intelligence
LLMs learn scientific taste from institutional traces across the social sciences
Mar 17, 2026Ziqin Gong, Ning Li, Huaikang ZhouLLM Evaluation
School of Economics and Management, Tsinghua University, Beijing, China
Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
Mar 17, 2026Xiaojie Gu, Sherry T. Tong, Aosong Feng +8LLM Evaluation Multi-Hop Reasoning
The University of Tokyo · Yale University · Stanford University +3
BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs
Mar 17, 2026Sangyeon Yoon, Hyesoo Hong, Wonje Jeung +5Personalized Language Models LLM Evaluation
Yonsei University · LG AI Research
How often do Answers Change? Estimating Recency Requirements in Question Answering
Mar 17, 2026Bhawna Piryani, Zehra Mert, Adam JatowtTemporal QA Temporal Reasoning in Language Models
University of Innsbruck Innsbruck, Austria · MEF University Istanbul, Turkey
Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic
Mar 17, 2026Finnur Ágúst Ingimundarson, Steinunn Rut Friðriksdóttir, Bjarki Ármannsson +2LLM Evaluation Benchmark Validity
University of Zürich · University of Iceland · The Árni Magnússon Institute for Icelandic Studies
More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verification
Mar 17, 2026Song Tae-EunAutomated Peer Review LLM Evaluation
Daejeon Jungang Cheonggua Co., Ltd.
Are Large Language Models Truly Smarter Than Humans?
Mar 17, 2026Eshwar Reddy M, Sourav KarmakarData Contamination in Language Models LLM Evaluation
Applied AI Scientist, Health Vectors · Senior AI Scientist, Intuit India
SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era
Mar 17, 2026Han Jang, Junhyeok Lee, Kyu Sung ChoiLLM Evaluation Scientific Communication
Seoul National University · Seoul National University Hospital · Seoul National University College of Medicine
Assessing the Pedagogical Readiness of Large Language Models as AI Tutors in Low-Resource Contexts: A Case Study of Nepal's K-10 Curriculum
Mar 17, 2026Pratyush Acharya, Prasansha Bharati, Yokibha Chapagain +2AI in Education LLM Evaluation
VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?
Mar 16, 2026Srijan Bansal, Jiao Fangkai, Yilun Zhou +3LLM Self-Correction Language Model Error Detection
Salesforce AI Research · Nanyang Technological University · A*STAR
Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN
Mar 16, 2026Ritajit Dey, Iadh Ounis, Graham McDonald +1Knowledge Conflicts in Language Models Temporal Reasoning in Language Models
University of Glasgow, Glasgow, UK · University of Strathclyde, Glasgow, UK
When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making
Mar 16, 2026Nazia RiasatLLM Evaluation LLM Reliability
North Dakota State University
Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks
Mar 16, 2026Timo FreieslebenLLM Evaluation
Munich Center for Mathematical Philosophy (MCMP) LMU Munich
Pretraining and Benchmarking Modern Encoders for Latvian
Mar 16, 2026Arturs ZnotinsLanguage Model Pretraining Low-Resource Language Processing
Institute of Mathematics and Computer Science, University of Latvia, 29 Raina bulv., Riga, LV-1459, Latvia
Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
Mar 16, 2026Jaehyeok Lee, Xiaoyuan Yi, Jing Yao +4LLM Evaluation LLM Alignment
Sungkyunkwan University · Microsoft Research Asia · Singapore University of Technology and Design
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
Mar 16, 2026Jon-Paul CacioliLLM Evaluation Language Model Calibration
Independent Researcher, Melbourne, Australia
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangCommonsense Reasoning Language Model Reasoning Evaluation
Georgia Institute of Technology
CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
Mar 15, 2026Junhang Cheng, Fang Liu, Jia Li +3Code Generation LLM Code Generation
Beihang University, Beijing, China · Wuhan University, Wuhan, China
An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs
Mar 15, 2026Qian Zhu, Xinnan Guo, Jingjing Huo +5Domain Adaptation for LLMs LLM Alignment
Ant Group

Mar 18, 2026J/K move · Enter open · S save

Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought

Zichen Xie, Wenxi Wang

University of Virginia.

Formal Verification LLM Evaluation CoT Reasoning

Open paper AskPDF arXiv

As Large Language Models (LLMs) increasingly assist secure software development, their ability to meet the rigorous demands of Rust program verification remains unclear. Existing evaluations treat Rust verification as a black box, assessing models only by binary pass or fail outcomes for proof hints. This obscures whether models truly understand the logical deductions required for verifying nontrivial Rust code. To bridge this gap, we introduce VCoT-Lift, a framework that lifts low-level solver reasoning into high-level, human-readable verification steps. By exposing solver-level reasoning as an explicit Verification Chain-of-Thought, VCoT-Lift provides a concrete ground truth for fine-grained evaluation. Leveraging VCoT-Lift, we introduce VCoT-Bench, a comprehensive benchmark of 1,988 VCoT completion tasks for rigorously evaluating LLMs' understanding of the entire verification process. VCoT-Bench measures performance along three orthogonal dimensions: robustness to varying degrees of missing proofs, competence across different proof types, and sensitivity to the proof locations. Evaluation of ten state-of-the-art models reveals severe fragility, indicating that current LLMs fall well short of the reasoning capabilities exhibited by automated theorem provers.