LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

EVE: A Domain-Specific LLM Framework for Earth Intelligence
Mar 20, 2026Àlex R. Atrio, Antonio Lopez, Jino Rohit +6Earth Science LLM Evaluation
Pi School · Mistral AI · Translated +1
SEAR: Schema-Based Evaluation and Routing for LLM Gateways
Mar 20, 2026Zecheng Zhang, Han Zheng, Yue XuLLM Routing LLM Evaluation
Strukto.AI, USA · Infron.AI, USA
TextReasoningBench: Does Reasoning Really Improve Text Classification in Large Language Models?
Mar 20, 2026Xinyu Guo, Yazhou Zhang, Jing QinLarge Language Model-Based Text Classification LLM Evaluation
School of Computer Science and Technology, Tianjin University, China · School of Nursing, The Hong Kong Polytechnic University, HongKong
ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
Mar 19, 2026Tianlong Wang, Pinqiao Wang, Weili Shi +1LLM Evaluation LLM Planning
University of Virginia
Correct Chains, Wrong Answers: Dissociating Reasoning from Output in LLM Logic
Mar 19, 2026Abinav Rao, Sujan Rachuri, Nikhil VemuriLLM Reasoning LLM Evaluation
FinTradeBench: A Financial Reasoning Benchmark for LLMs
Mar 19, 2026Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan +2Financial Reasoning in Language Models Financial Reasoning Benchmarks
University of Central Florida
How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Mar 19, 2026Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13Audio Understanding Audio-Language Model Evaluation
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan
Evaluating Counterfactual Strategic Reasoning in Large Language Models
Mar 19, 2026Dimitrios Georgousis, Maria Lymperaiou, Angeliki Dimitriou +2Strategic Reasoning Risks in Language Models LLM Evaluation
National Technical University of Athens
Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans
Mar 19, 2026Christian Di Maio, Tommaso Guidi, Luigi Quarantiello +4LLM Evaluation Human-AI Interaction
DIISM, University of Siena, Italy · Department of Computer Science, University of Pisa, Italy · LUISS University, Rome, Italy
ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs
Mar 19, 2026Wanjia Zhao, Ludwig Schmidt, Yejin Choi +3LLM Evaluation AI Agent Benchmarks
Stanford University · Microsoft Research
Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions
Mar 19, 2026Xiaoyi LiLanguage Model Post-Training LLM Alignment
GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Norms
Mar 19, 2026Masayuki Kawarada, Kodai Watanabe, Soichiro MurakamiLLM Decision-Making LLM Evaluation
CyberAgent
Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought
Mar 18, 2026Zichen Xie, Wenxi WangFormal Verification LLM Evaluation
University of Virginia.
GRAFITE: Generative Regression Analysis Framework for Issue Tracking and Evaluation
Mar 18, 2026Ja Young Lee, Mírian Silva, Mohamed Nasr +6LLM Evaluation Data Contamination in Language Models
IBM Research - AI
Toward Measuring Structural Drift in LLM Communication Loops
Mar 18, 2026Wael Hafez, Amir Nazeri, Chenan WeiLLM Evaluation
Semarx Research LLC Alexandria, VA, USA
Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark
Mar 18, 2026Yao Wang, Xin Liu, Zhuochen Liu +5LLM Evaluation
Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Japan · Intelligent Platforms Research Institute, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan · Digital Science Center, University of Innsbruck, Innsbruck, Austria +2
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Mar 18, 2026Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6Automated Peer Review Scholarly Peer Review
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
Mar 17, 2026Yasi Zhang, Tianyu Chen, Mingyuan Zhou +3RL for Language Models Reinforcement Learning
University of California, Los Angeles · The University of Texas at Austin · Google Research
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
Mar 17, 2026Ryo Kamoi, Ameya Godbole, Binglin Zhou +5LLM Evaluation Human Behavior Simulation
Penn State University · University of Southern California · Microsoft Corporation
Evaluating Ill-Defined Tasks in Large Language Models
Mar 17, 2026Yi Zhou, Basel ShbitaLLM Evaluation Instruction Following
IBM Research, San Jose, CA
LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Mar 17, 2026Lifu Tu, Rongguang Wang, Tao Sheng +2Text-to-SQL Language Model Robustness
Oracle AI
Me, Myself, and $π$ : Evaluating and Explaining LLM Introspection
Mar 17, 2026Atharv Naphade, Samarth Bhargav, Sean Lim +1Language Model Introspection Self-Reflection in Language Model Agents
Carnegie Mellon University
Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models
Mar 17, 2026Mohamed Adel, Bashar Alhafni, Nizar HabashLLM Evaluation Arabic NLP
Computational Approaches to Modeling Language Lab 1New York University Abu Dhabi · 2Mohamed bin Zayed University of Artificial Intelligence
LLMs learn scientific taste from institutional traces across the social sciences
Mar 17, 2026Ziqin Gong, Ning Li, Huaikang ZhouLLM Evaluation
School of Economics and Management, Tsinghua University, Beijing, China
Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
Mar 17, 2026Xiaojie Gu, Sherry T. Tong, Aosong Feng +8LLM Evaluation Multi-Hop Reasoning
The University of Tokyo · Yale University · Stanford University +3
BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs
Mar 17, 2026Sangyeon Yoon, Hyesoo Hong, Wonje Jeung +5Personalized Language Models LLM Evaluation
Yonsei University · LG AI Research
How often do Answers Change? Estimating Recency Requirements in Question Answering
Mar 17, 2026Bhawna Piryani, Zehra Mert, Adam JatowtTemporal QA Temporal Reasoning in Language Models
University of Innsbruck Innsbruck, Austria · MEF University Istanbul, Turkey
Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic
Mar 17, 2026Finnur Ágúst Ingimundarson, Steinunn Rut Friðriksdóttir, Bjarki Ármannsson +2LLM Evaluation Benchmark Validity
University of Zürich · University of Iceland · The Árni Magnússon Institute for Icelandic Studies
More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verification
Mar 17, 2026Song Tae-EunAutomated Peer Review LLM Evaluation
Daejeon Jungang Cheonggua Co., Ltd.
Are Large Language Models Truly Smarter Than Humans?
Mar 17, 2026Eshwar Reddy M, Sourav KarmakarData Contamination in Language Models LLM Evaluation
Applied AI Scientist, Health Vectors · Senior AI Scientist, Intuit India

Mar 20, 2026J/K move · Enter open · S save

EVE: A Domain-Specific LLM Framework for Earth Intelligence

Àlex R. Atrio, Antonio Lopez, Jino Rohit, Yassine El Ouahidi+5

Pi School · Mistral AI · Translated+1

Earth Science LLM Evaluation

Open paper AskPDF arXiv

We introduce Earth Virtual Expert (EVE), the first open-source, end-to-end initiative for developing and deploying domain-specialized LLMs for Earth Intelligence. At its core is EVE-Instruct, a domain-adapted 24B model built on Mistral Small 3.2 and optimized for reasoning and question answering. On newly constructed Earth Observation and Earth Sciences benchmarks, it outperforms comparable models while preserving general capabilities. We release curated training corpora and the first systematic domain-specific evaluation benchmarks, covering MCQA, open-ended QA, and factuality. EVE further integrates RAG and a hallucination-detection pipeline into a production system deployed via API and GUI, supporting 350 pilot users so far. All models, datasets, and code are ready to be released under open licenses as contributions to our field at huggingface.co/eve-esa and github.com/eve-esa.