LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants
Mar 10, 2026Zuhao Zhang, Chengyue Yu, Yuante Li +3Web Application Generation LLM Evaluation
Inclusion AI, Ant Group, Hangzhou, Zhejiang, China · Shanghai Jiao Tong University, Shanghai, China · Carnegie Mellon University, Pittsburgh, PA, USA
CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Mar 10, 2026Xiangsen Chen, Xuan Feng, Shuo Chen +5LLM Evaluation Cyber Threat Intelligence
Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1
From Scalars to Tensors: Declared Losses Recover Epistemic Distinctions That Neutrosophic Scalars Cannot Express
Mar 10, 2026Tony MasonEpistemic Uncertainty LLM Evaluation
University of British Columbia, Vancouver, Canada · Georgia Institute of Technology, Atlanta, USA
Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking
Mar 9, 2026Eyhab Al-MasriLLM Evaluation LLM Agent Reliability
School of Engineering and Technology, University of Washington (Tacoma)
Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines
Mar 9, 2026Akshay Gulati, Kanha Singhania, Tushar Banga +8Financial Reasoning in Language Models Financial Reasoning Benchmarks
The Future University
How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
Mar 9, 2026Songhee Han, Jueun Shin, Jiyoon Han +2LLM-as-a-Judge LLM Evaluation
Florida State University, Tallahassee, FL 32306, USA
How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms
Mar 9, 2026JV RoigHallucination in Language Models LLM Evaluation
Kamiwaza AI
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Mar 9, 2026Jianfei Xiao, Xiang Yu, Chengbing Wang +8LLM Personalization Personalized Language Model Alignment
University of Science and Technology of China · National University of Singapore
CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases
Mar 9, 2026Xiaona Xue, Yiqiao Huang, Jiacheng Li +9LLM Evaluation Instruction Following
Jiutian Artificial Intelligence Research Institute, China Mobile, Beijing, China
Benchmark for Assessing Olfactory Perception of Large Language Models
Mar 8, 2026Eftychia Makri, Nikolaos Nakis, Laura Sisson +4LLM Evaluation
Department of Electrical Engineering, Yale University, New Haven, CT, USA · Human Nature Lab, Yale University, New Haven, CT, USA · Patina, New York, NY, USA +1
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
Mar 8, 2026David Beauchemin, Richard KhouryLLM Evaluation LLM Reliability
Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Quebec City, Quebec, Canada
Writing literature reviews with AI: principles, hurdles and some lessons learned
Mar 8, 2026Saadi Lahlou, Annabelle Gouttebroze, Atrina Oraee +1LLM Sycophancy LLM Evaluation
London School of Economics and Political Science · Paris Institute for Advanced Study
Large Language Model for Discrete Optimization Problems: Evaluation and Step-by-step Reasoning
Mar 8, 2026Tianhao Qian, Guilin Qi, Z. Y. Wu +3LLM Evaluation Large Language Model-Guided Optimization
School of Mathematics, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Mathematical Sciences, Chongqing Normal University, Chongqing, China
Evaluating LLMs as Human Surrogates in Controlled Experiments
Mar 8, 2026Adnan Hoq, Tim WeningerHuman Behavior Simulation Large Language Model-Based Social Simulation
University of Notre Dame Notre Dame, Indiana, USA
MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs
Mar 8, 2026Abdessalam Bouchekif, Shahd Gaben, Samer Rashwani +5Legal Reasoning LLM Evaluation
Hamad Bin Khalifa University, Qatar · Nazarbayev University, Kazakhstan
Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
Mar 8, 2026Jiyeon Kim, Hyunji Lee, Dylan Zhou +6LLM Evaluation Continual Learning for LLMs
KAIST AI · UNC Chapel Hill · Google +3
Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction
Mar 7, 2026Özgür Togay, Javier Garcia-Bernardo, Florian Kunneman +1Stance Detection Political Discourse Analysis
Department of Methodology and Statistics, Utrecht University · Department of Languages, Literature and Communication, Utrecht University
FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets
Mar 7, 2026Jan Ravnik, Matjaž Ličen, Felix Bührmann +3Table QA LLM Evaluation
Qubera AG, Gartenstrasse 33, 8002 Zürich · University of Zurich, Department of Informatics, Binzmühlestrasse 14, 8050 Zürich
CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs
Mar 7, 2026Siyi Li, Jiajun Shi, Shiwen Ni +9LLM Evaluation Efficient Language Model Reasoning
University of Science and Technology of China · Shenzhen Institutes of Advanced Technology, CAS · ByteDance SEED +6
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Mar 6, 2026Warren JohnsonPrompt Compression LLM Inference Efficiency
Plexor Labs Principal Researcher
Qworld: Question-Specific Evaluation Criteria for LLMs
Mar 6, 2026Shanghua Gao, Yuchang Su, Pengwei Sui +2LLM Evaluation
Department of Biomedical Informatics, Harvard Medical School · Department of Medicine, Brigham and Women’s Hospital · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University +2
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan
CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation
Mar 6, 2026Mohammed Baharoon, Thibault Heintz, Siavash Raissi +6Radiology Report Evaluation LLM Evaluation
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Radiation Oncology, Mass General Brigham, Boston, MA · King Fahad Hospital, Al-Ahsa Health Cluster, Al Hofuf, Saudi Arabia +2
Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring
Mar 6, 2026Jonas Kubesch, Lena Huber, Clemens HavasAutomated Essay Scoring Educational Assessment
Salzburg University of Applied Sciences, Salzburg, Austria
Lost in Stories: Consistency Bugs in Long Story Generation by LLMs
Mar 6, 2026Junjie Li, Xinrui Guo, Yuhao Wu +3Narrative Generation Reasoning Consistency in Language Models
Microsoft, Beijing, China · Singapore University of Technology and Design
Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics
Mar 6, 2026Srishti Palani, Vidya SetlurLLM Evaluation Visual Analytics
Tableau Research, Salesforce, Palo Alto, CA, USA
Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues
Mar 6, 2026Eric Gong, Nathan E. Sanders, Bruce SchneierLarge Language Model-Based Social Simulation LLM Evaluation
Harvard College · Berkman-Klein Center, Harvard University · Harvard Kennedy School
DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models
Mar 5, 2026Alexander SheppertLLM Evaluation Factual Knowledge in Language Models
Legacy Health · Capitol Technology University
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
Can LLMs Capture Expert Uncertainty? A Comparative Analysis of Value Alignment in Ethnographic Qualitative Research
Mar 5, 2026Arina Kostina, Marios Dikaiakos, Alejandro Porcel +1LLM Evaluation LLM Uncertainty Estimation
University of Cyprus, Cyprus · Trinetra Investment Management LLP, UK · University of Cambridge, UK

Mar 10, 2026J/K move · Enter open · S save

MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants

Zuhao Zhang, Chengyue Yu, Yuante Li, Chenyi Zhuang+2

Inclusion AI, Ant Group, Hangzhou, Zhejiang, China · Shanghai Jiao Tong University, Shanghai, China · Carnegie Mellon University, Pittsburgh, PA, USA

Web Application Generation LLM Evaluation Automated Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

With the rapid advancement of Large Language Models (LLMs) in code generation, human-AI interaction is evolving from static text responses to dynamic, interactive HTML-based applications, which we term MiniApps. These applications require models to not only render visual interfaces but also construct customized interaction logic that adheres to real-world principles. However, existing benchmarks primarily focus on algorithmic correctness or static layout reconstruction, failing to capture the capabilities required for this new paradigm. To address this gap, we introduce MiniAppBench, the first comprehensive benchmark designed to evaluate principle-driven, interactive application generation. Sourced from a real-world application with 10M+ generations, MiniAppBench distills 500 tasks across six domains (e.g., Games, Science, and Tools). Furthermore, to tackle the challenge of evaluating open-ended interactions where no single ground truth exists, we propose MiniAppEval, an agentic evaluation framework. Leveraging browser automation, it performs human-like exploratory testing to systematically assess applications across three dimensions: Intention, Static, and Dynamic. Our experiments reveal that current LLMs still face significant challenges in generating high-quality MiniApps, while MiniAppEval demonstrates high alignment with human judgment, establishing a reliable standard for future research. Our homepage is available in miniappbench.github.io.