LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications
Mar 5, 2026Yunfan Zhang, Yijie Bei, Jetashree Ravi +1LLM Evaluation Instruction Following
Columbia University · Fireworks AI
Spatial Competence Benchmark
Mar 5, 2026Jash Vira, Ashley HarrisSpatial Reasoning Benchmarks Spatial Reasoning
Independent Researcher · Maptek
Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
Mar 4, 2026Qianyun Guo, Yibo Li, Yue Liu +1LLM Evaluation LLM Personalization
National University of Singapore, Singapore.
Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model
Mar 4, 2026Jakub PrejznerQuantization LLM Quantization
BitSharp, Independent Researcher, Rzeszów, Poland
Traces of Social Competence in Large Language Models
Mar 4, 2026Tom Kouwenhoven, Michiel van der Meer, Max van DuijnTheory of Mind LLM Evaluation
Leiden Institute of Advanced Computer Science Leiden University, Netherlands
FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation
Mar 4, 2026Juhyun Oh, Nayeon Lee, Chani Jung +5Language Model Generation Evaluation Language Model Safety Evaluation
School of Computing, KAIST, Daejeon, Republic of Korea · Samsung, Suwon, Republic of Korea
A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality
Mar 4, 2026Arther Tian, Alex Ding, Frank Chen +2Language Model Generation Evaluation LLM Evaluation
DGrid AI
Evaluating Large Language Models for Gait Classification Using Text-Encoded Kinematic Waveforms
Mar 4, 2026Carlo Dindorf, Jonas Dully, Rebecca Keilhauer +2LLM Evaluation Gait Analysis
Department of Sports Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany · Fraunhofer Institute for Experimental Software Engineering IESE, Kaiserslautern, Germany · German Research Center of Artificial Intelligence, Kaiserslautern, Germany
CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents
Mar 4, 2026Martin Kostelník, Michal Hradiš, Martin DočekalDocument Understanding Historical Document Analysis
Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic
In-Context Environments Induce Evaluation-Awareness in Language Models
Mar 4, 2026Maheep ChaudharyEvaluation Awareness in Language Models LLM Evaluation
Independent
T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoning
Mar 4, 2026Qinsi Wang, Hancheng Ye, Jinhee Kim +12Structured Reasoning LLM Reasoning
Duke University · UT Austin · Meta
Tucano 2 Cool: Better Open Source LLMs for Portuguese
Mar 3, 2026Nicholas Kluge Corrêa, Aniket Sen, Shiza Fatimah +4Portuguese Language Models Language Modeling
Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Center for Science and Thought +2
Verify to Amplify: Improving Reasoning via Learned Chain-of-Thought Verification
Mar 3, 2026Maria-Florina Balcan, Avrim Blum, Kiriaki Fragkia +2LLM Evaluation LLM Answer Verification
Carnegie Mellon University · Toyota Technological Institute at Chicago · Northwestern University
Chart Deep Research in LVLMs via Parallel Relative Policy Optimization
Mar 3, 2026Jiajin Tang, Gaoyang, Wenjie Wang +2Chart Understanding Large Vision-Language Models
ByteDance · ShanghaiTech University · School of Computer Science and Engineering, Sun Yat-sen University
Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs
Mar 3, 2026Prarthana Bhattacharyya, Joshua Mitton, Ralph Abboud +1Knowledge Tracing AI in Education
Eedi · Learning Engineering Virtual Institute
OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets
Mar 3, 2026Jiyuan Shen, Peiyue Yuan, Atin Ghosh +2Document Information Extraction Multimodal Large Language Models
SAP · Stanford University
From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench
Mar 3, 2026Weikang Shi, Houxing Ren, Junting Pan +8Generative AI in Education Intelligent Tutoring Systems
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · CPII under InnoHK · Shenzhen Loop Area Institute
How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities
Mar 3, 2026Ziwen Xu, Kewei Xu, Haoming Xu +8LLM Evaluation Language Model Steering
Zhejiang University · Alibaba Group
A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities
Mar 3, 2026Faiz Ghifari Haznitrama, Faeyza Rishad Ardi, Alice OhLLM Evaluation
School of Computing, KAIST, Daejeon, South Korea
Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning
Mar 2, 2026Justin WaughLLM Evaluation LLM Reasoning
Approximate Labs
CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production
Mar 2, 2026Yixin Nie, Lin Guan, Zhongyao Ma +19LLM Post-Training Conversational Agents
Meta Superintelligence Labs · OpenAI · xAI +1
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
Mar 2, 2026Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov +1LLM Evaluation Bias in LLM-as-a-Judge
Slingshot AI · Department of Engineering, University of Cambridge
Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions
Mar 2, 2026Vineeth Venugopal, Soroush Mahjoubi, Elsa OlivettiLLM Reliability LLM Evaluation
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, 02139, MA, USA.
RubricBench: Aligning Model-Generated Rubrics with Human Standards
Mar 2, 2026Qiyuan Zhang, Junyi Zhou, Yufei Wang +8LLM Evaluation Rubric-Based Evaluation
City University of Hong Kong · Tencent Hunyuan · McGill - Mila & Quebec AI Institute +2
DEP: A Decentralized Large Language Model Evaluation Protocol
Mar 1, 2026Jianxiang Peng, Junhao Li, Hongxiang Wang +15LLM Evaluation Benchmark Contamination
TJUNLP Lab, Tianjin University, Tianjin, China · National Supercomputing Center in Tianjin · Yuanhui AI
CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
Mar 1, 2026Yiyue Qian, Shinan Zhang, Yun Zhou +3LLM Evaluation LLM-as-a-Judge
Amazon AWS Generative AI Innovation Center · Amazon AWS Bedrock
Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
Mar 1, 2026Michael Hardy, Yunsung KimLLM Alignment LLM Evaluation
Stanford University
Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Feb 28, 2026Tianpeng Zheng, Zhehan Jiang, Jiayi Liu +1Computerized Adaptive Testing LLM Evaluation
Institute of Medical Education, Health Science Center, Peking University, Beijing, China · School of Public Health, Peking University, Haidian District, Beijing, China · Peking University Health Science Center-Chaoxing Joint Laboratory for Digital and Smart Medical, Peking University, Beijing, China +1
Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research
Feb 28, 2026Yubo Dong, Nianhao You, Yuxuan Hou +5Autonomous Research Agents LLM Evaluation
College of Computer Science and Technology · Ant Group, Hangzhou, China
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge

Mar 5, 2026J/K move · Enter open · S save

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

Yunfan Zhang, Yijie Bei, Jetashree Ravi, Pawel Garbacki

Columbia University · Fireworks AI

LLM Evaluation Instruction Following Generative AI Evaluation Language Modeling

Open paper AskPDF arXiv

Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.