LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers228 papers in the last four weeks, down 18% on the four weeks before. 1.5% of all new papers.

Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning
Mar 27, 2026Rongjin Li, Zichen Tang, Xianghe Wang +9LLM Evaluation Scientific Reasoning in Language Models
Beijing University of Posts and Telecommunications
ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Mar 27, 2026Inês Vieira, Inês Calvo, Iago Paulo +6LLM Evaluation Language Model Generation Evaluation
1NOVA University of Lisbon, Portugal · 2NOVA LINCS
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Mar 27, 2026Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19Portuguese Language Models LLM Training
NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações +2
CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law
Mar 27, 2026JiHyeok Jung, TaeYoung Yoon, HyunSouk ChoLegal Reasoning LLM Evaluation
KAIST AI · Law School, Ajou University · Department of Artificial Intelligence, Ajou University
Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators
Mar 27, 2026Mengjia Wu, Yi Zhang, Robin Haunschild +1Automated Peer Review Scholarly Peer Review
Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney (Australia) · Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569 Stuttgart (Germany) · Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich (Germany)
Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
Mar 27, 2026Xue Liu, Xin Ma, Yuxin Ma +36LLM Evaluation Rubric-Based Evaluation
ByteDance Seed
Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats
Mar 27, 2026Pierre Epron, Adrien Coulet, Mehwish AlamNamed Entity Recognition LLM Information Extraction
HeKA | U1346, DIG · Inria, Inserm, Université Paris Cité, HeKA, UMR 1346, Paris, France · Télécom Paris, Institut Polytechnique de Paris, France +2
LLM Benchmark-User Need Misalignment for Climate Change
Mar 27, 2026Oucheng Liu, Lexing Xie, Jing JiangLLM Evaluation
School of Computing, The Australian National University, Canberra, Australia
Can Small Models Reason About Legal Documents? A Comparative Study
Mar 26, 2026Snehit VaddiSmall Language Models Legal Reasoning
Independent Researcher
Robust Reasoning Benchmark
Mar 26, 2026Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko +1Language Model Robustness LLM Evaluation
University of Toronto · Vector Institute
In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts
Mar 26, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation Knowledge Conflicts in Language Models
Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +2
Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodology
Mar 26, 2026Zhou Hanlin, Chan Huah YongLLM Routing LLM Evaluation
School of Computer Sciences, Universiti Sains Malaysia (USM), Penang, Malaysia · Xiamen Software Vocational & Technical College, Xiamen, China
Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment
Mar 26, 2026Warren Johnson, Charles LeeLLM Routing Small Language Models
Plexor Labs, Sammamish, WA, USA · Project Autobots, Seattle, WA, USA
Navigating the Prompt Space: Improving LLM Classification of Social Science Texts Through Prompt Engineering
Mar 26, 2026Erkan Gunes, Christoffer Florczak, Tevfik Murat YildirimLarge Language Model-Based Text Classification LLM Prompting
Constructor University · Aalborg University · University of Stavanger
From Untestable to Testable: Metamorphic Testing in the Age of LLMs
Mar 25, 2026Valerio TerragniAutomated Software Testing Metamorphic Testing
University of Auckland
Fine-Tuning A Large Language Model for Systematic Review Screening
Mar 25, 2026Kweku Yamoah, Noah Schroeder, Emmanuel Dorley +2Systematic Review Automation LLM Fine-Tuning
University of Florida, Gainesville, FL, 32611
ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models
Mar 25, 2026Kemal DüzkarLLM Evaluation Scientific Reasoning in Language Models
Olivenet Kyrenia, Cyprus
Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali
Mar 25, 2026Ananda Rimal, Adarsha RimalLow-Resource Language Processing Domain Adaptation for LLMs
Nepal Engineering College · Dept. of Computer Science & Engineering, Nepal Engineering College · Tribhuvan University +1
IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
Mar 24, 2026Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1LLM Evaluation Cultural Knowledge in Language Models
The University of Edinburgh
The Diminishing Returns of Early-Exit Decoding in Modern LLMs
Mar 24, 2026Rui Wei, Rui Du, Hanfei Yu +4Early-Exit Neural Networks LLM Inference Acceleration
Stevens Institute of Technology · Northeastern University · Stony Brook University
Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots
Mar 24, 2026Licol Zeinfeld, Alona Strugatski, Ziva Bar-Dov +3Educational Assessment Generative AI Evaluation
Weizmann Institute of Science
GTO Wizard Benchmark
Mar 24, 2026Marc-Antoine Provost, Nejc Ilenic, Christopher Solinas +1Agent Benchmarks Game-Playing Agents
GTO Wizard
Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks
Mar 24, 2026Fatih UenalLLM Evaluation Legal NLP
University of Colorado Boulder
LLMORPH: Automated Metamorphic Testing of Large Language Models
Mar 24, 2026Steven Cho, Stefano Ruberto, Valerio TerragniLLM Evaluation Language Model Robustness
University of Auckland, Auckland, New Zealand · JRC European Commission, Ispra, Italy
LLM Olympiad: Why Model Evaluation Needs a Sealed Exam
Mar 24, 2026Jan Christian Blaise Cruz, Alham Fikri AjiLLM Evaluation Benchmark Validity
MBZUAI
UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities
Mar 24, 2026Qi Jia, Haodong Zhao, Dun Pei +7AI Agent Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Can an LLM Detect Instances of Microservice Infrastructure Patterns?
Mar 24, 2026Carlos Eduardo Duarte, Neil B. Harrison, Filipe Figueiredo Correia +2Software Engineering LLM Evaluation
∗INESC TEC, Faculdade de Engenharia, Universidade do Porto, Porto, Portugal · †Department of Computer Science, Utah Valley University, Utah, USA · ‡INESC TEC, Porto, Portugal
Can Large Language Models Reason and Optimize Under Constraints?
Mar 24, 2026Fabien Bernier, Salah Ghamizi, Pantelis Dogoulis +1Constraint-Guided Reasoning in Language Models LLM Evaluation
SnT – University of Luxembourg, Luxembourg · Luxembourg Institute of Health (LIH), Luxembourg
JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Mar 24, 2026Yuhui Wang, Zhixiong Yang, Ming Zhang +10LLM Evaluation Multi-Turn Dialogue Evaluation
1Fudan University
Ran Score: a LLM-based Evaluation Score for Radiology Report Generation
Mar 24, 2026Ran Zhang, Yucong Lin, Zhaoli Su +13Radiology Report Evaluation Radiology Report Generation
School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450003, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China +7

Mar 27, 2026J/K move · Enter open · S save

Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Rongjin Li, Zichen Tang, Xianghe Wang, Xinyi Hu+8

Beijing University of Posts and Telecommunications

LLM Evaluation Scientific Reasoning in Language Models Multimodal Reasoning

Open paper AskPDF arXiv

With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose ScholScan, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm.