LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability
Apr 17, 2026Yongkang Li, Panagiotis Eustratiadis, Yixing Fan +1LLM Evaluation Retrieval Robustness
University of Amsterdam, The Netherlands · Chinese Academy of Sciences, China
MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition
Apr 17, 2026Farhad Abtahi, Abdolamir Karbalaie, Eduardo Illueca-Fernandez +1Belief Revision LLM Evaluation
Department of Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institutet, 17177 Stockholm, Sweden · Department of Clinical Physiology, Karolinska University Hospital, 17176 Stockholm, Sweden · Department of Biomedical Engineering and Health Systems, KTH Royal Institute of Technology, 14157 Huddinge, Sweden +2
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Apr 17, 2026Judith Sieker, Sina ZarrießLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals
Apr 17, 2026Jeremy Qin, Maksym AndriushchenkoLLM Evaluation Forecasting Benchmarks
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Artificial Effort
Apr 17, 2026Federico Belotti, Stefano Coniglio, Antonio Cosma +1LLM Evaluation
University of Bergamo, Italy.
Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language
Apr 17, 2026Dianqing Lin, Tian Lan, Jiali Zhu +7LLM Evaluation
College of Computer Science, Inner Mongolia University, China
KWBench: Measuring Unprompted Problem Recognition in Knowledge Work
Apr 17, 2026Ankit MalooLLM Evaluation
Clio AI
The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
Apr 17, 2026Jon-Paul CacioliLLM Evaluation Language Model Calibration
Independent Researcher Melbourne, Australia
LLMbench: A Comparative Close Reading Workbench for Large Language Models
Apr 16, 2026David M. BerryLLM Evaluation LLM Interpretability
University of Sussex
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
Apr 16, 2026Tingjia Miao, Wenkai Jin, Muhua Zhang +19LLM Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University · Zhiyuan College, Shanghai Jiao Tong University · School of Physics and Astronomy, Shanghai Jiao Tong University +7
Fabricator or dynamic translator?
Apr 16, 2026Lisa Vasileva, Karin SimLLM Evaluation Hallucination Detection
Language Weaver
Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options
Apr 16, 2026Nahyun Lee, Guijin SonLLM Evaluation Multiple-Choice Question Answering
Chung-Ang University · Seoul National University
Prompt-Driven Code Summarization: A Systematic Literature Review
Apr 16, 2026Afia Farjana, Zaiyu Cheng, Antonio MastropaoloLLM Evaluation LLM Prompting
William & Mary, USA
PeerPrism: Peer Evaluation Expertise vs Review-writing AI
Apr 16, 2026Soroush Sadeghian, Alireza Daqiq, Radin Cheraghi +3LLM Evaluation AI-Generated Text Detection
Reviewerly Toronto, Ontario, Canada · University of Toronto, Reviewerly Toronto, Ontario, Canada · UC Berkeley, Reviewerly Berkeley, California, United States
MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Apr 15, 2026Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires +6LLM Evaluation Multilingual Language Model Evaluation
Maritaca AI · Jusbrasil
LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB
Apr 15, 2026Vekil Bekmyradov, Noah C. Pütz, Thomas Bartz-BeielsteinShortcut Learning in Language Models LLM Evaluation
THK-AI Research Cluster, TH Köln
The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious
Apr 15, 2026Ferdinand M. SchesslLLM Evaluation Multi-Turn Dialogue Evaluation
Independent Researcher
LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text
Apr 15, 2026Jason Potteiger, Andrew Hong, Ito ZapataHuman Preference Evaluation LLM Evaluation
Dimension Labs
LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
Apr 15, 2026Sumeet Ramesh Motwani, Daniel Nichols, Charles London +17Long-Horizon LLM Reasoning CoT Reasoning
University of Oxford · Lawrence Livermore National Laboratory (LLNL) · MBZUAI +4
From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
Apr 15, 2026Itay Itzhak, Eliya Habba, Gabriel Stanovsky +1LLM Evaluation Human Preference Evaluation
Technion – Israel Institute of Technology · The Hebrew University of Jerusalem
A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models
Apr 15, 2026Andrei Marian Feier, Veysel Kocaman, Yigit Gul +6LLM Safety LLM Safety Evaluation
John Snow Labs Inc.
On Cost-Effective LLM-as-a-Judge Improvement Techniques
Apr 15, 2026Ryan Lail, Luke MarkhamLLM-as-a-Judge LLM Evaluation
Composo AI.
ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models
Apr 15, 2026Jason Z WangLLM Evaluation Open-Weight Language Models
Independent
The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models
Apr 15, 2026Jason Z WangLLM Evaluation
Independent
BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks
Apr 15, 2026Sebastian Nagl, Matthias GrabmairLLM Evaluation Legal NLP
Technical University of Munich, Munich, Germany
Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models
Apr 15, 2026Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen +5Diffusion Language Models Diffusion Language Model Inference
Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, USA · Department of Computer Science & Engineering, Texas A&M University, College Station, USA · Department of Biochemistry, Case Western Reserve University, Cleveland, USA +2
Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints
Apr 15, 2026Md. Fahad Ullah Utsho, Mohd. Ruhul Ameen, Akif Islam +2LLM Evaluation Large Reasoning Models
Department of Information and Communication Engineering, University of Rajshahi, Bangladesh · College of Engineering and Computer Sciences, Marshall University, Huntington, WV, USA · Department of Computer Science and Engineering, University of Rajshahi, Bangladesh
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific Agent Benchmarks
Kahlert School of Computing University of Utah
One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
Apr 14, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1Language Model Robustness Instruction Tuning
University of Southern California, Los Angeles, USA · Intel AI, USA

Apr 17, 2026J/K move · Enter open · S save

On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability

Yongkang Li, Panagiotis Eustratiadis, Yixing Fan, Evangelos Kanoulas

University of Amsterdam, The Netherlands · Chinese Academy of Sciences, China

LLM Evaluation Retrieval Robustness Adversarial Robustness of Language Models Language Model Robustness Dense Retrieval

Open paper AskPDF arXiv

Decoder-only large language models (LLMs) are increasingly replacing BERT-style architectures as the backbone for dense retrieval, achieving substantial performance gains and broad adoption. However, the robustness of these LLM-based retrievers remains underexplored. In this paper, we present the first systematic study of the robustness of state-of-the-art open-source LLM-based dense retrievers from two complementary perspectives: generalizability and stability. For generalizability, we evaluate retrieval effectiveness across four benchmarks spanning 30 datasets, using linear mixed-effects models to estimate marginal mean performance and disentangle intrinsic model capability from dataset heterogeneity. Our analysis reveals that while instruction-tuned models generally excel, those optimized for complex reasoning often suffer a ``specialization tax,'' exhibiting limited generalizability in broader contexts. For stability, we assess model resilience against both unintentional query variations~(e.g., paraphrasing, typos) and malicious adversarial attacks~(e.g., corpus poisoning). We find that LLM-based retrievers show improved robustness against typos and corpus poisoning compared to encoder-only baselines, yet remain vulnerable to semantic perturbations like synonymizing. Further analysis shows that embedding geometry (e.g., angular uniformity) provides predictive signals for lexical stability and suggests that scaling model size generally improves robustness. These findings inform future robustness-aware retriever design and principled benchmarking. Our code is publicly available at https://github.com/liyongkang123/Robust_LLM_Retriever_Eval.