LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaTransformer Interpretability LLM Evaluation
Universitat Pompeu Fabra1 · ICREA2
The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation
Jan 9, 2026Aparna Elangovan, Lei Xu, Mahsa Elyasi +8LLM Evaluation Generative AI Evaluation
Independent Researcher · Amazon · Gazi University +6
Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding
Jan 8, 2026Sungmok Jung, Yeonkyoung So, Joonhak Lee +3LLM Evaluation LLM Fine-Tuning
Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Jan 7, 2026Yao Dou, Benjamin Mamut, Wei XuLegal NLP LLM Evaluation
Georgia Institute of Technology
Value-Action Alignment in Large Language Models under Privacy-Prosocial Conflict
Jan 7, 2026Guanyu Chen, Chenxiao Yu, Xiyang HuLLM Evaluation LLM Alignment
Arizona State University · University of Southern California
EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning
Jan 6, 2026Mingyang Wei, Dehai Min, Zewen Liu +8LLM Evaluation Healthcare
1Emory University · University of Illinois Chicago · 3Microsoft
Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs
Jan 6, 2026Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura +1LLM Evaluation Human Preference Modeling
CyberAgent · Nara Institute of Science and Technology · Institute of Science Tokyo
ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems
Jan 5, 2026Noel ThomasSymbolic Reasoning in Language Models LLM Reasoning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Measuring all the noises of LLM Evals
Dec 24, 2025Sida I. WangLLM Evaluation Automated Evaluation
FAIR at Meta
Psychometric Comparability of LLM-Based Digital Twins
Dec 22, 2025Yufei Zhang, Zhihao MaLLM Evaluation Personality Modeling in Language Models
Computational Communication Collaboratory, School of Journalism and Communication, Nanjing University, Nanjing 210023, China
Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
Dec 7, 2025Zhiyu Xu, Jia Liu, Yixin Wang +1LLM Evaluation Latent Variable Models
Department of Statistics, Columbia University. · Department of Statistics, University of Michigan
RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories
Dec 3, 2025Roy Rinberg, Usha Bhalla, Igor Shilov +2LLM Evaluation LLM Unlearning
Harvard University · MATS · Imperial College London +1
ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text
Dec 2, 2025Kerry Luo, Michael Fu, Joshua Peguero +6VLM Evaluation LLM Evaluation
Algoverse AI Research
A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
Dec 1, 2025Suqing Liu, Runlong Ye, Christopher Eaton +2LLM Evaluation Computer Science Education
McMaster University, Hamilton, Ontario, Canada · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada · Research Institute for the Study of University Pedagogy, University of Toronto Mississauga, Mississauga, Ontario, Canada +1
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model Evaluation LLM Evaluation
AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Nov 26, 2025Sarina Xi, Vishisht Rao, Justin Payan +1Language Model Error Detection LLM Evaluation
Carnegie Mellon University
Large language models replicate and predict human cooperation across experiments in game theory
Nov 6, 2025Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal +1LLM Evaluation Game Theory
Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya +1
Sequential Bayesian Evaluation of Large Language Model Behavior
Nov 4, 2025Saatvik Kher, Shang Wu, Rachel Longjohn +2LLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of California, Irvine · Department of Statistics, University of California, Irvine
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Nov 4, 2025Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2LLM Evaluation Long-Context Language Model Evaluation
Carnegie Mellon University, Language Technologies Institute
A Quantitative Study of Sustained Focus in Large Language Models via Repetitive Deterministic Prediction Tasks
Nov 2, 2025Wanda Hou, Leon Zhou, Hong-Ye Hu +3LLM Evaluation LLM Reliability
EdenCode Inc. · Stevenson School · Harvard University +2
Idea2Plan: Exploring AI-Powered Research Planning
Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM Evaluation LLM-as-a-Judge
University of Michigan · Microsoft Research
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Oct 27, 2025Liwei Jiang, Yuanjun Chai, Margaret Li +7Open-Ended Generation LLM Evaluation
University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University +3
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios
Oct 23, 2025Xiaoyuan Wu, Roshni Kaushik, Wenkai Li +2LLM Evaluation LLM-as-a-Judge
Fujitsu Research of America, Inc. · Carnegie Mellon University
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Oct 23, 2025Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9Creativity Assessment LLM Evaluation
University of Pittsburgh · Johns Hopkins University · University of Notre Dame +4
KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
Oct 21, 2025Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6Multilingual Language Model Evaluation LLM Evaluation
Naver Cloud · Graduate School of Data Science, Seoul National University
The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Models
Oct 19, 2025Shivam Ratnakar, Sanjay RaghavendraRetrieval-Augmented Generation LLM Evaluation
University of Southern California
BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
Oct 18, 2025Tian Xia, Tianrun Gao, Wenhao Deng +4LLM Evaluation LLM Grounding
Westlake University, China · Uniforce AI, China
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
Oct 10, 2025Yuchen Lu, Run Yang, Yichen Zhang +6LLM Evaluation Reasoning Evaluation
1Shanghai University of Finance and Economics

Jan 9, 2026J/K move · Enter open · S save

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

Nora Graichen, Iria de-Dios-Flores, Gemma Boleda

Universitat Pompeu Fabra1 · ICREA2

Transformer Interpretability LLM Evaluation Syntactic Generalization in Language Models

Open paper AskPDF arXiv

We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on over 3,000 datapoints spanning a wide range of syntactic phenomena, languages, models, and methods. We take the data to collectively show that TLMs encode a non-trivial amount of syntactic knowledge. Behavioral evidence shows strong performance on formal syntactic phenomena, but weaker and more variable performance on phenomena at the syntax-semantics interface. Performance is also consistently lower for languages with less digital support. Probing and mechanistic studies further support the presence of syntactic knowledge in TLMs. Yet, because most work remains observational and methodologically heterogeneous, insight into the detailed computational mechanisms underlying syntactic processing remains limited. At the same time, the literature remains heavily concentrated on English and BERT-like models. We discuss the implications of our results and provide recommendations for future research.