LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs
May 7, 2026Adrián Gude, Roi Santos-Ríos, Francis Bond +3LLM Evaluation
Universidade da Coruña, CITIC · Palacký University, Olomouc · Independent Researcher
Strat-LLM: Stratified Strategy Alignment for LLM-based Stock Trading with Real-time Multi-Source Signals
May 7, 2026Wenliang Huang, Zengyi YuLLM Evaluation LLM Alignment
School of Economics Zhejiang University of Technology Hangzhou, China · Faculty of Education East China Normal University Shanghai, China
Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
May 7, 2026Yang Xu, Jiefu Zhang, Haixiang Sun +3LLM Evaluation Selective Inference
Purdue University · Johns Hopkins University · Purdue University, USA
Logic-Regularized Verifier Elicits Reasoning from LLMs
May 7, 2026Xinyu Wang, Changzhi Sun, Lian Cheng +4LLM Evaluation LLM Answer Verification
Department of Computer Science and Technology, East China Normal University · Institute of Artificial Intelligence (TeleAI), China Telecom
CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency
May 7, 2026Hirofumi Ota, Naoto Iwase, Yuki Ichihara +2LLM Evaluation Anytime-Valid Inference
arXiv:2605.05873v1
What Makes a Latent Thought Valid? Defining and Auditing Thought Representations in LLMs
May 7, 2026Fahd Seddik, Fatemeh FardLLM Evaluation LLM Interpretability
Department of Computer Science, University of British Columbia, Kelowna, BC, Canada
I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs
May 7, 2026Abdullah AbdullahLLM Evaluation LLM Fine-Tuning
orinu Inc. Hwaseong, Republic of Korea
MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge
May 6, 2026Perry E. RadauLLM Evaluation Healthcare
Department of Radiology, University of Calgary · Child and Adolescent Imaging Research (CAIR) Program · Alberta Children’s Hospital Research Institute +1
The Shared Latent Structure of LLM Psychometric Responses
May 6, 2026Hubert Plisiecki, Sabina Siudaj, Kacper Dudzic +4LLM Evaluation Personality Modeling in Language Models
IDEAS Research Institute · University of Warsaw
Curated AI beats frontier LLMs at pharma asset discovery
May 6, 2026Łukasz Kidziński, Kevin ThomasLLM Evaluation Healthcare
Gosset Research
BenCSSmark: Making the Social Sciences Count in LLM Research
May 6, 2026Arnault Chatelain, Étienne Ollion, Qianwen Guan +7LLM Evaluation Benchmark Design
CREST (École Polytechnique, ENSAE, CNRS), 5 avenue Le Chatelier, 91120 Palaiseau, France · LLF (Université Paris Cité and CNRS), UFRL Olympe de Gouges, 13 place Paul Ricoeur, 75013 Paris, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +1
Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
May 6, 2026Aofan Liu, Jingxiang MengPrompt Sensitivity LLM Evaluation
1Peking University. · University of Chicago.
Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)
May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Audio-Language Model Evaluation LLM Evaluation
Google USA & Germany
CANDI: Contextual Alignment for Niche Domains Question Answering
May 6, 2026Megha Chakraborty, Darssan L. Eswaramoorthi, Het Riteshkumar Shah +5LLM Evaluation LLM Grounding
University of South Carolina · IIIT-Delhi · Indian Institute of Technology Madras
How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models
May 6, 2026Sai Sourabh MadurLLM Evaluation Moral Reasoning in Language Models
Independent researcher (ex-Meta)
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
May 6, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Evaluation LLM Alignment
Department of Computer Science, University of Oxford
From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions
May 6, 2026Xiaolong Wang, Zhe Zhao, Song Lai +5LLM Evaluation LLM Prompting
Squirrel Ai Learning Shanghai, China · City University of Hong Kong Hong Kong, China · Zhejiang Normal University Squirrel Ai Learning Jinhua, China +2
Evaluating Advanced Prompting on Gemini Flash for Multi-Hop Biomedical QA
May 5, 2026Ahmed Bajaber, Mohammed AlliheediMulti-Hop QA LLM Evaluation
1Saudi Med AI Lab (SMAIL) 2Prince Sultan University, Riyadh, Saudi Arabia · 2Prince Sultan University, Riyadh, Saudi Arabia · 3Al-Baha University, Al Baha, Saudi Arabia
Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs
May 5, 2026Stefano Bannò, Kate Knill, Mark GalesLLM Evaluation Automated Essay Scoring
ALTA Institute, Department of Engineering, University of Cambridge (UK)
ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions
May 5, 2026Serafim BatzoglouLLM Evaluation Structural Causal Models
Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing
May 5, 2026Humam Khan, Md Tabrez Nafis, Shahab Saquib Sohail +2LLM Evaluation LLM Prompting
Department of Computer Science and Engineering, SEST Jamia Hamdard New Delhi, India · Energy, Resources and Industry WSP Calgary, Canada
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
May 5, 2026David Gringras, Misha SalahshoorLLM Evaluation LLM Auditing
Harvard University · AISST, Cambridge, MA
The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models
May 5, 2026Daniel Drucker, Kyle MahowaldLLM Self-Correction LLM Evaluation
Department of Philosophy The University of Texas at Austin · Department of Linguistics The University of Texas at Austin
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
May 5, 2026Shuyue Stella Li, Rui Xin, Teng Xiao +8RL for Language Models LLM Evaluation
University of Washington · 2Allen Institute for AI · University of Pennsylvania
MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following
May 5, 2026Jaeyun Lee, Junyoung Koh, Zeynel Tok +2LLM Evaluation LLM-as-a-Judge
University of Oxford · Yonsei University
TriBench-Ko: Evaluating LLM Risks in Judicial Workflows
May 5, 2026Haesung Lee, Gyubin Choi, Eun-Ju Lee +5LLM Evaluation Language Model Safety Evaluation
Center for Trustworthy AI, Seoul National University · Graduate School of Data Science, Seoul National University · Responsible AI Team, KT +1
OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking
May 5, 2026Yiding Ma, Chengyun Ruan, Kaibo Huang +3LLM Evaluation Forecasting Benchmarks
Beijing University of Posts and Telecommunications
PatRe: A Full-Stage Office Action and Rebuttal Generation Benchmark for Patent Examination
May 5, 2026Qiyao Wang, Xinyi Chen, Longze Chen +4LLM Evaluation Legal Document Analysis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Dalian University of Technology +2
Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
May 5, 2026Zhongjian Zhang, Yue Yu, Mengmei Zhang +3LLM Evaluation LLM Reasoning with Graphs
Beijing University of Posts and Telecommunications · China Telecom Bestpay · Beihang University
Revisiting the Travel Planning Capabilities of Large Language Models
May 5, 2026Bo-Wen Zhang, Jin Ye, Peng-Yu Hua +4LLM Self-Correction LLM Evaluation
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

May 7, 2026J/K move · Enter open · S save

More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs

Adrián Gude, Roi Santos-Ríos, Francis Bond, Dan Flickinger+2

Universidade da Coruña, CITIC · Palacký University, Olomouc · Independent Researcher

LLM Evaluation

Open paper AskPDF arXiv

This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text, in this case, English news text. We focus in particular on the evaluation of syntactic properties through formal grammar frameworks. Our analysis compares two generations of LLMs in the context of two human-authored English news datasets from two different years. Employing the Head-Driven Phrase Structure Grammar (HPSG) formalism, we investigate the distributions of syntactic structures and lexical types of AI-generated texts and contrast them with the corresponding distributions in the human-authored New York Times (NYT) articles. We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types. We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models. These findings point to future work in studying effects of instruction tuning, which, while enhancing coherence and adherence to prompts, may narrow the expressive range of model output.