LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Modeling Memory-Dependent Reliability of LLMs: A Hidden Markov Model
Jul 24, 2026Robab Aghazadeh Chakherlou, Siddartha Khastgir, Peter Popov +1LLM Evaluation Hierarchical Bayesian Modeling
WMG, University of Warwick, Coventry, United Kingdom · Centre for Software Reliability, City St George’s, University of London, London, United Kingdom
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
Jul 24, 2026Junda Zhao, Shurui Zhou, Eldan CohenLLM Evaluation Automated Software Testing
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada
Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science
Jul 24, 2026Davide Scarso, Hugo Noronha de Almeida, Joaquim PinaLLM Evaluation LLM Auditing
CIUHCT - Interuniversity Centre for the History of Science and Technology · MARE - Marine and Environmental Sciences Centre, ARNET - Aquatic Research Network Associate Laboratory
Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Jul 24, 2026Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao +1Creativity Assessment LLM Evaluation
Judge Business School, University of Cambridge, Cambridge, CB2 1AG, United Kingdom · Institute of Metabolic Science, School of Clinical Medicine, University of Cambridge, CB2 0QQ, United Kingdom · Department of Computer Science, University of Manchester, M13 9PL, United Kingdom +2
From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models
Jul 24, 2026Shixin Fang, Jiachen Wo, Wenjuan Qin +2LLM Evaluation
Fudan University
NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
Jul 24, 2026Yuchen Zhou, Niels Bobet, Maribel AcostaLLM Evaluation Knowledge Graphs
Technical University of Munich, Munich, Germany
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management
Jul 24, 2026Yi Zhang, Hongyang Wang, Zheng Hao Leong +11LLM Evaluation
School of Environment, Tsinghua University, Beijing, China. · Fudan University, Shanghai, China. · Nanjing University, Nanjing, China. +4
On Improving Faithfulness of Podcasts from Documents
Jul 24, 2026Soumya Dutta, Tejas Indulal Dhamecha, Pannaga ShivaswamyContextual Faithfulness LLM Evaluation
1Indian Institute of Science · 2Adobe Research
When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs
Jul 23, 2026Anna Mosolova, Djamé SeddahMultilingual Language Model Evaluation LLM Evaluation
INRIA Paris
Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable
Jul 23, 2026Prerit AhujaFinancial Services LLM Evaluation
Independent Researcher · London-based
One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
Jul 23, 2026Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen +2LLM Evaluation Clarification Question Generation
MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"
Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizers
Jul 23, 2026Vasudha Bhatnagar, Purnima Bindal, Vikas Kumar +1LLM Evaluation Text Summarization
Department of Computer Science, University of Delhi, Delhi, 110007, Delhi, India · Department of Statistics, Ramjas College, University of Delhi, Delhi, 110007, Delhi, India
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Jul 23, 2026Hiroki TambaLLM Evaluation Language Model Bias Evaluation
Independent researcher.
Profiling Lightweight Large Language Models
Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference Efficiency LLM Evaluation
Graduate School of Science and Engineering, Saitama University, Saitama, Japan
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Jul 22, 2026Donghwan KimLLM Evaluation Language Model Ensembles
Aidentyx Inc., San Jose, CA, USA
Evaluating Large Language Models for Symbolic Security Protocol Analysis
Jul 22, 2026Paolo Modesti, Syed Ahmed, Ioannis Sfyrakis +1LLM Evaluation Cybersecurity
Teesside University, Middlesbrough, United Kingdom
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
Jul 22, 2026Yichuan Liu, Daniel Cummings, Nick VadlamudiLLM Evaluation Time Series QA
Tenstorrent Austin, TX, USA
Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups
Jul 22, 2026Haran Shani-Narkiss, Michael Fire, Oren TsurLLM Evaluation LLM Alignment
University College London, London, England. · Ben-Gurion University of the Negev, Be’er Sheba, Israel.
Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study
Jul 22, 2026Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha +5LLM Evaluation Moral Reasoning in Language Models
Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia · Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Jul 22, 2026Beining Wang, Weihang Su, Hongtao Tian +8RL for Language Models LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · 2WeChat, Tencent
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
Jul 22, 2026Siyi Hao, Yidi Cao, Linhao Yu +2LLM Evaluation LLM Alignment
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · The International Joint Institute of Tianjin University, Fuzhou, China
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Jul 22, 2026Xinke Tong, Xuanming Zhang, Tianyi Tang +10Numerical Reasoning in Language Models LLM Evaluation
Qwen Team, Alibaba Group · Tsinghua University · Alibaba Group
Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
Jul 21, 2026Mahdiyeh Farajidizaji, Vatsal RainaLLM Evaluation Instruction Following
Khajeh Nasir Toosi University of Technology · Apta AI, Spark AI Research
Total Variation Distance Estimation in Autoregressive Models
Jul 21, 2026Eric Price, Kevin Tian, Zhiyang Xun +1LLM Evaluation LLM Inference
University of Texas at Austin
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
Jul 21, 2026Netanel EliavLLM Evaluation Language Model Generation Evaluation
Machine Human Intelligence Lab (MHIL)
XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding
Jul 21, 2026Hongchen Wei, Yuanzhe Wang, Bei Liu +9Long-Context Retrieval LLM Evaluation
1Wuhan University · 2Microsoft
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
Jul 21, 2026Shaswata Mitra, Subash Neupane, Trisha Chakraborty +4LLM Evaluation LLM Fine-Tuning
The University of Alabama · Meharry Medical College · Mississippi State University +1
Structured Output Collapses Answer Diversity Across 44 Language Models
Jul 20, 2026Tapan ParikhLLM Evaluation Structured Prompting
Cornell Tech
Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Jul 20, 2026Liam SwayneLLM Evaluation LLM Reasoning
It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
Jul 20, 2026Kevin Du, Clara Kümpel, Michelle Wastl +1LLM Evaluation Knowledge Conflicts in Language Models
ETH Zürich · University of Zurich · UC San Diego

Jul 24, 2026, stat.MLJ/K move · Enter open · S save

Modeling Memory-Dependent Reliability of LLMs: A Hidden Markov Model

Robab Aghazadeh Chakherlou, Siddartha Khastgir, Peter Popov, Xingyu Zhao

WMG, University of Warwick, Coventry, United Kingdom · Centre for Software Reliability, City St George’s, University of London, London, United Kingdom

LLM Evaluation Hierarchical Bayesian Modeling LLM Reliability

Open paper AskPDF arXiv

Reliability assessment of large language models (LLMs) seeks to estimate the probability that a model produces correct responses under a specified operational profile. Conventional benchmark-based evaluation, often summarized by aggregate accuracy, provides a point estimate of performance but does not characterize the uncertainty associated with reliability claims. Currently, statistical inference methods for LLM reliability assessment are emerging. However, a key assumption underlying these models is that test outcomes can be treated as independent repeated trials. This assumption may be inappropriate in sequential settings, where later responses depend on earlier interactions through retained context, error propagation, or an evolving interaction state. We extend a hierarchical Bayesian framework for LLM reliability assessment by relaxing the assumption of independent task outcomes and introducing a Hidden Markov Model to capture sequential dependence in benchmark-constructed interaction sessions. In this formulation, outcomes are generated from a latent interaction state evolving according to a first-order Markov process, capturing changes in interaction context. Through experiments using Anthropic Claude and OpenAI on four datasets, we demonstrate the potential impact of sequential dependence on reliability assessment. The results suggest that ignoring sequential dependence may lead to overconfident reliability estimates.

Momentum

Related topics

Latest papers 1,643