LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
May 16, 2026Aleksandr ChurilovLLM Evaluation Hallucination in Language Models
Independent Researcher
PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts
May 16, 2026Khizar Hussain, Murat KantarciogluLLM Evaluation Language Model Error Detection
Virginia Tech
HalluScore: Large Language Model Hallucination Question Answering Benchmark
May 16, 2026Aisha Alansari, Hamzah LuqmanMultilingual Language Model Evaluation LLM Evaluation
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals · SDAIA-KFUPM Joint Research Center for Artificial Intelligence
PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks
May 16, 2026Máté Metzger, Nadnapang PhophichitLLM Evaluation Machine Translation
Independent Researcher · International Buddhist Studies College · Mahachulalongkornrajavidyalaya University
GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models
May 16, 2026Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta +1LLM Evaluation LLM Auditing
Clark Atlanta University · United International University
Exploring Lightweight Large Language Models for Court View Generation
May 16, 2026Zhitian Hou, Tianyong Hao, Nanli Zeng +2LLM Evaluation Small Language Models
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · 3China Mobile Internet Co., Ltd.
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning
May 15, 2026Shradha Agarwal, Deepak Rajbhar, Tariq JMathematical Reasoning Benchmarks LLM Evaluation
Department of Nuclear Engineering and Computer Science Missouri University of Science and Technology Rolla, Missouri
SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs
May 15, 2026Avijit Shil, Suman SamuiLLM Evaluation Factual Consistency Evaluation
Maulana Abul Kalam Azad University of Technology, West Bengal, India · National Institute of Technology Durgapur, West Bengal, India
A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation
May 15, 2026Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein KarimiLLM Evaluation Learning to Rank
University of Waterloo, Waterloo, Ontario, Canada.
Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments
May 15, 2026Izabella Krzeminska, Michal Butkiewicz, Ewa KomkowskaLLM Evaluation Psychometric Validation
Orange Research, AI Center, Warsaw, Poland
Evaluating Chinese Ambiguity Understanding in Large Language Models
May 15, 2026Junwen Mo, Yuanzhi Lu, Yifang Xue +2LLM Evaluation
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · School of Software Engineering, South China University of Technology, Guangzhou Higher Education Mega Centre, Panyu District, Guangzhou, 510006, Guangdong, China
MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models
May 15, 2026Weixin Liu, Congning Ni, Shelagh A. Mulvaney +4LLM Evaluation Healthcare
Vanderbilt University, Nashville, TN, USA · Vanderbilt University Medical Center, Nashville, TN, USA · Virginia Tech, Blacksburg, VA, USA
RTL-BenchMT: Dynamic Maintenance of RTL Generation Benchmark Through Agent-Assisted Analysis and Revision
May 15, 2026Jing Wang, Shang Liu, Hangan Zhou +1LLM Evaluation Electronic Design Automation
Hong Kong University of Science and Technology · Clear Water Bay, Hong Kong
FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
May 14, 2026Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov +4LLM Evaluation Quantitative Finance
Lime FinTech
Neural Activation Patterns Across Language Model Architectures: A Comprehensive Analysis of Cognitive Task Performance
May 14, 2026Mahdi Naser-Moghadasi, Faezeh GhaderiLLM Evaluation Decoder-Only Language Models
Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX · University of Texas at Arlington
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
May 14, 2026Gaojie Jin, Yong Tao, Lijia Yu +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, University of Exeter · Institute of AI for Industries, Chinese Academy of Sciences · Department of Mathematics and Computer Science, Eindhoven University of Technology
LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
May 14, 2026Philipp Mondorf, Samuel J. Bell, Jesse Dodge +1LLM Evaluation Neural Network Robustness
FAIR at Meta · LMU Munich · Munich Center for Machine Learning +1
Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory
May 14, 2026Songwei Dong, Zihan Chen, Chengshuai Shi +3Continual Learning for LLMs LLM Evaluation
University of Virginia · Princeton University
LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design
May 14, 2026Marilyn Zhang, Tianfeng Chen, Fabián Barzuna +2LLM Evaluation AI Agent Benchmarks
Zero-Shot Goal Recognition with Large Language Models
May 14, 2026Kin Max Piamolini Gusmão, Nathan Gavenski, Nir Oren +1Zero-Shot Learning LLM Evaluation
PUCRS · King’s College London · University of Aberdeen +1
Small, Private Language Models as Teammates for Educational Assessment Design
May 14, 2026Chris Davis Jaldi, Anmol Saini, Shan Zhang +3LLM Evaluation LLM-as-a-Judge
Wright State University, USA · University of Florida, USA · TIB – Leibniz Information Centre for Science and Technology, Germany
XDomainBench: Diagnosing Reasoning Collapse in High-Dimensional Scientific Knowledge Composition
May 14, 2026Gong Zhiren, Tiantong Wu, Jiaming Zhang +9AI for Science LLM Evaluation
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · Department of Mathematics and Mathematical Statistics, Umeå University, Sweden +2
Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation
May 14, 2026GAng PengMultilingual Language Model Evaluation LLM Evaluation
Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China
Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)
May 14, 2026Assaf Gerner, Netta Madvil, Nadav Barak +11LLM Evaluation RAG Evaluation
Deepchecks, Ramat Gan, Israel · Ben-Gurion University, Beer Sheva, Israel
The Great Pretender: A Stochasticity Problem in LLM Jailbreak
May 14, 2026Jean-Philippe Monteuuis, Cong Chen, Jonathan PetitLLM Safety Benchmarks LLM Evaluation
Qualcomm Technologies, Inc.
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
May 13, 2026Stefan Baack, Christo Buschek, Maty BohacekLLM Evaluation Benchmark Design
Independent Researcher · Mozilla Fellowship · Stanford University
Do Language Models Align with Brains? Prediction Scores Are Not Enough
May 13, 2026Xiao JiaLLM Evaluation
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Inter-Rater Reliability LLM Evaluation
Rochester Institute of Technology · Google Research
(How) Do Large Language Models Understand High-Level Message Sequence Charts?
May 13, 2026Mohammad Reza MousaviLLM Evaluation Chart Understanding
Department of Informatics, King’s College London
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
May 13, 2026Andrea MorandiLLM Evaluation Pairwise Comparison
Cisco Systems, Inc.

May 16, 2026, cs.CRJ/K move · Enter open · S save

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

Aleksandr Churilov

Independent Researcher

LLM Evaluation Hallucination in Language Models Software Supply Chain Security Code Language Models LLM Code Generation

Open paper AskPDF arXiv

Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting -- the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically; following coordinated disclosure with PyPI Security and Socket.dev, 53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry's existing defenses, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.

Momentum

Related topics

Latest papers 1,643