LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Jun 13, 2026Yitao LiLLM Evaluation LLM-as-a-Judge
Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
Jun 13, 2026Jierui Zhang, Siyuan Tan, Xinhang Li +8Mathematical Reasoning Benchmarks LLM Evaluation
School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · Graduate College for Engineers Beijing University of Posts and Telecommunications Beijing, China · School of Mathematical Sciences Fudan University Shanghai, China +6
PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino
Jun 13, 2026Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3LLM Evaluation Language Modeling
AI Singapore · Nanyang Technological University · UK AI Security Institute +2
Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning
Jun 13, 2026Yiwei Chen, Lichi Li, Kai Cheung +2LLM Evaluation LLM Fine-Tuning
Cisco Systems, Inc., San Francisco, CA, USA · Michigan State University, East Lansing, MI, USA
Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning
Jun 13, 2026Sanhorn Chen, Xiaoyang Chen, Boyu Liu +1Irregular Time-Series Modeling LLM Evaluation
University of Illinois Urbana Champaign
TriAdReview: Triangular Adversarial Review Architecture for Multi-Model Technical Document Generation
Jun 13, 2026Zhiqiang Zhou, Junliang Dai, Xu LingMulti-Agent LLM Systems LLM Evaluation
Hunan Chemical Industry Vocational and Technical College, Hunan, China
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Jun 12, 2026Shaojie YinMultilingual Language Model Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory Shanghai, China
FactoryLLM: A Safe and Open-Source AI Playground for Evaluating LLMs in Smart Factories
Jun 12, 2026Yash Pulse, Yong-Bin Kang, Abhik Banerjee +2Manufacturing LLM Evaluation
Swinburne University of Technology, Melbourne, Australia
Simulating Students' Java Programming Errors with Large Language Models
Jun 12, 2026Ali Keramati, Jie Cao, Iman Mohammadi +2LLM Evaluation Programming Education
Utah State University
LLMs Contain Multitudes: How Deployment Context Reshapes Model-Level Preferences and Values
Jun 11, 2026Filip Trhlik, Aoife O'Flynn, Angela Yu +2LLM Evaluation Language Model Bias Evaluation
University of Cambridge · 2ALTA Institute · 3Leverhulme Centre for the Future of Intelligence +1
DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation
Jun 11, 2026Li Zhang, Yuzhen Shi, Yiran Hu +15LLM Evaluation Legal QA
University of Pittsburgh · Alibaba Group · University of Waterloo +8
One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders
Jun 11, 2026Minghao Luo, Liang ChenLLM Evaluation RAG Security
The Chinese University of Hong Kong
QIAS 2026: Overview of the Shared Task on Islamic Inheritance Reasoning
Jun 11, 2026Abdessalam Bouchekif, Somaya Eltanbouly, Samer Rashwani +5LLM Evaluation Legal Reasoning
Hamad bin Khalifa University, Qatar · Nazarbayev University, Kazakhstan
SupraBench: A Benchmark for Supramolecular Chemistry
Jun 11, 2026Tianyi Ma, Yijun Ma, Zehong Wang +6LLM Evaluation Molecular Property Prediction
University of Notre Dame · University of Connecticut
Which Models Perform Better in Inheritance Reasoning?
Jun 11, 2026Mohammed Amine Mouhoub, Chahinez BouchekifLLM Evaluation LLM Reliability
Paris Dauphine University · University of Abou Bekr Belkaïd
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
Jun 11, 2026Joe DwyerLLM Evaluation Efficient Language Model Training
Department of Computer Information Science ECPI University Virginia Beach, VA 23462
Evaluating Pluralism in LLMs through Latent Perspectives
Jun 11, 2026Laura Majer, Jan Šnajder, Martin TutekLLM Evaluation LLM Alignment
TakeLab, University of Zagreb, Croatia.
VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation
Jun 11, 2026Yijun Shen, Minghao Shao, Yichen Zhao +4LLM Evaluation Code Generation
Center for Data Science, NYU Shanghai, China · NYU Tandon School of Engineering, USA · NYU Abu Dhabi, UAE
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection
Jun 11, 2026Fuqiang Niu, Bowen ZhangLLM Evaluation Stance Detection
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
Authority, Truth, and Citation Bias: A Large-Scale Multi-Domain Benchmark for Studying Epistemic Susceptibility in Large Language Models
Jun 11, 2026Aryan Khurana, Aravind Ramana RN, Dhruv KumarLLM Evaluation Hallucination in Language Models
BITS Pilani, Pilani Campus, Rajasthan, India.
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
Jun 11, 2026Jiaqi Luo, Jiarun Dai, Zhile Chen +8LLM Evaluation AI Agent Security Benchmarks
Fudan University · Shanghai Artificial Intelligence Laboratory · Concordia AI +1
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
Jun 11, 2026Yu Fu, Yongqi Kang, Yong ZhaoLLM Evaluation AI-Assisted Scientific Research
Sichuan University
GENIE: A Fine-Grained Measure for Novelty
Jun 11, 2026Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng +2LLM Evaluation Language Model Generation Evaluation
The University of Texas at Austin · New York University
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Jun 10, 2026Alan Cooney, David Africa, Geoffrey IrvingLLM Evaluation Deception in Language Models
AI Security Institute
Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation
Jun 10, 2026Zahra Habibzadeh, Paria Khoshtab, Amir Mesbah +1LLM Evaluation LLM Grounding
Tehran Institute for Advanced Studies, Khatam University, Iran
Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
Jun 10, 2026Paul He, Shiva Kasiviswanathan, Dominik JanzingLLM Evaluation Multi-Turn Dialogue Evaluation
NTU Singapore · Amazon Research, Tübingen, Germany · Amazon
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

Jun 13, 2026, cs.AIJ/K move · Enter open · S save

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

Yitao Li

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Anytime-Valid Inference Change-Point Detection Drifting Models

Open paper AskPDF arXiv

Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.

Momentum

Related topics

Latest papers 1,631