LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment
Aug 3, 2026Priyanka BajajLLM Auditing Anomaly Detection
Independent Researcher
A Human Audit of OpenAIs AI-Generated Mathematical Proofs
Aug 3, 2026Mikołaj Sienicki, Krzysztof SienickiLLM Auditing Human Oversight of AI
Polish-Japanese Academy of Information Technology ul. Koszykowa 86, 02-008 Warsaw, Poland, European Union · Chair of Theoretical Physics of Naturally Intelligent Systems (NIS)1 ul. Lipowa 2 / Topolowa 19, 05-807 Podkowa Leśna, Poland, European Union
RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
Aug 3, 2026Divyansh Singh, Reza Davari, Afra MashhadiLLM-as-a-Judge LLM Auditing
Microsoft · University of Florida · University of Washington
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM Auditing Sparse Attention
The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA
Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese
Aug 3, 2026Naho Orita, Hayato Ogawa, Daisuke KawaharaLLM Auditing Social Bias in Language Models
Waseda University
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
Aug 2, 2026Wenhui Chen, Jianlin Chen, Ziyao Lin +2LLM Evaluation LLM-as-a-Judge
University of Macau · South China University of Technology · Independent Researcher
Auditable Release Control for Pedagogical Leakage in LLM Tutors
Aug 1, 2026Nizam KadirLLM Auditing LLM Guardrails
Science, Mathematics and Technology Singapore University of Technology and Design (SUTD) Singapore
Leak It: Per-Document Extraction Beyond Aggregate Membership Inference
Jul 31, 2026Victor MaricatoMembership Inference Attacks LLM Auditing
Karolinska Institutet, Sweden
FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
Jul 31, 2026Martin LukkBenchmark Auditing LLM Auditing
University of Toronto
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM Evaluation Benchmark Auditing
Aleph Alpha Research, Heidelberg, Germany.
AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
Jul 30, 2026Xiangning Lin, Shenzhe Zhu, Shu Yang +23AI Accountability LLM Auditing
2CMU · 3UT Austin · University of Toronto +11
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Jul 30, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeScalable Oversight AI Agent Reliability
Dipartimento di Informatica, Università di Pisa, Pisa, Italy
Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory
Jul 30, 2026Vishwajith RameshMemory-Augmented Language Models LLM Auditing
Vy Labs, Inc.
Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Coupling
Jul 29, 2026Zuyuan Zhang, Hanqing Yang, Carlee Joe-Wong +1Multi-Agent LLM Systems AI Agent Auditing
Department of Electrical and Computer Engineering, The George Washington University · Department of Electrical and Computer Engineering, Carnegie Mellon University
Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLM
Jul 29, 2026Huixiang Zhang, Mahzabeen EmuLLM Auditing Causal Interventions in Language Models
Georgia Institute of Technology · North Ave · Atlanta, GA 30332 USA +3
TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning
Jul 28, 2026Rwaida Alssadi, Muntaser Syed, Balaji Kasula +6Data Provenance LLM Auditing
Florida Institute of Technology
Dynamic Parameterization Is Not Dynamic Inference
Jul 28, 2026Zongfei Li, Yuan-yih Shang, Guozhong LuoEfficient Transformer Inference LLM Auditing
Forecasting Side Effects of Activation Steering
Jul 28, 2026Chong Yong Ong, Alson Wei Jie Sim, Peixin Zhang +1Language Model Steering LLM Auditing
Singapore Management University
Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
Jul 27, 2026Atharva Pandey, Gautam JajooLLM Auditing Human Behavior Simulation
Kairosity
Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosis
Jul 27, 2026Tianqiao Zhao, Meng Yue, Jianhui WangLLM Auditing Multimodal Model Evaluation
The University of Texas at Arlington, Arlington, TX, USA · Brookhaven National Laboratory, Upton, NY, USA · Southern Methodist University, Dallas, TX, USA
LEX-EC: A Lexical Evidence-Channel Audit Framework for Zero-Shot LLM Personality Classification in Black-Box Settings
Jul 27, 2026Brittany Harbison, Ashok K. GoelZero-Shot Text Classification LLM Auditing
Georgia Institute of Technology Atlanta, Georgia, USA
The Half-Lives of Generative-AI Evidence: A 40-Record Audit, a Claim-Currency Framework, and a Reflexive Case of Frontier-Model-Assisted Research
Jul 27, 2026Carlo IaconoLLM Auditing AI-Assisted Scientific Research
Charles Sturt University, Australia
Auditing Alignment Controllability in LLMs via Political Axes
Jul 26, 2026Bartol Bućan, Nikola Sočec, Sarah Isufi +5LLM Alignment Language Model Steering
Student at University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia · It From Bit d.o.o., Zagreb, Croatia · University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia
Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
Jul 25, 2026Jeff OttersonAutomated Software Testing LLM Auditing
W. P. Carey School of Business, Arizona State University
Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science
Jul 24, 2026Davide Scarso, Hugo Noronha de Almeida, Joaquim PinaLLM Evaluation LLM Auditing
CIUHCT - Interuniversity Centre for the History of Science and Technology · MARE - Marine and Environmental Sciences Centre, ARNET - Aquatic Research Network Associate Laboratory
Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines
Jul 24, 2026Santhiya RajanLLM Auditing LLM Agent Reliability
Multiverse Computing
Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
Jul 24, 2026Yunting Song, Matthew Watson, Peter Grabowski +1LLM Safety Benchmarks LLM Auditing
Google Mountain View, CA 94043
Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
Jul 23, 2026Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda +1LLM Auditing Deception in Language Models
dreadnode, Principal AI Research Engineer · dreadnode, StaffAI Security Researcher · dreadnode, Distinguished Engineer +1
Code Monitor Red Teaming for Public-Test-Passing Code
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenLLM Auditing Code Generation Evaluation
University of Electronic Science and Technology of China Chengdu, China
Auditing Evidence Use in Medical LLM Diagnosis
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenLLM Auditing Medical Diagnosis
University of Electronic Science and Technology of China Chengdu, China

Aug 3, 2026, cs.LGJ/K move · Enter open · S save

Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment

Priyanka Bajaj

Independent Researcher

LLM Auditing Anomaly Detection

Open paper AskPDF arXiv

AI systems can fail silently. The failure propagates through training loops, evaluation pipelines, and production monitoring stacks until downstream harm makes it visible. This paper introduces evaluation blindness: a measurement function M exhibits evaluation blindness with respect to failure class F when it produces readings indistinguishable from a healthy state while the system is actually failing, with no auxiliary signal flagging the gap. The problem surfaces at two lifecycle stages the literature has treated separately. At training time, reward models are gamed, importance-sampling corrections are silently miscalculated, and benchmark contamination inflates fine-tuning evaluations, all while loss curves look healthy and gradient updates proceed normally. At deployment time, monitoring fails to catch six classes of production failure, including an Operational category that is 100% silent by structural definition. We provide a formal detectability predicate unifying both stages. Four training-time case studies trace concrete breakdowns, including a real implementation bug in TRL PR #6594 where gradients are corrupted as loss decreases normally. A six-class taxonomy validated against 50 real-world incidents from court documents and regulatory filings finds that 53% of verifiable public failures were silent. A failure budget framework ties acceptable failure rates to use-case risk class. The implication is direct: measurement infrastructure is a correctness concern across the full AI lifecycle, not just at evaluation time. Data, code, and taxonomy schema are at https://github.com/priyanka25aug/llm-failure-taxonomy.

Momentum

Related topics

Latest papers 453