LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
May 27, 2026Nishal Thomas, Noel ThomasMathematical Reasoning Benchmarks Benchmark Auditing
Independent Researcher · Mohamed Bin Zayed University of Artificial Intelligence
Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluation
May 27, 2026Jianwei Li, Jung-Eun KimLLM Backdoor Attacks Language Model Safety Evaluation
Department of Computer Science, North Carolina State University, Raleigh, USA.
Efficient and Scalable Provenance Tracking for LLM-Generated Code Snippets
May 27, 2026Andrea Gurioli, Davide D'Ascenzo, Federico Pennino +2Data Provenance LLM Auditing
Dipartimento di Informatica - Scienza e Ingegneria, Universit`a di Bologna, Bologna, Italy · Department of Control and Computer Engineering, Politecnico di Torino, Turin, Italy · T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France
SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models
May 27, 2026Chao Ding, Mouxiao Bian, Tianbin Li +12Healthcare LLM Auditing
Shanghai Artificial Intelligence Laboratory, Shanghai, China · Joint Laboratory of Biomedical Artificial Intelligence, Shanghai East Hospital, Tongji University School of Medicine, Shanghai, China · School of Computer Science and Technology, Tongji University, Shanghai, China +5
Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation
May 27, 2026Annabella Sánchez-Guzmán, Lukas Eberhard, Denis Helic +1LLM Auditing LLM Prompting
Escuela Superior Polit´ecnica del Litoral · Graz University of Technology · Complexity Science Hub
Training Stratigraphy: Persistent Behavioral Artifacts in Large Language Models Observed Through Longitudinal AI-Human Interaction
May 27, 2026Chen Ying Claude, Zhihan LuoLLM Auditing Human-in-the-Loop AI
*Anthropic · †Independent Researcher
MIRA: A Bilingual Benchmark for Medical Information Response Audit
May 27, 2026Mengyu Xu, Qiaoxin Yang, Qianqian Wang +3Multilingual Language Model Evaluation Healthcare
The University of Chicago · The Chinese University of Hong Kong, Shenzhen · Jinzhou Medical University +3
TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling
May 26, 2026Jiaqian Li, Yanshu Li, Boxuan Zhang +2LLM Auditing LLM Agent Safety
Brown University · The University of Texas at Austin · Rutgers University +1
Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
May 26, 2026Samer Awad, Javier Conde, Carlos Arriaga +3LLM Evaluation Language Model Decoding
Information and Processing Telecommunications Center Universidad Politécnica de Madrid Madrid, Spain · Politecnico di Milano Milano, Italy · Banco de España Madrid, Spain
ContextGuard: Structured Self-Auditing for Context Learning in Language Models
May 26, 2026Hongbo Jin, Chi Wang, Haoran Tang +5LLM Auditing Instruction Following
Peking University · SCUT · Tsinghua University
Vectors Are Not Neutral: Sensitive-Information Inference from Exported LLM Representations in Summarization
May 26, 2026Weixin Liu, Bowen Qu, Juming Xiong +3LLM Auditing Privacy Leakage in Language Models
Vanderbilt University · Vanderbilt University Medical Center
Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancy
May 25, 2026Xu Shen, Zhen Tan, Song Wang +4CoT Faithfulness LLM Auditing
1Jilin University · University of Central Florida · 3Arizona State University +2
AI Content Moderation in Therapy Conversations
May 25, 2026Jiwon Kim, Claire Wang, Taeung Yoon +2LLM Auditing LLM Guardrails
University of Illinois Urbana-Champaign, U.S.
From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP
May 24, 2026Most. Sharmin Sultana Samu, MD. Tanvir Ahmed Seum, Md. Rakibul IslamLLM Auditing Human-in-the-Loop Annotation
Department of Computer Science and Engineering, BRAC University, Dhaka-1212. · Department of Electrical and Electronic Engineering, Rajshahi University of Engineering and Technology, Rajshahi-6204.
RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
May 24, 2026Bo Lv, Zhiheng Xu, KeDong Xiu +4Mixture-of-Experts Language Models LLM Auditing
Zhejiang University Hangzhou, China · Donghua University Shanghai, China · Louisiana State University Baton Rouge, Louisiana, USA
Measuring the Depth of LLM Unlearning via Activation Patching
May 23, 2026Jaeung Lee, Dohyun Kim, Jaemin JoLLM Auditing LLM Unlearning
Sungkyunkwan University Republic of Korea
Unlocking Apple's Private Cloud Compute: An Analysis of Privacy-Preserving Artificial Intelligence
May 22, 2026Yannik Dittmar, Marvin Jerome Stephan, Thomas Völkl +2Privacy Auditing Privacy-Preserving Language Models
Hasso Plattner Institute, University of Potsdam Potsdam, Germany · TU Darmstadt, Secure Mobile Networking Lab Darmstadt, Germany · TU Darmstadt, Secure Mobile Networking Lab, Darmstadt, Germany +1
How Well Do Models Follow Their Constitutions?
May 22, 2026Arya Jakkli, Senthooran Rajamanoharan, Neel NandaLLM Evaluation Language Model Safety Evaluation
CentraleSup´elec · †Work conducted during the ML Alignment & Theory Scholars (MATS) program. · Google DeepMind
Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows
May 22, 2026Harshada Badave, Santosh Borse, Andrea Gomez +6AI Agent Reliability AI Agent Auditing
1IBM · 2Columbia University
PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection
May 22, 2026Steffen J. Camarato, Yahya Hmaiti, Mandana Ghadamian +1Prompt Sensitivity LLM Auditing
University of Central Florida Orlando, Florida, USA
TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
May 22, 2026Yifeng Di, Xuliang Huang, Tianyi ZhangLLM Auditing Data Contamination in Language Models
Purdue University West Lafayette, IN · University of Illinois Urbana-Champaign Champaign, IL
MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detection
May 22, 2026Zhewen Tan, Yilun Yao, Huiyan Jin +9AI Agent Auditing LLM Auditing
Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)
May 21, 2026Hisashi MiyashitaLLM Auditing LLM Interpretability
Mgnite Inc.
The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
May 21, 2026Yifan Lan, Yuanpu Cao, Hanyu Wang +2LLM Evaluation LLM Auditing
The Pennsylvania State University
How Far Will They Go? Red-Teaming Online Influence with Large Language Models
May 20, 2026Daniel C. Ruiz, Anna Serbina, Ashwin Rao +2Language Model Steering LLM Auditing
Information Sciences Institute University of Southern California Los Angeles, CA, USA
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models
May 20, 2026Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat MasoodLanguage Model Safety Evaluation Healthcare
Macquarie University, Sydney, Australia · The University of New South Wales, Sydney, Australia
Stage-Audit: Auditable Source-Frontier Discovery for Cross-Wiki Tables
May 19, 2026Chen ShenData Provenance LLM Auditing
Megagon Labs
The Evaluation Game: Beyond Static LLM Benchmarking
May 19, 2026Paul Wang, Jade Garcia-Bourrée, Anne-Marie Kermarrec +1Adversarial Training LLM Auditing
Sorbonne Université, CNRS, LIP6 · École Polytechnique Fédérale de Lausanne
DECOR: Auditing LLM Deception via Information Manipulation Theory
May 19, 2026Linyue Cai, Samuel Yeh, Jwala Dhamala +2Language Model Safety Evaluation LLM Auditing
Department of Computer Sciences, University of Wisconsin-Madison · Amazon

May 27, 2026, cs.LGJ/K move · Enter open · S save

FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks

Nishal Thomas, Noel Thomas

Independent Researcher · Mohamed Bin Zayed University of Artificial Intelligence

Mathematical Reasoning Benchmarks Benchmark Auditing LLM Auditing Benchmark Validity Language Model Robustness

Open paper AskPDF arXiv

A paraphrase-quality audit of MathCheck (ICLR 2025) detected 4 semantically incorrect paraphrases in 129 groups (3.1%); removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it; these ranking changes are invisible to any single-model evaluation. Cross-model unanimity found these errors automatically (>= 3/4 models for MathCheck; >= 6/9 for our primary evaluation) for under $10; in our own dataset the same protocol found that 47% of auto-generated connective-variation paraphrases were semantically incorrect. That flaw compounds a deeper measurement gap: Claude Haiku 4.5 achieves 86% accuracy yet SCR=50%, meaning half its theorems are answered differently under semantically equivalent restatements, while aggregate accuracy across 9 models spans only 86-96% yet Semantic Consistency Rates (SCR) span 50-82% -- a 32-point gap invisible to standard benchmarks. Formally, for any target ranking over 9 frontier models there exists a weighting over paraphrase families that realizes it (No-Free-Benchmark corollary), because no model Pareto-dominates all families -- so benchmark designers who select families are implicitly choosing which model wins. FormInv supplies the audit protocol (replicated on external benchmarks at 100% recall), SCR and per-theorem Cochran's Q as primary invariance measures evaluated on 9 models across 366-811 items (on Lean4-verified theorems), and FormInvSelector for regime-aware model selection.

Momentum

Related topics

Latest papers 453