LLM Auditing

LLM: Large Language Model

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

644 papers82 papers in the last four weeks, level with the four weeks before. 0.5% of all new papers.

ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attacks
Mar 30, 2026Chihan Huang, Huaijin Wang, Shuai WangMembership Inference Attacks Privacy Leakage in Language Models
HKUST, Hong Kong, China
Dual-Stage LLM Framework for Scenario-Centric Semantic Interpretation in Driving Assistance
Mar 29, 2026Jean Douglas Carvalho, Hugo Taciro Kenji, Ahmad Mohammad Saber +3LLM Auditing Autonomous Driving Safety Evaluation
Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers
Mar 26, 2026Mingmeng Geng, Yuhang Dong, Thierry PoibeauAI-Generated Text Detection LLM Auditing
École Normale Supérieure (ENS) – Université Paris Sciences et Lettres (PSL) · Laboratoire Lattice · Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
The Anatomy of Uncertainty in LLMs
Mar 26, 2026Aditya Taparia, Ransalu Senanayake, Kowshik Thopalli +1LLM Auditing LLM Reliability
School of Computing and Augmented Intelligence, Arizona State University · Lawrence Livermore National Laboratory
MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting
Mar 26, 2026Anisha Roy, Dip RoyMembership Inference Attacks LLM Auditing
Department of Electronics and Communication Engineering, Jaypee Institute of Information Technology, Noida, India · Department of Computer Science and Engineering, Indian Institute of Technology Patna, India
LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale
Mar 25, 2026Muhammed Saeed, Simon RazniewskiFactual Knowledge in Language Models LLM Reliability
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
Adaptively Robust LLM Monitoring via Activation Watermarking
Mar 24, 2026Toluwani Aremu, Daniil Ognev, Samuele Poppi +1LLM Auditing LLM Watermarking
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)
PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset
Mar 24, 2026Sumin Yu, Juhyeon Park, Taesup MoonAI-Assisted Decision Making LLM Auditing
ECE, Seoul National University · IPAI, Seoul National University · ASRI / INMC / AIIS, Seoul National University
Detecting Non-Membership in LLM Training Data via Rank Correlations
Mar 24, 2026Pranav Shetty, Mirazul Haque, Zhiqiang Ma +1LLM Auditing
JPMorgan AI Research
Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Mar 23, 2026Richard J. YoungCoT Faithfulness Faithfulness of Language Model Explanations
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA
Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning
Mar 23, 2026Domonkos VargaLLM Auditing LLM Evaluation
Transparent Screening for LLM Inference and Training Impacts
Mar 23, 2026Arnault Pachot, Thierry PetitLLM Auditing Responsible AI
Emotia, Paris, France
Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks
Mar 23, 2026Yiliang Song, Hongjun An, Jiangan Chen +4LLM Evaluation Benchmark Contamination
Institute of Artificial Intelligence (TeleAI), China Telecom · Guangxi Normal University · Northwestern Polytechnical University
Epistemic Observability in Language Models
Mar 20, 2026Tony Mason, Vaastav AnandLLM Hallucination Detection LLM Auditing
Max Planck Institute for Software Systems
Auditing Preferences for Brands and Cultures in LLMs
Mar 18, 2026Jasmine Rienecker, Katarina Mpofu, Naman Goel +4Jurisdictional Bias in Language Models LLM Auditing
Stupid Human · University of Oxford
DebugLM: Learning Traceable Training Data Provenance for LLMs
Mar 18, 2026Wenjie Jacky Mo, Qin Liu, Xiaofei Wen +3Data Provenance in Language Models Training Data Attribution
University of California, Davis · University of Southern California
Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning
Mar 17, 2026Shenao Yan, Shan Jin, Shimaa Ahmed +4LLM Auditing Backdoor Detection
University of Connecticut · Visa Research
Are Large Language Models Truly Smarter Than Humans?
Mar 17, 2026Eshwar Reddy M, Sourav KarmakarData Contamination in Language Models LLM Evaluation
Applied AI Scientist, Health Vectors · Senior AI Scientist, Intuit India
Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models
Mar 16, 2026Nandini Arimanda, Achyuth Mukund, Sakthi Balan Muthiah +1Intersectional Bias in Language Models Social Bias in Language Models
Department of Computer Science and Engineering, Shiv Nadar University Chennai, Tamil Nadu, India · School of AI and CS, Plaksha University, Mohali, Punjab, India
InterPol: De-anonymizing LM Arena via Interpolated Preference Learning
Mar 16, 2026Minsung Cho, Jaehyung KimLanguage Model Fingerprinting Privacy Leakage in Language Models
Yonsei University
Seamless Deception: Larger Language Models Are Better Knowledge Concealers
Mar 15, 2026Dhananjay Ashok, Ruth-Ann Armstrong, Jonathan MayDeception in Language Models LLM Auditing
Information Sciences Institute, University of Southern California
AEX: Non-Intrusive Multi-Hop Attestation and Provenance for LLM APIs
Mar 15, 2026Yongjie GuanRemote Attestation LLM Auditing
Zhejiang University of Technology
Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance
Mar 13, 2026Brecht Verbeken, Joke Van den Broeck, Inge De Cleyn +4LLM Auditing Generative AI in Education
Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · Harvard University, Cambridge, Massachusetts 02138, USA
LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation
Mar 12, 2026Himel Ghosh, Nick Elias WernerLanguage Model Bias Evaluation LLM Evaluation
Technical University of Munich, Germany · Sapienza University of Rome, Italy
Human-Centred LLM Privacy Audits: Findings and Frictions
Mar 12, 2026Dimitri Staufer, Kirsten Morehouse, David Hartmann +1Privacy Auditing LLM Auditing
TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany +1
A Hybrid Knowledge-Grounded Framework for Safety and Traceability in Prescription Verification
Mar 11, 2026Yichi Zhu, Kan Ling, Xu Liu +3Knowledge Graphs Biomedical KGs
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
CLaRE-ty Amid Chaos: Quantifying Representational Entanglement to Predict Ripple Effects in LLM Editing
Mar 11, 2026Manit Baser, Alperen Yildiz, Dinil Mon Divakaran +1Language Model Editing Representation Geometry in Language Models
National University of Singapore, Singapore · A*STAR Institute for Infocomm Research (A*STAR I2R), Singapore
Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI
Mar 8, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +4Verifiable Outsourced Computation LLM Auditing
Virginia Tech · Washington University in St. Louis
Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing
Mar 7, 2026Arash Marioriyad, Ali Nouri, Mohammad Hossein Rohban +1Deception in Language Models Language Model Safety Evaluation
Sharif University of Technology
Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion
Mar 6, 2026Hari Shankar, Vedanta S P, Sriharini Margapuri +3Cultural Bias in Language Models Cross-Cultural Language Model Evaluation
IIIT Hyderabad · IIIT Kottayam · IIT Kharagpur

Mar 30, 2026J/K move · Enter open · S save

ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attacks

Chihan Huang, Huaijin Wang, Shuai Wang

HKUST, Hong Kong, China

Membership Inference Attacks Privacy Leakage in Language Models LLM Auditing

Open paper AskPDF arXiv

The pervasive deployment of deep learning models across critical domains has concurrently intensified privacy concerns due to their inherent propensity for data memorization. While Membership Inference Attacks (MIAs) serve as the gold standard for auditing these privacy vulnerabilities, conventional MIA paradigms are increasingly constrained by the prohibitive computational costs of shadow model training and a precipitous performance degradation under low False Positive Rate constraints. To overcome these challenges, we introduce a novel perspective by leveraging the principles of model reprogramming as an active signal amplifier for privacy leakage. Building upon this insight, we present ReproMIA, a unified and efficient proactive framework for membership inference. We rigorously substantiate, both theoretically and empirically, how our methodology proactively induces and magnifies latent privacy footprints embedded within the model's representations. We provide specialized instantiations of ReproMIA across diverse architectural paradigms, including LLMs, Diffusion Models, and Classification Models. Comprehensive experimental evaluations across more than ten benchmarks and a variety of model architectures demonstrate that ReproMIA consistently and substantially outperforms existing state-of-the-art baselines, achieving a transformative leap in performance specifically within low-FPR regimes, such as an average of 5.25% AUC and 10.68% TPR@1%FPR increase over the runner-up for LLMs, as well as 3.70% and 12.40% respectively for Diffusion Models.