LLM Auditing

LLM: Large Language Model

Ask about this topic

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

644 papers82 papers in the last four weeks, level with the four weeks before. 0.5% of all new papers.

  1. Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

    Apr 20, 2026Ziyang LiuLLM Auditing

    Independent Researcher

  2. ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding

    Apr 20, 2026Zirui Wang, Yusen Hou, Shaofeng Liang +4LLM AuditingLanguage Model Fingerprinting

    The Hong Kong University of Science and Technology (Guangzhou) · Deep Interdisciplinary Intelligence Lab (DI2Lab)

  3. AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code

    Apr 19, 2026William M. ParrisLLM AuditingStatic Code Analysis

    BDB Labs | Jurisprudential AI Governance Initiative

  4. PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations

    Apr 19, 2026Patrick KeoughHealthcareLLM Auditing

    Independent Researcher Bremerton, Washington, USA

  5. The Granularity Gap: A Multi-Dimensional Longitudinal Audit of Sycophancy in Gemini Models

    Apr 19, 2026Patrick KeoughLLM SycophancyLLM Alignment

    Independent Researcher Bremerton, WA, USA

  6. Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation

    Apr 18, 2026Michelle Star, Andrew Aquilina, Yu-Ru LinLLM EvaluationEmotional Support Conversation

    School of Computing and Information, University of Pittsburgh

  7. One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them

    Apr 18, 2026Ali Holmov, Paul Youssef, Nandi Schoots +1Knowledge EditingTransformer Interpretability

    Technical University of Munich · Marburg University · University of Oxford

  8. DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training

    Apr 18, 2026Ziwen Pan, Zihan Liang, Jad Kabbara +1LLM AuditingSocial Bias in Language Models

    Emory University · MIT

  9. Introspection Adapters: Training LLMs to Report Their Learned Behaviors

    Apr 18, 2026Keshav Shenoy, Li Yang, Abhay Sheshadri +4LLM AuditingLanguage Model Introspection

    Anthropic Fellows · Ashwood Centre on AI Science and Policy, University of Cambridge · Anthropic

  10. Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases

    Apr 17, 2026Eric Gan, Aryan Bhatt, Buck Shlegeris +2LLM Auditing

    Redwood Research

  11. LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning

    Apr 17, 2026Mahir Labib Dihan, Abir MuhtasimLLM Auditing

    Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET) Dhaka, Bangladesh

  12. The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime

    Apr 14, 2026Jason Z WangUncertainty CalibrationLanguage Model Calibration

    Independent

  13. Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies

    Apr 10, 2026Avni MittalLLM AuditingLLM Safety Evaluation

  14. Spectral Geometry of LoRA Adapters Encodes Training Objective and Predicts Harmful Compliance

    Apr 10, 2026Roi PaulLLM AuditingLow-Rank Adaptation

    Independent Researcher

  15. A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges

    Apr 8, 2026Chenchen Kuai, Jiwan Jiang, Zihao Zhu +8Correlated Errors in Language ModelsLLM Auditing

    Texas A&M University · Marquette University · University of Utah

  16. What Does a Sharing Question Add? Auditing LLM Survey Scores for Misinformation

    Apr 8, 2026Zonghuan Xu, Xiang Zheng, Yutao Wu +1LLM EvaluationLLM Auditing

    Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · City University of Hong Kong, Hong Kong SAR, China · Deakin University, Australia

  17. Epistemic Blinding: An Inference-Time Protocol for Auditing Prior Contamination in LLM-Assisted Analysis

    Apr 7, 2026Michael CuccareseLLM AuditingData Contamination in Language Models

  18. Auditable Agents

    Apr 7, 2026Yi Nian, Aojie Yuan, Haiyue Zhang +7AI AccountabilityAI Agent Auditing

    University of Southern California · Arizona State University · University of Tennessee, Knoxville +1

  19. Can We Trust a Black-box LLM? LLM Untrustworthy Boundary Detection via Bias-Diffusion and Multi-Agent Reinforcement Learning

    Apr 7, 2026Xiaotian Zhou, Di Tang, Xiaofeng Wang +1LLM AuditingLanguage Model Bias Evaluation

    Worcester Polytechnic Institute, USA · Indiana University Bloomington, USA

  20. CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

    Apr 7, 2026Akshat Dasula, Prasanna Desikan, Jaideep Srivastava +2LLM AuditingClinical Language Model Evaluation

    Centific AI Research, Redmond, USA · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA

  21. LatentAudit: Real-Time White-Box Faithfulness Monitoring for Retrieval-Augmented Generation with Verifiable Deployment

    Apr 7, 2026Zhe Yu, Wenpeng Xing, Meng HanLLM AuditingRAG Evaluation

    Binjiang Institute of Zhejiang University · Communication University of Zhejiang · Zhejiang University

  22. Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

    Apr 6, 2026Zhenhang Shang, Yingzhe Yu, Kani ChenNeural Network VerificationLLM Auditing

    The Hong Kong University of Science and Technology, Hong Kong, China

  23. How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline

    Apr 5, 2026Mir Tafseer Nayeem, Davood RafieiLanguage Model PretrainingLLM Auditing

    Department of Computing Science University of Alberta

  24. Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research

    Apr 4, 2026Max Hao Lu, Ryan Ellegood, Rony Rodriguez-Ramirez +1LLM-Assisted AnnotationLLM Auditing

    Harvard University / United States

  25. Structural Rigidity and the 57-Token Predictive Window: A Physical Framework for Inference-Layer Governability in Large Language Models

    Apr 4, 2026Gregory M. RuddellLanguage Model Safety EvaluationInference-Time Language Model Alignment

    SnailSafe.ai — Carson City, Nevada

  26. Measuring LLM Trust Allocation Across Conflicting Software Artifacts

    Apr 3, 2026Noshin Ulfat, Ahsanul Ameen Sabit, Soneya Binta HossainLLM ReliabilitySoftware Engineering

    University of Texas at Dallas Texas, USA · University of Texas at Dallas Richardson, Texas, USA

  27. SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models

    Apr 3, 2026Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur +3Social Bias in Language ModelsLanguage Model Bias Evaluation

    Enkrypt AI

  28. Combating Data Laundering in LLM Training

    Apr 2, 2026Muxing Li, Zesheng Ye, Sharon Li +1LLM AuditingData Contamination in Language Models

    University of Melbourne · University of Wisconsin-Madison

  29. G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMs

    Apr 1, 2026Ravi Ranjan, Utkarsh Grover, Xiaomin Lin +1Membership Inference AttacksPrivacy Leakage in Language Models

    Florida International University, Miami, USA · University of South Florida, Tampa, USA

  30. Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?

    Mar 31, 2026Max Kaufmann, David Lindner, Roland S. Zimmermann +1CoT MonitoringCoT Supervision

    Google DeepMind