LLM Auditing

LLM: Large Language Model

Ask about this topic

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

644 papers82 papers in the last four weeks, level with the four weeks before. 0.5% of all new papers.

  1. Design Behaviour Codes (DBCs): A Taxonomy-Driven Layered Governance Benchmark for Large Language Models

    Mar 5, 2026G. Madan Mohan, Veena Kiran Nambiar, Kiranmayee JanardhanLLM Safety BenchmarksLLM Guardrails

    Founder, Yonih Ventures, Bangalore, India | Canada · Professor, Ramaiah University of Applied Sciences, Bangalore, India · Data Scientist, Bangalore, India

  2. From Unfamiliar to Familiar: Detecting Pre-training Data via Gradient Deviations in Large Language Models

    Mar 5, 2026Ruiqi Zhang, Lingxiang Wang, Hainan Zhang +2Membership Inference AttacksLLM Auditing

    Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University

  3. Monitoring Emergent Reward Hacking During Generation via Internal Activations

    Mar 4, 2026Patrick Wilhelm, Thorsten Wittkopp, Odej KaoReward HackingEmergent Misalignment in Language Models

    Technical University of Berlin, Germany · BIFOLD - Berlin Institute for the Foundations of Learning and Data, Germany

  4. A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

    Mar 4, 2026Arther Tian, Alex Ding, Frank Chen +2Language Model Generation EvaluationLLM Evaluation

    DGrid AI

  5. Real Money, Fake Models: Deceptive Model Claims in Shadow APIs

    Mar 2, 2026Yage Zhang, Yukun Jiang, Zeyuan Chen +3LLM AuditingBlack-Box Model Auditing

    CISPA Helmholtz Center for Information Security

  6. CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era

    Feb 26, 2026Kaiwen Shi, Weixiang Sun, Zheyuan Zhang +3Citation VerificationLLM Hallucination Detection

    University of Notre Dame · Lehigh University

  7. A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring

    Feb 26, 2026Usman Anwar, Julianna Piskorz, David D. Baek +6Language Model SteganographyLLM Security

    University of Cambridge · Massachusetts Institute of Technology · UK AI Safety Institute +3

  8. Assessing Deanonymization Risks with Stylometry-Assisted LLM Agent

    Feb 26, 2026Boyang Zhang, Yang ZhangPrivacy Leakage in Language ModelsAuthorship Attribution

    CISPA Helmholtz Center for Information Security

  9. IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

    Feb 26, 2026Yanpei Guo, Wenjie Qu, Linyu Wu +7Verifiable Outsourced ComputationLLM Auditing

    National University of Singapore · Nanyang Technological University · Independent Researcher +1

  10. CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safety

    Feb 26, 2026Umid Suleymanov, Rufiz Bayramov, Suad Gafarli +4Zero-Shot Policy TransferLanguage Model Safety Evaluation

    Department of Computer Science, Virginia Tech · School of IT and Engineering, ADA University · School of Law, ADA University

  11. Hidden Topics: Measuring Sensitive AI Beliefs with List Experiments

    Feb 25, 2026Maxim ChupilkinProbability Elicitation from Language ModelsLLM Alignment

  12. A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models

    Feb 21, 2026Trishita Tiwari, Ari Trachtenberg, G. Edward SuhMemorization in Language ModelsLLM Auditing

    Cornell University · Boston University · NVIDIA, Cornell University

  13. LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

    Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety EvaluationLLM Safety Benchmarks

    Princeton University · Phillips Exeter Academy · Independent

  14. What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

    Feb 19, 2026Dimitri Staufer, Kirsten MorehousePrivacy Leakage in Language ModelsLLM Auditing

    TU Berlin, Berlin, Germany · Columbia University, New York, USA

  15. Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs

    Feb 19, 2026Arka Pal, Louai Zahran, William Gvozdjak +2Verifiable Outsourced ComputationLLM Inference

    Ritual · MIT · Columbia University

  16. RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

    Feb 19, 2026Yunseok Han, Yejoon Lee, Jaeyoung DoCoT FaithfulnessLarge Reasoning Models

    AIDAS Laboratory, 1IPAI & 2ECE, Seoul National University

  17. A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Models

    Feb 17, 2026Noa Linder, Meirav Segal, Omer Antverg +5LLM Refusal BehaviorRefusal Behavior in Language Models

    Irregular · University of Zurich

  18. Discovering Implicit Large Language Model Alignment Objectives

    Feb 17, 2026Edward Chen, Sanmi Koyejo, Carlos GuestrinLLM AlignmentLLM Interpretability

    Stanford University.

  19. Automatically Finding Reward Model Biases

    Feb 16, 2026Atticus Wang, Iván Arcuschin, Arthur ConmyReward ModelingLLM Auditing

    MATS · Massachusetts Institute of Technology · Independent

  20. Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

    Feb 16, 2026Kawin Mayilvaghanan, Siddhant Gupta, Ayush KumarLanguage Model Bias EvaluationDemographic Bias in Language Models

    Observe.AI Bangalore, India

  21. NEST: Nascent Encoded Steganographic Thoughts

    Feb 15, 2026Artem KarpovLanguage Model Safety EvaluationLLM Auditing

    Independent.

  22. Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

    Feb 10, 2026Wei Yang, Shixuan Li, Heng Ping +3LLM AuditingPreference Optimization

    University of Southern California

  23. Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

    Feb 3, 2026Anish Sathyanarayanan, Aditya Nagarsekar, Aarush RathoreCoT FaithfulnessFaithfulness of Language Model Explanations

    BITS Pilani, K. K. Birla Goa Campus

  24. Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines

    Jan 30, 2026Saeid Jamshidi, Kawser Wazed Nafi, Arghavan Moradi Dakhel +2LLM AuditingMCP Security

    SWAT Laboratory, Polytechnique Montr´eal, Montr´eal, Canada · SæT Laboratory, Polytechnique Montr´eal, Montr´eal, Canada

  25. Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas

    Jan 29, 2026Katherine Elkins, Jon ChunMoral Reasoning in Language ModelsLLM Auditing

    Kenyon College Gambier, OH

  26. Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection?

    Jan 26, 2026Devansh Srivastav, David Pape, Lea SchönherrLanguage Model Safety EvaluationLLM Auditing

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  27. White-Box Sensitivity Auditing with Steering Vectors

    Jan 23, 2026Hannah Cyberey, Yangfeng Ji, David EvansAlgorithmic AuditingLLM Auditing

    University of Virginia

  28. Pro-AI Bias in Large Language Models

    Jan 20, 2026Benaya Trabelsi, Jonathan Shaki, Sarit KrausLanguage Model Bias EvaluationLLM Auditing

    Bar Ilan University

  29. A Scalable Entity-Based Framework for Auditing Bias in Large Language Models

    Jan 18, 2026Akram Elbouanani, Aboubacar Tuo, Adrian PopescuLLM AuditingPolitical Bias in Language Models

    Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

  30. PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

    Jan 7, 2026Songjun Tu, Yiwen Ma, Jiahao Lin +6Language Model Error DetectionAutomated Peer Review

    Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. +1