LLM Auditing

LLM: Large Language Model

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.

  1. Discovering Implicit Large Language Model Alignment Objectives

    Feb 17, 2026Edward Chen, Sanmi Koyejo, Carlos GuestrinLLM AlignmentLLM Interpretability

  2. Automatically Finding Reward Model Biases

    Feb 16, 2026Atticus Wang, Iván Arcuschin, Arthur ConmyReward ModelingLLM Auditing

  3. Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs

    Feb 12, 2026Longyuan Zhu, Hairan Hua, Linlin Miao +2LLM EvaluationLLM Auditing

  4. Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

    Feb 10, 2026Wei Yang, Shixuan Li, Heng Ping +3LLM AuditingPreference Optimization

  5. Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

    Feb 3, 2026Anish Sathyanarayanan, Aditya Nagarsekar, Aarush RathoreCoT FaithfulnessFaithfulness of Language Model Explanations

  6. Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

    Feb 2, 2026Martino Ciaperoni, Marzio Di Vece, Roberto Pellungrini +3LLM InterpretabilityLLM Auditing

  7. Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines

    Jan 30, 2026Saeid Jamshidi, Kawser Wazed Nafi, Arghavan Moradi Dakhel +2LLM AuditingMCP Security

  8. Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection?

    Jan 26, 2026Devansh Srivastav, David Pape, Lea SchönherrLanguage Model Safety EvaluationLLM Auditing

  9. White-Box Sensitivity Auditing with Steering Vectors

    Jan 23, 2026Hannah Cyberey, Yangfeng Ji, David EvansAlgorithmic AuditingLLM Auditing

  10. Pro-AI Bias in Large Language Models

    Jan 20, 2026Benaya Trabelsi, Jonathan Shaki, Sarit KrausLanguage Model Bias EvaluationLLM Auditing

  11. A Scalable Entity-Based Framework for Auditing Bias in Large Language Models

    Jan 18, 2026Akram Elbouanani, Aboubacar Tuo, Adrian PopescuLLM AuditingPolitical Bias in Language Models

  12. PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

    Jan 7, 2026Songjun Tu, Yiwen Ma, Jiahao Lin +6Language Model Error DetectionAutomated Peer Review

  13. Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

    Jan 6, 2026David Hartmann, Lena Pohlmann, Lelia Hanslik +3LLM AuditingAlgorithmic Fairness

  14. Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

    Dec 3, 2025Oren Rachmil, Avishag Shapira, Roy Betser +5Language Model Safety EvaluationLLM Auditing

  15. MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS

    Nov 28, 2025Kaixiang Wang, Zhaojiacheng Zhou, Bunyod Suvonov +8Multi-Agent LLM SystemsLLM Auditing

  16. Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness

    Oct 13, 2025Sleem Abdelghafar, Maryam Aliakbarpour, Christopher JermaineData LeakagePrivacy Auditing

  17. Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5

    Oct 8, 2025Shrestha Ghosh, Luca Giordano, Yujia Hu +2Factual Knowledge in Language ModelsLLM Auditing

  18. The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

    Oct 7, 2025Matthieu Bou, Nyal Patel, Arjun Jagota +2Reinforcement LearningLLM Auditing

  19. Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

    Oct 7, 2025Yanming Li, Cédric Eichler, Nicolas Anciaux +3Data ProvenanceDataset Watermarking

  20. What Is The Political Content in LLMs' Pre- and Post-Training Data?

    Sep 26, 2025Tanise Ceron, Dmitry Nikolaev, Dominik Stammbach +1LLM AuditingPolitical Bias in Language Models

  21. The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

    Sep 22, 2025Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou +5LLM EvaluationLLM Auditing

  22. Speculating LLMs' Chinese Training Data Pollution from Their Tokens

    Aug 25, 2025Qingjie Zhang, Di Wang, Haoting Qian +7Data Contamination in Language ModelsLLM Auditing

  23. Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming

    Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20LLM Safety BenchmarksLanguage Model Safety Evaluation

  24. PRISON: Unmasking the Criminal Potential of Large Language Models

    Jun 19, 2025Xinyi Wu, Geng Hong, Pei Chen +3Language Model Safety EvaluationLLM Auditing

  25. But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

    May 23, 2025Leon Eshuijs, Archie Chaudhury, Alan McBeth +1LLM EvaluationLLM-as-a-Judge