LLM Auditing

LLM: Large Language Model

Ask about this topic

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

644 papers82 papers in the last four weeks, level with the four weeks before. 0.5% of all new papers.

  1. Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

    Jan 6, 2026David Hartmann, Lena Pohlmann, Lelia Hanslik +3LLM AuditingAlgorithmic Fairness

    Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe +2

  2. Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

    Dec 3, 2025Oren Rachmil, Avishag Shapira, Roy Betser +5Language Model Safety EvaluationLLM Auditing

    Fujitsu Research of Europe · Ben-Gurion University of the Negev

  3. MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS

    Nov 28, 2025Kaixiang Wang, Zhaojiacheng Zhou, Bunyod Suvonov +8Multi-Agent LLM SystemsLLM Auditing

    Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China

  4. Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness

    Oct 13, 2025Sleem Abdelghafar, Maryam Aliakbarpour, Christopher JermaineData LeakagePrivacy Auditing

    Rice University Houston, USA

  5. Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5

    Oct 8, 2025Shrestha Ghosh, Luca Giordano, Yujia Hu +2Factual Knowledge in Language ModelsLLM Auditing

    University of Tübingen · ScaDS.AI & TU Dresden · VNU University of Engineering and Technology

  6. The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

    Oct 7, 2025Matthieu Bou, Nyal Patel, Arjun Jagota +2Reinforcement LearningLLM Auditing

    1Imperial College London · 2Amazon AGI

  7. Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

    Oct 7, 2025Yanming Li, Cédric Eichler, Nicolas Anciaux +3Data ProvenanceDataset Watermarking

    PETSCRAFT · Petscraft, Inria, Palaiseau, France · INSA CVL, Bourges, France +6

  8. WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

    Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning BenchmarksBenchmark Auditing

    Nanyang Technological University

  9. What Is The Political Content in LLMs' Pre- and Post-Training Data?

    Sep 26, 2025Tanise Ceron, Dmitry Nikolaev, Dominik Stammbach +1LLM AuditingPolitical Bias in Language Models

    Bocconi University, Italy · University of Manchester, UK · Princeton University, USA

  10. The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

    Sep 22, 2025Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou +5LLM EvaluationLLM Auditing

    CUHK · JHU · CMU +3

  11. Speculating LLMs' Chinese Training Data Pollution from Their Tokens

    Aug 25, 2025Qingjie Zhang, Di Wang, Haoting Qian +7Data Contamination in Language ModelsLLM Auditing

    Tsinghua University · Ant Group · Nanyang Technological University +1

  12. Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming

    Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20LLM Safety BenchmarksLanguage Model Safety Evaluation

    Cherise · School of Computation, Information and Technology, Technical University of Munich (TUM), Munich, Germany · Department of Engineering Science, University of Oxford, Oxford, United Kingdom +9

  13. PRISON: Unmasking the Criminal Potential of Large Language Models

    Jun 19, 2025Xinyi Wu, Geng Hong, Pei Chen +3Language Model Safety EvaluationLLM Auditing

    Fudan University · Shanghai Innovation Institute

  14. Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

    May 21, 2025Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus +1Prompt Injection DefensePrompt Injection Attacks on LLMs

    Imperial College London