LLM Auditing

LLM: Large Language Model

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.

  1. A Statistical Framework for Training Data Auditing of Large Language Models

    Nov 19, 2026Shengjie Niu, Yeheng Ge, Jian HuangLLM AuditingMembership Inference Attacks

  2. Auditing black-box clinical prediction models with a large language model through queries alone: a blinded test with planted flaws

    Nov 18, 2026Zhiyu Liu, Xinyu YangBlack-Box Model AuditingLLM Auditing

  3. What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution

    Nov 17, 2026S. M Seefat AlamAttention Head AnalysisLLM Auditing

  4. LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana

    Nov 16, 2026Yasir Rahman, Ishraq Samiha, Shahnewaz Karim Sakib +1Software Vulnerability DetectionCybersecurity

  5. A Crowd-Audited Atlas of Implicit Bias in LLM-Powered Agents

    Nov 16, 2026Jason Alexander, Yuxuan Li, Michael Licamele +2Social Bias in Language ModelsDemographic Bias in Language Models

  6. AudioTrace: File-Level Membership Detection for Audio Large Language Models

    Nov 16, 2026Guangzhi Sun, Xiao Zhan, Shuai Wang +2LLM AuditingPrivacy Leakage in Language Models

  7. Auditing a Moving Target: Evidence Anchoring and the Expiry of Fairness Audits of Provider-Served Language Models

    Nov 16, 2026Mika FacchiniLLM AuditingLanguage Model Bias Evaluation

  8. Tight PAC-Bayes Generalisation Guarantees for Large Language Model Judges

    Nov 16, 2026Tom A. Lamb, Philip H. S. Torr, Tim G. J. RudnerLLM-as-a-JudgeLanguage Model Safety Evaluation

  9. Necessary for What? Auditing Demographic Disclosure to LLM Synthetic Respondents

    Nov 16, 2026Aspen Wang, Yen Yu Shan, Lun-Wei KuLLM AuditingDemographic Bias in Language Models

  10. Designing Deceptive Model Organisms that Successfully Evade Auditing

    Nov 16, 2026Vasisht Duddu, Mark Wang, Lipeng He +1AI SafetyDeception in Language Models

  11. When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing

    Nov 16, 2026Minxuan Hu, Qishi Zhan, Ziheng Chen +4LLM RoutingCost-Aware Inference

  12. Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems

    Nov 15, 2026Spandan Roy, Subhabrata Majumdar, Soudeep DebAI-Assisted Decision MakingLLM Auditing

  13. The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)

    Nov 15, 2026Bahaeddin TürkoğluOpen-Weight Language ModelsML Reproducibility

  14. Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems

    Nov 15, 2026Thilo Tamme, Anton Hantel, Jakob ScheppersLLM AuditingLLM Reliability

  15. A Backdoor Trigger Is Legible in Fine-Tune Weights

    Nov 14, 2026Snigdha Chandan KhilarLLM AuditingBackdoor Defense in LLMs

  16. The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search

    Nov 14, 2026Gaoyuan Du, Xiaoyang Liu, Rex Zhou +4Automated Machine LearningLLM Safety Evaluation

  17. Label-Free Detection of Structural Bias in LLM-Based Recommender Systems via Randomized Paraphrasing: The Case of Verbosity

    Nov 13, 2026Carlos Peláez-González, David Herrera-Poyatos, Francisco Herrera +1Language Model Bias EvaluationRecommender Systems

  18. From Bytes to Behavior: Request-Level Attribution from Live GPU Hidden States

    Nov 13, 2026Jonathan Pan, Yidi Yuan, Xin Wei Chia +3LLM AuditingLLM Inference

  19. AI Reviews Are Biased Towards the Appendix

    Nov 13, 2026Saptarshi Ghosh, Mohammad Saim, Tianyu JiangAutomated Peer ReviewBias in LLM-as-a-Judge

  20. An Operationalization of Opaque Serial Depth

    Nov 12, 2026Nathan Sheffield, Alek Westover, Lukas Finnveden +3Implicit Reasoning in Language ModelsLLM Interpretability

  21. Auditing AI peer reviewers: a benchmark on real scientific papers

    Nov 12, 2026Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro +2Automated Peer ReviewLanguage Model Error Detection

  22. Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing

    Nov 11, 2026Hongfu Gao, Jiang Hu, Pengfei Liu +1Anytime-Valid InferenceLLM-Assisted Annotation