LLM Auditing

LLM: Large Language Model

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.

  1. Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

    Mar 17, 2026Shenao Yan, Shan Jin, Shimaa Ahmed +4LLM AuditingBackdoor Detection

  2. Are Large Language Models Truly Smarter Than Humans?

    Mar 17, 2026Eshwar Reddy M, Sourav KarmakarData Contamination in Language ModelsLLM Evaluation

  3. Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models

    Mar 16, 2026Nandini Arimanda, Achyuth Mukund, Sakthi Balan Muthiah +1Intersectional Bias in Language ModelsSocial Bias in Language Models

  4. Seamless Deception: Larger Language Models Are Better Knowledge Concealers

    Mar 15, 2026Dhananjay Ashok, Ruth-Ann Armstrong, Jonathan MayDeception in Language ModelsLLM Auditing

  5. Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance

    Mar 13, 2026Brecht Verbeken, Joke Van den Broeck, Inge De Cleyn +4LLM AuditingGenerative AI in Education

  6. LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

    Mar 12, 2026Himel Ghosh, Nick Elias WernerLanguage Model Bias EvaluationLLM Evaluation

  7. Human-Centred LLM Privacy Audits: Findings and Frictions

    Mar 12, 2026Dimitri Staufer, Kirsten Morehouse, David Hartmann +1Privacy AuditingLLM Auditing

  8. A Hybrid Knowledge-Grounded Framework for Safety and Traceability in Prescription Verification

    Mar 11, 2026Yichi Zhu, Kan Ling, Xu Liu +3Knowledge GraphsBiomedical KGs

  9. CLaRE-ty Amid Chaos: Quantifying Representational Entanglement to Predict Ripple Effects in LLM Editing

    Mar 11, 2026Manit Baser, Alperen Yildiz, Dinil Mon Divakaran +1Language Model EditingRepresentation Geometry in Language Models

  10. Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI

    Mar 8, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +4Verifiable Outsourced ComputationLLM Auditing

  11. Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing

    Mar 7, 2026Arash Marioriyad, Ali Nouri, Mohammad Hossein Rohban +1Deception in Language ModelsLanguage Model Safety Evaluation

  12. Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion

    Mar 6, 2026Hari Shankar, Vedanta S P, Sriharini Margapuri +3Cultural Bias in Language ModelsCross-Cultural Language Model Evaluation

  13. Design Behaviour Codes (DBCs): A Taxonomy-Driven Layered Governance Benchmark for Large Language Models

    Mar 5, 2026G. Madan Mohan, Veena Kiran Nambiar, Kiranmayee JanardhanLLM Safety BenchmarksLLM Guardrails

  14. From Unfamiliar to Familiar: Detecting Pre-training Data via Gradient Deviations in Large Language Models

    Mar 5, 2026Ruiqi Zhang, Lingxiang Wang, Hainan Zhang +2Membership Inference AttacksLLM Auditing

  15. Monitoring Emergent Reward Hacking During Generation via Internal Activations

    Mar 4, 2026Patrick Wilhelm, Thorsten Wittkopp, Odej KaoReward HackingEmergent Misalignment in Language Models

  16. Real Money, Fake Models: Deceptive Model Claims in Shadow APIs

    Mar 2, 2026Yage Zhang, Yukun Jiang, Zeyuan Chen +3LLM AuditingBlack-Box Model Auditing

  17. A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring

    Feb 26, 2026Usman Anwar, Julianna Piskorz, David D. Baek +6Language Model SteganographyLLM Security

  18. Assessing Deanonymization Risks with Stylometry-Assisted LLM Agent

    Feb 26, 2026Boyang Zhang, Yang ZhangPrivacy Leakage in Language ModelsAuthorship Attribution

  19. IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

    Feb 26, 2026Yanpei Guo, Wenjie Qu, Linyu Wu +7Verifiable Outsourced ComputationLLM Auditing

  20. CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safety

    Feb 26, 2026Umid Suleymanov, Rufiz Bayramov, Suad Gafarli +4Zero-Shot Policy TransferLanguage Model Safety Evaluation

  21. A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models

    Feb 21, 2026Trishita Tiwari, Ari Trachtenberg, G. Edward SuhMemorization in Language ModelsLLM Auditing

  22. LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

    Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety EvaluationLLM Safety Benchmarks

  23. What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

    Feb 19, 2026Dimitri Staufer, Kirsten MorehousePrivacy Leakage in Language ModelsLLM Auditing

  24. Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs

    Feb 19, 2026Arka Pal, Louai Zahran, William Gvozdjak +2Verifiable Outsourced ComputationLLM Inference

  25. RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

    Feb 19, 2026Yunseok Han, Yejoon Lee, Jaeyoung DoCoT FaithfulnessLarge Reasoning Models