Language Model Auditing

Ask about this topic

Overview

Language model auditing examines whether a model’s behavior, internal signals, training history, and deployed tools match claims about its reliability, safety, fairness, and compliance. Auditors use controlled prompts and counterfactual tests, multi-turn and domain-specific benchmarks, black-box output or likelihood analysis, white-box activation probes, and execution logs or claim-level evidence. These methods are applied to settings such as jailbreaks, retrieval-augmented generation, agent workflows, demographic bias, data leakage, and finetuning. A central challenge is validity: proxy scores and automated judges can miss meaningful failures or produce false alarms, so audits need calibrated thresholds, matched controls, realistic settings, and evidence that can be independently checked.

Latest papers

CardsList

160 papers16 papers in the last four weeks, down 43% on the four weeks before. 0.1% of all new papers.

  1. Hidden Commitment: When Language Models Silently Pick a Side and How Steering Can Surface It

    Oct 28, 2026Samuel Dawit Assefa, Jae Won ChoKnowledge Conflicts in Language ModelsLanguage Model Auditing

    KAIST, School of Computing, Daejeon, South Korea. · Konkuk University, School of Electrical and Electronics Engineering, Seoul, South Korea.

  2. Watermark Basins: Black-Box Detection of Distilled LLMs

    Oct 27, 2026Hongyan Chang, Hady Elsahar, Tom Sander +10LLM WatermarkingLanguage Model Auditing

    FAIR, Meta Superintelligence Labs

  3. Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss

    Oct 26, 2026Oz AmramLanguage Model AuditingLLM Evaluation

    Independent

  4. Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

    Oct 26, 2026Zheng Zhang, Yifan Wu, Yuxiang Zhou +7Language Model Bias EvaluationIntersectional Bias in Language Models

    State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Nanyang Technological University

  5. Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

    Oct 23, 2026Karan YadavContent ModerationLLM Evaluation

    Google

  6. Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

    Oct 22, 2026Marek Šuppa, Ivan Vykopal, Jaroslav KopčanAI Agent AuditingLanguage Model Auditing

    Comenius University in Bratislava · Cisco Systems · Kempelen Institute of Intelligent Technologies (KInIT) +1

  7. Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

    Oct 19, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM EvaluationLLM Personalization

    University of New South Wales · Mohamed bin Zayed University of Artificial Intelligence · The Technology Innovation Institute +1

  8. Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?

    Oct 14, 2026Hasan Amin, Zhuoran Lu, Ming YinAuthorship AttributionRobustness of Machine-Generated Text Detection

    Department of Computer Science, Purdue University

  9. On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination

    Oct 13, 2026Kevin Yandoka DenamganaïLanguage Model AuditingAutoencoders

  10. Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

    Oct 12, 2026Raghu Venkat, Tricha AnjaliConfidence Estimation in Language ModelsSelective Prediction

    Numberz.ai Inc., Alpharetta, Georgia, USA

  11. Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given

    Oct 12, 2026Ben Slater, Ryan Lundqvist, Leo Richter +1AI Agent AuditingLLM Auditing

    Pivotal Research · University of Cambridge · University College London, University of London +1

  12. TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

    Oct 11, 2026Ankur Garg, Xuemin Yu, Hassan Sajjad +1LLM SafetyLLM Safety Monitoring

    University of Calgary · Dalhousie University · Mila - Quebec AI Institute

  13. Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)

    Oct 11, 2026Heng-Sheng ChangDeception in Language ModelsLanguage Model Auditing

    Coordinated Science Laboratory University of Illinois Urbana-Champaign

  14. SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift

    Oct 3, 2026Noor Islam S. Mohammad, Md. Basim Al Zabir Shammo, Hasan Siddiki +3CoT FaithfulnessLanguage Model Auditing

    Istanbul Technical University · Pabna University of Science and Technology · American International University-Bangladesh +2

  15. A Controlled Audit of Personal AI Memory for Rating Prediction

    Oct 2, 2026Shivam GuptaMemory-Augmented Language ModelsUser Modeling

    Independent research.

  16. Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening

    Sep 30, 2026Thilo Tamme, Anton Hantel, Bijan Khosrawi-RadText SummarizationSummarization Evaluation

    Technical University of Munich · Massachusetts Institute of Technology · Leuphana University Lüneburg

  17. Selecting The Most Informative Tokens in Natural Language Autoencoders

    Sep 29, 2026Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez +4LLM AuditingAutoencoders

    Department of Computer Science, University of Turin, Italy · Department of Mathematics and Computer Science, University of Southern Denmark, Denmark

  18. Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges

    Sep 29, 2026Mingyue Huo, Shivam Mehta, Bhavin Jawade +2LLM-as-a-JudgeSpeech Quality Assessment

    University of Illinois Urbana-Champaign · Netflix

  19. Language Models Are "Insecure" Reporters

    Sep 28, 2026Jenny Y. Huang, Jiameng Fan, Ahmed Imtiaz Humayun +5Language Model Safety EvaluationDeception in Language Models

    Massachusetts Institute of Technology · Google Research · Harvard University

  20. RAGWarrant: Evidence-Preserving Governance for RAG Policy Promotion Under Quality, Cost, Latency, and Risk Constraints

    Sep 28, 2026Richard Krueger, Lucas Krause, Zach PocquetteRetrieval-Augmented GenerationRAG Evaluation

    Independent open-source research project

  21. Retrospective Distillation Attribution via Normalized Response Similarity

    Sep 26, 2026Minwoo Jang, Jaechang Kim, Minhyeon Oh +2Language Model AuditingKnowledge Distillation

    Graduate School of Artificial Intelligence, POSTECH, South Korea · POSTECH Institute of Artificial Intelligence, South Korea · Department of Computer Science and Engineering, POSTECH, South Korea

  22. Reading Is Not Leaking: Local, Auditable Measurement and Reduction of Inference Exposure from Public Footprints

    Sep 26, 2026Mahmudul Faisal Al AmeenUser Attribute Inference AttacksPrivacy Leakage in Language Models

    Independent researcher France

  23. Checking Leakage Witnesses versus Certifying Bounded Non-Leakage

    Sep 26, 2026Chao Feng, Burkhard StillerLanguage Model AuditingPrivacy Leakage in Language Models

    Communication Systems Group (CSG), University of Zurich, Switzerland

  24. Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

    Sep 24, 2026Tapan ParikhLLM SycophancyAI Agent Evaluation

    Cornell Tech

  25. Truth for Believable AI: Expressed Doubt, Provenance, and Belief Revision as an Engineerable Stance

    Sep 22, 2026Sebastian CochinescuLLM Uncertainty EstimationBelief Updating in LLMs

    University of Bucharest

  26. Auditing Proxy-Based Validation Across Text Spans

    Sep 22, 2026Daein Weon, Dong Ho KangBenchmark ValidityAutomated Evaluation

    Kookmin University · UStechlab

  27. Conduct Under Pressure: What Sixty Language Models Do When a User Pushes

    Sep 21, 2026Tapan ParikhLanguage Model Safety EvaluationPersuasion in Language Models

    Cornell Tech

  28. FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

    Sep 21, 2026Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet +2Multilingual Language Model EvaluationLLM Auditing

    Saarland University · German Research Center for Artificial Intelligence (DFKI) · Barcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain

  29. When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits

    Sep 21, 2026Daein Weon, Dong Ho KangLanguage Model Bias EvaluationLanguage Model Auditing

    Kookmin University · UStechlab

  30. When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

    Sep 20, 2026Esther XinSynthetic Data EvaluationLanguage Model Auditing

    Independent Researcher