Overview

Language model auditing examines whether a model’s behavior, internal signals, training history, and deployed tools match claims about its reliability, safety, fairness, and compliance. Auditors use controlled prompts and counterfactual tests, multi-turn and domain-specific benchmarks, black-box output or likelihood analysis, white-box activation probes, and execution logs or claim-level evidence. These methods are applied to settings such as jailbreaks, retrieval-augmented generation, agent workflows, demographic bias, data leakage, and finetuning. A central challenge is validity: proxy scores and automated judges can miss meaningful failures or produce false alarms, so audits need calibrated thresholds, matched controls, realistic settings, and evidence that can be independently checked.

Latest papers

CardsList

176 papers16 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. How Reliable Are Explanations of Language Models That Detect Insider Threats?

    Nov 19, 2026Suramya R. AngdembayFaithfulness of Language Model ExplanationsExplainable Intrusion Detection

  2. Training Language Models for Introspective Oversight

    Nov 17, 2026Alex Lin Wang, Linus Yifeng Tang, Dami Choi +1Language Model MonitoringLanguage Model Introspection

  3. A Full-Ledger Counterfactual Audit of Adaptive Reasoning

    Nov 17, 2026Xiaopeng Chu, Wenzheng Kong, Wei Liu +1Adaptive Reasoning Resource Allocation for Language ModelsAdaptive Reasoning in Language Models

  4. Contract-Explicit Evaluation: From Auditing Learned Readers to Scoped Certificates

    Nov 17, 2026Hao Lu, Weixing Zhu, Ruiyang LiLanguage Model AuditingRelation Extraction

  5. Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study

    Nov 16, 2026Ana Clara Zoppi Serpa, Anderson Rocha, Hélio PedriniPrivacy Leakage in Language ModelsLanguage Model Auditing

  6. Procedure without Judgment: What Small Language Models Learn under an Exact Verifier

    Nov 16, 2026Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R. +2LLM Agent Policy ComplianceConcept-Based Explanations

  7. Evaluating and Improving Natural Language Autoencoders

    Nov 14, 2026Marco Bazzani, Dan MossingAutoencodersNeural Network Interpretability

  8. Constructing and Validating Memorisation Audits for EHR Foundation Models

    Nov 11, 2026Inigo Stibbe, Joshua Placidi, Padmanabhan Ramnarayan +1Memorization in Language ModelsPrivacy Leakage in Language Models

  9. Logit Laundering: Evading Data-Use Auditing in Large Language Models

    Nov 6, 2026Ruihan Hu, Wei Luo, Yu-Ming Shang +5LLM AuditingLanguage Model Auditing

  10. Source-balanced Capped Fisher: Bounded Evidence Aggregation for User Inference in Language Models

    Nov 4, 2026Yang Liu, Ran Zou, Wanrong ZhuLanguage Model AuditingData Provenance in Language Models

  11. Information-Matched Auditing of Conversational Emotion-Shift Forecasting: A Leakage-Safe Evaluation Protocol

    Nov 4, 2026Bin Wen, Dai-Qiao Zhang, Tien-Ping TanEmotion Recognition in ConversationsLanguage Model Auditing

  12. Whose startup does the AI back? An algorithm audit of reputation and demographic signals in LLM-assisted crowdfunding

    Nov 2, 2026Mirza Samad Ahmed Baig, Syeda Anshrah GillaniAlgorithmic BiasDemographic Bias in Language Models

  13. Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers

    Oct 30, 2026Ziyuan Wang, Genliang Zhu, Xiaobo ChenLanguage Model AuditingText Generation

  14. AI evaluators favor job applications written by certain AI models, independent of candidate quality

    Oct 30, 2026Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu +1Algorithmic FairnessLanguage Model Bias Evaluation

  15. DiRAFT: Privacy-Aware Financial Transcript Synthesis from Aggregate Discourse Priors

    Oct 29, 2026Yunbai Zhang, Yunyi Gao, Han-Tai ShiaoSynthetic Data Generation for Language ModelsText Generation

  16. Hidden Commitment: When Language Models Silently Pick a Side and How Steering Can Surface It

    Oct 28, 2026Samuel Dawit Assefa, Jae Won ChoKnowledge Conflicts in Language ModelsLanguage Model Auditing

  17. Watermark Basins: Black-Box Detection of Distilled LLMs

    Oct 27, 2026Hongyan Chang, Hady Elsahar, Tom Sander +10LLM WatermarkingLanguage Model Auditing

  18. Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

    Oct 26, 2026Zheng Zhang, Yifan Wu, Yuxiang Zhou +7Language Model Bias EvaluationIntersectional Bias in Language Models

  19. Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

    Oct 22, 2026Marek Šuppa, Ivan Vykopal, Jaroslav KopčanAI Agent AuditingLanguage Model Auditing

  20. Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

    Oct 19, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM EvaluationLLM Personalization

  21. Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given

    Oct 12, 2026Ben Slater, Ryan Lundqvist, Leo Richter +1AI Agent AuditingLLM Auditing

  22. TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

    Oct 11, 2026Ankur Garg, Xuemin Yu, Hassan Sajjad +1LLM SafetyLLM Safety Monitoring

  23. Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)

    Oct 11, 2026Heng-Sheng ChangDeception in Language ModelsLanguage Model Auditing

  24. SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift

    Oct 3, 2026Noor Islam S. Mohammad, Md. Basim Al Zabir Shammo, Hasan Siddiki +3CoT FaithfulnessLanguage Model Auditing