LLM Auditing

LLM: Large Language Model

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

644 papers82 papers in the last four weeks, level with the four weeks before. 0.5% of all new papers.

TRUST: A Framework for Decentralized AI Service v.0.1
Apr 29, 2026Yu-Chao Huang, Zhen Tan, Mohan Zhang +3AI Agent Reliability AI Accountability
UNITES Lab, University of North Carolina at Chapel Hill · Arizona State University · Columbia University
Persuadability and LLMs as Legal Decision Tools
Apr 29, 2026Oisin Suttle, David LillisLLM Auditing LLM Decision-Making
School of Law and Criminology Maynooth University Maynooth, Ireland · School of Computer Science University College Dublin Dublin, Ireland
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
Apr 28, 2026David Hartmann, Manuel Tonneau, Angelie Kraft +7LLM Evaluation LLM Auditing
Weizenbaum Institute · TU Berlin · University of Oxford +9
Authority Inversion in LLM-Mediated Ubiquitous Systems: When Models Trust Users Over Sensors
Apr 28, 2026Long Zhang, Zi-bo Qin, Wei-neng ChenKnowledge Conflicts in Language Models LLM Auditing
School of Computer Science and Engineering, South China University of Technology, Guangzhou, China
Making AI-Assisted Grant Evaluation Auditable without Exposing the Model
Apr 28, 2026Kemal BicakciLLM Auditing Privacy-Preserving ML
Informatics Institute, Istanbul Technical University, Istanbul, Türkiye · Securify Information Technology and Security Training Consulting Inc., Ankara, Türkiye
Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM
Apr 28, 2026Vinith M. Suriyakumar, Ayush Sekhari, Lena Stempfle +5LLM Auditing Language Model Probing
MIT · BU · Thorn
Evaluating whether AI models would sabotage AI safety research
Apr 27, 2026Robert Kirk, Alexandra Souly, Kai Fronsdal +2Language Model Safety Evaluation LLM Auditing
UK AI Security Institute · UK AISI Research Affiliate
A Multi-Dimensional Audit of Politically Aligned Large Language Models
Apr 27, 2026Lisa Korver, Mohamed Mostagir, Sherief RedaLLM Auditing Political Bias in Language Models
Department of Computer Engineering Brown University Providence, RI 02906 · Ross School of Business University of Michigan Ann Arbor, MI 48109
One Size Fits None: Heuristic Collapse in LLM Investment Advice
Apr 26, 2026Jillian Ross, Andrew W. LoFinancial Services LLM Auditing
Computer Science and Artificial Intelligence Laboratory · Massachusetts Institute of Technology · Cambridge, MA, USA +1
When AI reviews science: Can we trust the referee?
Apr 26, 2026Jialiang Wang, Yuchen Liu, Hang Xu +7LLM Auditing Adversarial Attacks on LLMs
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, School of Computer Science and Engeering, Southeast University, Nanjing 210000, China · These authors contributed equally +2
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation
Apr 26, 2026Lichen Li, Hengguang Zhou, Yijun Liang +2RL for Code Generation Reward Hacking
Peking University · University of California, Los Angeles · University of Maryland +2
VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs
Apr 25, 2026Yurui Xiang, Xingyi Mao, Rui Sheng +7Healthcare LLM Auditing
Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation
Apr 24, 2026Matteo Pelossi, Rita Sevastjanova, Thilo Spinner +1LLM Auditing Social Bias in Language Models
ETH Zurich
Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning
Apr 24, 2026Chaoran Chen, Dayu Yuan, Peter KairouzLLM Auditing RL Fine-Tuning
Google
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Apr 23, 2026Harsh Kumar, Rahul Maity, Tanmay Joshi +4Language Model Pretraining LLM Backdoor Attacks
Manipal University Jaipur · NIT, Karnataka · Pragya Lab, BITS Pilani Goa, India +2
Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations
Apr 23, 2026Nalin Poungpeth, Nicholas Clark, Tanu MitraLLM Auditing Human-AI Interaction
Department of Cognitive Science Northwestern University Evanston, IL 60201, USA · Information School University of Washington Seattle, WA 98195 USA
PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning
Apr 23, 2026Xiaoyi Chen, Haoyuan Wang, Siyuan Tang +4LLM Auditing Privacy Leakage in Language Models
Indiana University Bloomington · Independent Researcher · Michigan State University +2
Reliability Auditing for Downstream LLM tasks in Psychiatry: LLM-Generated Hospitalization Risk Scores
Apr 23, 2026Shevya Panda, Shinjini Bose, Ananya JoshiPrompt Sensitivity LLM Auditing
Johns Hopkins University Baltimore, USA
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Apr 23, 2026Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7LLM Sycophancy LLM Auditing
Maritaca AI · JusBrasil
Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models
Apr 23, 2026Vipula Rawte, Ryan Rossi, Franck Dernoncourt +1Claim Attribution in Language Models LLM Answer Verification
Adobe · Adobe Research
Propensity Inference: Environmental Contributors to LLM Behaviour
Apr 22, 2026Olli Järviniemi, Oliver Makins, Jacob Merizian +2LLM Alignment LLM Auditing
UK AI Security Institute · Work done while at UK AISI.
Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways
Apr 22, 2026Guanjie Lin, Yinxin Wan, Shichao Pei +3LLM Auditing Black-Box Model Auditing
University of Massachusetts Boston · Boston, Massachusetts, USA · Arizona State University +2
Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs
Apr 22, 2026Krishiv Agarwal, Ramneet Kaur, Colin Samplawski +6Language Model Safety Evaluation LLM Auditing
University of Florida · SRI
Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Apr 22, 2026Melanie Subbiah, Haaris Mian, Nicholas Deas +3Gender Bias in Language Models LLM Auditing
Department of Computer Science, Columbia University · Department of Applied Physics and Applied Mathematics, Columbia University · Department of Psychology, Northwestern University
Detecting Data Contamination in Large Language Models
Apr 21, 2026Juliusz Janicki, Savvas Chamezopoulos, Evangelos Kanoulas +1Membership Inference Attacks LLM Auditing
University of Amsterdam · Elsevier
Auditing LLMs for Algorithmic Fairness in Casenote-Augmented Tabular Prediction
Apr 21, 2026Xiao Qi Lee, Ezinne Nwankwo, Angela ZhouLLM Auditing Algorithmic Fairness
Department of Computer Science San Jose State University · Department of Computer Science University of California, Berkeley · Department of Data Science and Operations University of Southern California
An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models
Apr 20, 2026Hanrui Luo, Shreyank N GowdaLanguage Model Safety Evaluation LLM Auditing
School of Computer Science, University of Nottingham, Nottingham, United Kingdom.
Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
Apr 20, 2026Ruixuan Liu, David Evans, Li XiongLLM Auditing Privacy Leakage in Language Models
Emory University · University of Virginia
Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols
Apr 20, 2026Fernando ReitichLLM Evaluation LLM Auditing
Imens, LLC · Weston, Florida, USA

Apr 29, 2026J/K move · Enter open · S save

TRUST: A Framework for Decentralized AI Service v.0.1

Yu-Chao Huang, Zhen Tan, Mohan Zhang, Pingzhi Li+2

UNITES Lab, University of North Carolina at Chapel Hill · Arizona State University · Columbia University

AI Agent Reliability AI Accountability AI Agent Auditing LLM Auditing Multi-Agent Systems AI Agent Governance+1

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) and Multi-Agent Systems (MAS) in high-stakes domains demand reliable verification, yet centralized approaches suffer four limitations: (1) Robustness, with single points of failure vulnerable to attacks and bias; (2) Scalability, as reasoning complexity creates bottlenecks; (3) Opacity, as hidden auditing erodes trust; and (4) Privacy, as exposed reasoning traces risk model theft. We introduce TRUST (Transparent, Robust, and Unified Services for Trustworthy AI), a decentralized framework with three innovations: (i) Hierarchical Directed Acyclic Graphs (HDAGs) that decompose Chain-of-Thought reasoning into five abstraction levels for parallel distributed auditing; (ii) the DAAN protocol, which projects multi-agent interactions into Causal Interaction Graphs (CIGs) for deterministic root-cause attribution; and (iii) a multi-tier consensus mechanism among computational checkers, LLM evaluators, and human experts with stake-weighted voting that guarantees correctness under 30% adversarial participation. We prove a Safety-Profitability Theorem ensuring honest auditors profit while malicious actors incur losses. All decisions are recorded on-chain, while privacy-by-design segmentation prevents reconstruction of proprietary logic. Across multiple LLMs and benchmarks, TRUST attains 72.4% accuracy (4-18% above baselines) and remains resilient against 20% corruption. DAAN reaches 70% root-cause attribution (vs. 54-63% for standard methods) with 60% token savings. Human studies validate the design (F1 = 0.89, Brier = 0.074). The framework supports (A1) decentralized auditing, (A2) tamper-proof leaderboards, (A3) trustless data annotation, and (A4) governed autonomous agents, pioneering decentralized AI auditing for safe, accountable deployment of reasoning-capable systems.