LLM Auditing

LLM: Large Language Model

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

CardsList

723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.

  1. Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

    May 6, 2025Cléa Chataigner, Rebecca Ma, Prakhar Ganesh +4LLM AuditingLanguage Model Robustness

  2. Beyond Public Access in LLM Pre-Training Data

    Apr 24, 2025Sruly Rosenblat, Tim O'Reilly, Ilan StraussPrivacy Leakage in Language ModelsLLM Auditing