LLM Auditing
LLM: Large Language Model
Overview
Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.
Latest papers
723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.
Discovering Implicit Large Language Model Alignment Objectives
Automatically Finding Reward Model Biases
Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System
NEST: Nascent Encoded Steganographic Thoughts
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs
Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge
Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models
Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models
Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines
Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas
Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection?
White-Box Sensitivity Auditing with Steering Vectors
Pro-AI Bias in Large Language Models
A Scalable Entity-Based Framework for Auditing Bias in Large Language Models
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS
Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness
Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5
The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
marks'' using invisible Unicode characters organized into (cue'', reply'') pairs. During an audit, prompts containing only cue'' fragments are issued to trigger regurgitation of the corresponding ``reply'', indicating document usage. To control false positives, we compare against held-out counterfactual marks and apply a ranking test, yielding a verifiable bound on the false positive rate. Empirically, we obtain a true positive rate of 96.7% at 0% false positive rate and reply regurgitation rates exceeding 28% per document with only 40 (4%) watermarked documents. The approach is minimally invasive, scalable across many sources, robust to standard processing pipelines, and achieves high detection power even when marked data is a small fraction of the fine-tuning corpus.