LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
May 15, 2026Parand A. Alamdari, Toryn Q. Klassen, Sheila A. McIlraithLLM Auditing Formal Verification
University of Toronto, Vector Institute, Canada
A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation
May 15, 2026Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein KarimiLLM Evaluation Learning to Rank
University of Waterloo, Waterloo, Ontario, Canada.
Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3LLM Auditing LLM Watermarking
University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta
Representation Without Reward: A JEPA Audit for LLM Fine-Tuning
May 14, 2026Biswa SenguptaLLM Auditing LLM Fine-Tuning
LLM Suite Team, JPMorgan Chase & Co.
Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks
May 14, 2026Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray +1LLM Safety Benchmarks Benchmark Auditing
Palo Alto Networks
AI Knows When It's Being Watched: Functional Strategic Action and Contextual Register Modulation in Large Language Models
May 14, 2026Vinicius Covas, Jorge Alberto Hidalgo ToledoLLM Auditing AI Agent Monitoring
Center for Applied Communication Research (CICA)
Mechanical Enforcement for LLM Governance:Evidence of Governance-Task Decoupling in Financial Decision Systems
May 14, 2026José Manuel de la Chica Rodríguez, Carlos Martí-GonzálezLLM Auditing LLM Safety
1Santander AI Lab, Grupo Santander
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research · Google · Google Deepmind
Auditing Agent Harness Safety
May 14, 2026Chengzhi Liu, Yichen Guo, Yepeng Liu +8AI Agent Auditing LLM Auditing
University of California, Santa Barbara · University of California, Berkeley · Microsoft Research +2
Identifying AI Web Scrapers Using Canary Tokens
May 13, 2026Steven Seiden, Triss Ren, Caroline Zhang +3LLM Auditing
Carnegie Mellon University CyLab Security & Privacy Institute Pittsburgh, PA, USA
Proof-Carrying Certificates for LLM Pipelines: A Trust-Boundary Architecture
May 13, 2026George KoomullilLLM Grounding LLM Auditing
Ascendr, Inc.
Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces
May 12, 2026Shixing Yu, Promit Ghosal, Kyra GanFeature Attribution LLM Auditing
Electrical and Computer Engineering · Cornell Tech · Department of Statistics +2
Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
May 12, 2026Andreas Maier, Jeta Sopa, Gozde Gul Sahin +2LLM Auditing LLM Prompting
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
CoT-Guard: Small Models for Strong Monitoring
May 12, 2026Nirav Diwan, Han Wang, Berkcan Kapusuzoglu +6LLM Auditing Adversarial Attacks on LLMs
†University of Illinois Urbana-Champaign · ♢Capital One
The Algorithmic Caricature: Auditing LLM-Generated Political Discourse Across Crisis Events
May 12, 2026Gunjan, Sidahmed Benabderrahmane, Talal RahwanSocial Media Analysis LLM Auditing
New York University (NYUAD), Division of Science, Computer Science Department
Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance
May 12, 2026Elias CalboreanuMulti-Agent LLM Systems AI Agent Auditing
Swift AI Lab, The Swift Group, LLC, Reston, VA 20190, USA
Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions
May 12, 2026Jagdish Tripathy, Marcus BuckmannLLM Auditing Implicit Bias
Bank of England
Behavioral Integrity Verification for AI Agent Skills
May 12, 2026Yuhao Wu, Tung-Ling Li, Hongliang LiuAgent Reliability AI Agent Auditing
Palo Alto Networks
When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel
May 12, 2026Wenkai Li, Fan Yang, Ananya Hazarika +2CoT Faithfulness LLM Auditing
Carnegie Mellon University · Fujitsu Research of America Inc.
OptArgus: A Multi-Agent System to Detect Hallucinations in LLM-based Optimization Modeling
May 12, 2026Zhong Li, Zihan Guo, Xiaohan Lu +5LLM Auditing Hallucination in Language Models
♣Great Bay University · ▲Peking University · ♡Jilin University +4
PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head
May 12, 2026Chieh-Yen Lin, Shao-Hua SunLLM Quantization LLM Evaluation
Appier AI Research · National Taiwan University
The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
May 12, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Auditing AI Safety Evaluation
Department of Computer Science, University of Oxford · Institute for Ethics in AI, University of Oxford · 3Responsible Technology Institute, University of Oxford
CPEMH: An Agentic Framework for Prompt-Driven Behavior Evaluation and Assurance in Foundation-Model Systems for Mental Health Screening
May 11, 2026Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar +1LLM Auditing Prompt Optimization
University of Waterloo · University of Waterloo Waterloo, ON, Canada
Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
May 11, 2026Hsing Wen Lin, Zong-Fu SieLLM Auditing Scientific Reproducibility
Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA · Michigan Institute for Data and AI in Society, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Taiwan
Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
May 11, 2026Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada +7LLM Auditing Multimodal Large Language Models
Department of Electrical & Computer Engineering · Department of Computer and Information Science and Engineering · Department of Clinical and Health Psychology +1
Acceptance Cards:A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims
May 11, 2026Phongsakon Mark Konrad, Toygar Tanyel, Serkan AyvazFine-Tuning Language Model Safety Evaluation
Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark · ProMake, Newark, DE, USA
Pseudo-Deliberation in Language Models: When Reasoning Fails to Align Values and Actions
May 11, 2026Sushrita Rakshit, Hanwen Zhang, Hua ShenLLM Alignment LLM Auditing
New York University · New York City, NY 10012 · New York University Shanghai
CHAINTRIX: A multi-pipeline LLM-augmented framework for automated smart-contract security auditing
May 10, 2026Gabriela Dobrita, Simona-Vasilica Oprea, Adela BaraLLM Auditing Software Vulnerability Detection
Bucharest University of Economic Studies, Department of Economic Informatics and Cybernetics, no.6 Piața Romană, Bucharest, 010374, Romania
BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence
May 9, 2026Jialing Gan, Junhao Dong, Songze LiPrompt Sensitivity LLM Auditing
Southeast University, China · Nanyang Technological University, Singapore
Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups
May 9, 2026Gautam VeldandaLLM Auditing Algorithmic Fairness
Independent Researcher

May 15, 2026, cs.AIJ/K move · Enter open · S save

Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems

Parand A. Alamdari, Toryn Q. Klassen, Sheila A. McIlraith

University of Toronto, Vector Institute, Canada

LLM Auditing Formal Verification AI Agent Monitoring AI Governance AI Safety Linear Temporal Logic+1

Open paper AskPDF arXiv

We examine one particular dimension of AI governance: how to monitor and audit AI-enabled products and services throughout the AI development lifecycle, from pre-deployment testing to post-deployment auditing. Combining principles from formal methods with SoTA machine learning, we propose techniques that enable AI-enabled product and service developers, as well as third party AI developers and evaluators, to perform offline auditing and online (runtime) monitoring of product-specific (temporally extended) behavioral constraints such as safety constraints, norms, rules and regulations with respect to black-box advanced AI systems, notably LLMs. We further provide practical techniques for predictive monitoring, such as sampling-based methods, and we introduce intervening monitors that act at runtime to preempt and potentially mitigate predicted violations. Experimental results show that by exploiting the formal syntax and semantics of Linear Temporal Logic (LTL), our proposed auditing and monitoring techniques are superior to LLM baseline methods in detecting violations of temporally extended behavioral constraints; with our approach, even small-model labelers match or exceed frontier LLM judges. Our predictive and intervening monitors significantly reduce the violation rates of LLM-based agents while largely preserving task performance. We further show through controlled experiments that LLMs' temporal reasoning shows a pronounced degradation in accuracy with increasing event distance, number of constraints, and number of propositions.

Momentum

Related topics

Latest papers 453