LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops
Jun 12, 2026Muhammad Osama, Maheera Amjad, Zartasha Mustansar +2Multi-Agent LLM Systems LLM Hallucination Mitigation
Data Science and Machine Learning Lab, SINES, NUST · CEME, NUST
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
Detecting Functional Memorization in Code Language Models
Jun 11, 2026Matthieu Meeus, Anil Ramakrishna, Matthew Grange +2Memorization in Language Models LLM Auditing
Meta · Imperial College London
Prefill Awareness in Large Language Models
Jun 10, 2026Andy Wang, Parv Mahajan, David Demitri Africa +3Language Model Safety Evaluation LLM Auditing
UK AI Security Institute
PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
Jun 10, 2026Pengfei He, Lesly Miculicich, Vishesh Sharma +5AI Agent Auditing LLM Auditing
Google Cloud AI Research · Google · Michigan State University
Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs
Jun 10, 2026Sanjay Adhikesaven, Haoxiang Sun, Sewon MinLLM Auditing
University of California, Berkeley · Allen Institute for AI
CANONIC: Governance Is Compilation
Jun 10, 2026Dexter HadleyData Provenance LLM Auditing
Founder and Chair, CANONIC Foundation · Director of AI, American Board of Precision Medicine
The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models
Jun 9, 2026Hakan MehmetcikMultilingual Language Model Evaluation LLM Auditing
Visiting Researcher, Kellogg Institute for International Studies, University of Notre Dame, IN, USA · Prof of International Relations, at the Faculty of Political Science, Marmara University, Istanbul, Türkiye
Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models
Jun 9, 2026Prajakta Kini, Avinash Reddy, Souradip Chakraborty +4LLM Alignment Language Model Safety Evaluation
University of Colorado Boulder · University of Central Florida · University of Maryland College Park +1
The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignment
Jun 9, 2026Filippo Tonini, Federico Torrielli, Anton Danholt Lautrup +3AI Agent Auditing LLM Auditing
University of Southern Denmark Odense, Denmark · University of Turin Turin, Italy · University of Southern Denmark Vejle, Denmark
When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models
Jun 9, 2026Sai Kartheek Reddy Kasu, Nils Lukas, Samuele PoppiCoT Faithfulness LLM Alignment
Independent Researcher · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Jun 9, 2026Leonard Engmann, Christian Medeiros Adriano, Holger GieseMixture-of-Experts Pruning LLM Auditing
Hasso Plattner Institute, University of Potsdam, Germany.
Advancing the State-of-the-Art in Empirical Privacy Auditing
Jun 9, 2026Nicole Mitchell, Galen Andrew, Arun Ganesh +2Data Leakage Privacy Auditing
Google Research San Francisco, CA · Google Research Seattle, WA
Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture
Jun 8, 2026Yoojin Nam, Jinhoon Jeong, Namkug KimLLM Answer Verification LLM Auditing
Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea · Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea · Aperivue, Incheon, Republic of Korea +2
Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models
Jun 8, 2026Bartłomiej Marek, Lorenzo Rossi, Vincent Hanke +4Privacy-Preserving Language Models Membership Inference Attacks
CISPA Helmholtz Center for Information Security
Auditable Graph-Guided Root Cause Analysis for Kubernetes Incidents
Jun 7, 2026Anastasiia Kuvshinova, Seungmin JinLLM Auditing Graph Reasoning
HSE University Moscow, Russia
Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs
Jun 7, 2026Rahul Gorijavolu, Kaushik Madapati, Pritika Vig +7LLM Evaluation LLM Sycophancy
MIT Critical Data, Massachusetts Institute of Technology, Cambridge, MA, USA · School of Medicine, Johns Hopkins University, Baltimore, MD, USA · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA +10
Auditing Proprietary Alignment in Large Language Models: A Comparative Framework Without a Ground-Truth Standard
Jun 7, 2026Alireza Arbabi, Florian KerschbaumLLM Alignment LLM Auditing
Department of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada
AeroSpectra Sentinel: An Auditable LLM Prompt-Chaining Decision-Support Workflow for Acute Asthma Risk Assessment from Respiratory Sounds and Clinical Signals
Jun 6, 2026Aueaphum AueawatthanaphisutLLM Auditing Clinical Decision Support
School of Information, Computer, and Communication Technology Sirindhorn International Institute of Technology, Thammasat University Pathum Thani, Thailand
Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms
Jun 6, 2026Hyunjin Cho, Youngji Roh, Jaehyung KimLLM Auditing Mechanistic Interpretability
Yonsei University
When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
Jun 6, 2026Enyi Jiang, Anders Gjølbye, Yibo Jacky Zhang +1Language Model Safety Evaluation LLM Auditing
1Stanford University · University of Illinois Urbana-Champaign · 3Technical University of Denmark
MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models
Jun 6, 2026Kaixin Lan, Mu You, Tao Fang +3Membership Inference Attacks LLM Auditing
Natural Language Processing & Portuguese-Chinese Machine Translation (NLP2CT) Laboratory, University of Macau, Macau SAR, China · Institute of International Language Services Studies, Macau Millennium College, Macau SAR, China · Innovation Research & Development, BoardWare Information System Limited
Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Models
Jun 5, 2026Sanjay Kariyappa, G. Edward SuhLLM Auditing Instruction Following
NVIDIA
Auditing Training Data in Domain-adapted LLMs: LoRA-MINT
Jun 5, 2026Gonzalo Mancera, Daniel DeAlcala, Aythami Morales +3Membership Inference Attacks LLM Auditing
BiometricsAI, Universidad Autonoma de Madrid, Spain · Department of Mathematics, Universidad de Las Palmas de Gran Canaria, Spain · Computer Engineering Department, Universidad Autonoma de Madrid, Spain
The Geography of Algorithmic Judgment: LLM Intermediaries, Place Identity, and Racial Steering in Housing Search
Jun 4, 2026Hana Samad, Trung Lam, Christoph Mügge-Durum +1LLM Auditing Social Bias in Language Models
Responsible AI Lab · Chief AI Officer and Head of the Responsible AI Lab, National Fair Housing Alliance, Washington DC, USA
Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
Jun 4, 2026Xinnong Zhang, Wanting Shan, Hanjia Lyu +2Counterfactual Evaluation LLM Auditing
1Fudan University · 3Shanghai Innovation Institute · University of Rochester
LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs
Jun 4, 2026Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke PoechData Leakage Memorization in Language Models
University of Southern Denmark
ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs
Jun 4, 2026Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber +1Tool-Augmented Language Model Agents LLM Auditing
SAP Labs
CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model
Jun 4, 2026Zeyang Yue, Chenfei Yan, Feifei Zhao +5Language Model Safety Evaluation LLM Auditing
School of Artificial Intelligence, Beihang University · 2BrainCog AI Lab, CASIA · School of Artificial Intelligence, UCAS +4
How Far Did They Go? The Persuasive Tactics of Covert LLM Agents in a Discontinued Field Experiment
Jun 3, 2026Kokil Jaidka, Saifuddin AhmedLLM Auditing LLM Agent Evaluation
Department of Communications and New Media, & Centre for Trusted Internet & Community, National University of Singapore, Singapore · Wee Kim Wee school of Communication and Information, Nanyang Technological University, Singapore

Jun 12, 2026, cs.LGJ/K move · Enter open · S save

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

Muhammad Osama, Maheera Amjad, Zartasha Mustansar, Arslan Shaukat+1

Data Science and Machine Learning Lab, SINES, NUST · CEME, NUST

Multi-Agent LLM Systems LLM Hallucination Mitigation LLM Auditing Clinical QA Medical LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed in healthcare settings, yet their tendency to hallucinate poses risks when clinical decisions are involved. This study examine whether LLMs recommend recently banned or withdrawn pharmaceuticals when answering clinical questions and tests an agent-based method for reducing such errors. We developed a five-agent "Trust but Verify" system using a single LLM backbone. To measure regulatory knowledge obsolescence, we created an adversarial dataset of 103 clinical MCQs where historically correct answers now refer to banned substances. This scale ensures statistical significance across various therapeutic classes. We evaluated three open-access model families (GPT-OSS, Llama-3, Falcon-3) under vanilla and agentic conditions. Performance was measured via pointwise score, label accuracy, Hallucination Error Rate (HER), and Component Fidelity (CF) score. We also observed clinical safety regression in proprietary models. In default configurations, all models showed high hallucination rates, consistently selecting banned drugs that matched training data patterns. Our proposed agentic architecture reduced HER by approximately 53% across models. Pointwise scores shifted from -0.25 (unsafe recommendation) toward 0.0 (appropriate refusal). The safety audit intercepted dangerous outputs even when models' parametric knowledge favored the banned substance. The proposed multi-agent framework offers a model-agnostic method for enforcing regulatory compliance that prioritizes patient safety over fluent text generation. Our work demonstrates a practical approach for deploying autonomous AI systems in safety-critical healthcare settings. It shows how real-time regulatory data can be integrated into LLM pipelines to support clinical decision-making.

Momentum

Related topics

Latest papers 453