LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models
May 18, 2026Payal Chandak, Victoria Alkin, David Wu +11Moral Reasoning in Language Models Healthcare
Position: Generative Engine Optimization Creates Underexamined Risks, Governance Must Target Concentration, Disclosure, and Academic Blind Spots
May 18, 2026Yizhu Wen, Nan Zhang, Haohan Yuan +3Generative Engine Optimization LLM Auditing
School of Informatics, Computing, and Engineering, Indiana University Bloomington, Bloomington, USA · The Media School, Indiana University Bloomington, Bloomington, USA · School of Data Science, University of North Carolina at Charlotte, Charlotte, USA +1
Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries
May 17, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangMemorization in Language Models LLM Auditing
Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA
Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents
May 16, 2026Weicheng XueLLM Alignment LLM Auditing
Virginia Tech
GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models
May 16, 2026Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta +1LLM Evaluation LLM Auditing
Clark Atlanta University · United International University
AI-Mediated Communication Can Steer Collective Opinion
May 15, 2026Stratis Tsirtsis, Kai Rawal, Chris Russell +2Opinion Dynamics LLM Auditing
1Hasso Plattner Institute · 2Oxford Internet Institute, University of Oxford · 3Weizenbaum Institute
Fully Open Meditron: An Auditable Pipeline for Clinical LLMs
May 15, 2026Xavier Theimer-Lienhard, Mushtaha El-Amin, Fay Elhassan +5Healthcare LLM Auditing
LiGHT, EPFL
Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
May 15, 2026Parand A. Alamdari, Toryn Q. Klassen, Sheila A. McIlraithLLM Auditing Formal Verification
University of Toronto, Vector Institute, Canada
A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation
May 15, 2026Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein KarimiLLM Evaluation Learning to Rank
University of Waterloo, Waterloo, Ontario, Canada.
Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3LLM Auditing LLM Watermarking
University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta
Representation Without Reward: A JEPA Audit for LLM Fine-Tuning
May 14, 2026Biswa SenguptaLLM Auditing LLM Fine-Tuning
LLM Suite Team, JPMorgan Chase & Co.
Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks
May 14, 2026Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray +1LLM Safety Benchmarks Benchmark Auditing
Palo Alto Networks
AI Knows When It's Being Watched: Functional Strategic Action and Contextual Register Modulation in Large Language Models
May 14, 2026Vinicius Covas, Jorge Alberto Hidalgo ToledoLLM Auditing AI Agent Monitoring
Center for Applied Communication Research (CICA)
Mechanical Enforcement for LLM Governance:Evidence of Governance-Task Decoupling in Financial Decision Systems
May 14, 2026José Manuel de la Chica Rodríguez, Carlos Martí-GonzálezLLM Auditing LLM Safety
1Santander AI Lab, Grupo Santander
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research · Google · Google Deepmind
Auditing Agent Harness Safety
May 14, 2026Chengzhi Liu, Yichen Guo, Yepeng Liu +8AI Agent Auditing LLM Auditing
University of California, Santa Barbara · University of California, Berkeley · Microsoft Research +2
Identifying AI Web Scrapers Using Canary Tokens
May 13, 2026Steven Seiden, Triss Ren, Caroline Zhang +3LLM Auditing
Carnegie Mellon University CyLab Security & Privacy Institute Pittsburgh, PA, USA
Proof-Carrying Certificates for LLM Pipelines: A Trust-Boundary Architecture
May 13, 2026George KoomullilLLM Grounding LLM Auditing
Ascendr, Inc.
Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces
May 12, 2026Shixing Yu, Promit Ghosal, Kyra GanFeature Attribution LLM Auditing
Electrical and Computer Engineering · Cornell Tech · Department of Statistics +2
Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
May 12, 2026Andreas Maier, Jeta Sopa, Gozde Gul Sahin +2LLM Auditing LLM Prompting
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
CoT-Guard: Small Models for Strong Monitoring
May 12, 2026Nirav Diwan, Han Wang, Berkcan Kapusuzoglu +6LLM Auditing Adversarial Attacks on LLMs
†University of Illinois Urbana-Champaign · ♢Capital One
The Algorithmic Caricature: Auditing LLM-Generated Political Discourse Across Crisis Events
May 12, 2026Gunjan, Sidahmed Benabderrahmane, Talal RahwanSocial Media Analysis LLM Auditing
New York University (NYUAD), Division of Science, Computer Science Department
Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance
May 12, 2026Elias CalboreanuMulti-Agent LLM Systems AI Agent Auditing
Swift AI Lab, The Swift Group, LLC, Reston, VA 20190, USA
Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions
May 12, 2026Jagdish Tripathy, Marcus BuckmannLLM Auditing Implicit Bias
Bank of England
Behavioral Integrity Verification for AI Agent Skills
May 12, 2026Yuhao Wu, Tung-Ling Li, Hongliang LiuAgent Reliability AI Agent Auditing
Palo Alto Networks
When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel
May 12, 2026Wenkai Li, Fan Yang, Ananya Hazarika +2CoT Faithfulness LLM Auditing
Carnegie Mellon University · Fujitsu Research of America Inc.
OptArgus: A Multi-Agent System to Detect Hallucinations in LLM-based Optimization Modeling
May 12, 2026Zhong Li, Zihan Guo, Xiaohan Lu +5LLM Auditing Hallucination in Language Models
♣Great Bay University · ▲Peking University · ♡Jilin University +4
PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head
May 12, 2026Chieh-Yen Lin, Shao-Hua SunLLM Quantization LLM Evaluation
Appier AI Research · National Taiwan University
The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
May 12, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Auditing AI Safety Evaluation
Department of Computer Science, University of Oxford · Institute for Ethics in AI, University of Oxford · 3Responsible Technology Institute, University of Oxford
CPEMH: An Agentic Framework for Prompt-Driven Behavior Evaluation and Assurance in Foundation-Model Systems for Mental Health Screening
May 11, 2026Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar +1LLM Auditing Prompt Optimization
University of Waterloo · University of Waterloo Waterloo, ON, Canada

May 18, 2026, cs.AIJ/K move · Enter open · S save

What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models

Payal Chandak, Victoria Alkin, David Wu, Maya Dagan+10

Moral Reasoning in Language Models Healthcare LLM Auditing Clinical Decision-Making Clinical Language Model Evaluation

Open paper AskPDF arXiv

Medicine is inherently pluralistic. Principles such as autonomy, beneficence, nonmaleficence, and justice routinely conflict, and such ethical dilemmas often sharply divide reasonable physicians. Good clinical practice navigates these tensions in concert with each patient's values rather than imposing a single ethical stance. The ethical values that large language models bring to medical advice, however, have not been systematically examined. We present a framework for auditing value pluralism in medical AI, comprising a benchmark of clinician-verified dilemmas and an attribution method that recovers value priorities directly from decisions. The ecosystem of frontier models spans physician-level value heterogeneity, and models discuss competing values in their reasoning (Overton pluralism) before committing to a decision. However, individual model decisions are near-deterministic across repeated sampling and semantic variations, failing to reproduce the distributional pluralism of the physician panel. Across benchmark cases, these consistent decisions reflect committed, systematic value preferences. While most model priorities fall within the natural range of inter-physician variation, some significantly underweight patient autonomy. A single LLM deployed without regard for its value priorities could amplify those priorities at scale to every patient it serves. Without explicit efforts to balance ethical perspectives with one or multiple models, these tools risk replacing clinical pluralism with a deployment monoculture.

Momentum

Related topics

Latest papers 453