LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

When AI reviews science: Can we trust the referee?
Apr 26, 2026Jialiang Wang, Yuchen Liu, Hang Xu +7LLM Auditing Adversarial Attacks on LLMs
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, School of Computer Science and Engeering, Southeast University, Nanjing 210000, China · These authors contributed equally +2
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation
Apr 26, 2026Lichen Li, Hengguang Zhou, Yijun Liang +2RL for Code Generation Reward Hacking
Peking University · University of California, Los Angeles · University of Maryland +2
VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs
Apr 25, 2026Yurui Xiang, Xingyi Mao, Rui Sheng +7Healthcare LLM Auditing
Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation
Apr 24, 2026Matteo Pelossi, Rita Sevastjanova, Thilo Spinner +1LLM Auditing Social Bias in Language Models
ETH Zurich
Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning
Apr 24, 2026Chaoran Chen, Dayu Yuan, Peter KairouzLLM Auditing RL Fine-Tuning
Google
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Apr 23, 2026Harsh Kumar, Rahul Maity, Tanmay Joshi +4Language Model Pretraining LLM Backdoor Attacks
1Manipal University Jaipur · 2NIT, Karnataka · 3Pragya Lab, BITS Pilani Goa, India +2
Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations
Apr 23, 2026Nalin Poungpeth, Nicholas Clark, Tanu MitraLLM Auditing Human-AI Interaction
Department of Cognitive Science Northwestern University Evanston, IL 60201, USA · Information School University of Washington Seattle, WA 98195 USA
PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning
Apr 23, 2026Xiaoyi Chen, Haoyuan Wang, Siyuan Tang +4LLM Auditing Privacy Leakage in Language Models
Indiana University Bloomington · Independent Researcher · Michigan State University +2
Reliability Auditing for Downstream LLM tasks in Psychiatry: LLM-Generated Hospitalization Risk Scores
Apr 23, 2026Shevya Panda, Shinjini Bose, Ananya JoshiPrompt Sensitivity LLM Auditing
Johns Hopkins University Baltimore, USA
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Apr 23, 2026Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7LLM Sycophancy LLM Auditing
Maritaca AI · JusBrasil
Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models
Apr 23, 2026Vipula Rawte, Ryan Rossi, Franck Dernoncourt +1LLM Answer Verification LLM Auditing
Adobe · Adobe Research
Propensity Inference: Environmental Contributors to LLM Behaviour
Apr 22, 2026Olli Järviniemi, Oliver Makins, Jacob Merizian +2LLM Alignment LLM Auditing
UK AI Security Institute · Work done while at UK AISI.
Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways
Apr 22, 2026Guanjie Lin, Yinxin Wan, Shichao Pei +3LLM Auditing Black-Box Model Auditing
University of Massachusetts Boston · Boston, Massachusetts, USA · Arizona State University +2
Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs
Apr 22, 2026Krishiv Agarwal, Ramneet Kaur, Colin Samplawski +6Language Model Safety Evaluation LLM Auditing
University of Florida · SRI
Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Apr 22, 2026Melanie Subbiah, Haaris Mian, Nicholas Deas +3Gender Bias in Language Models LLM Auditing
Department of Computer Science, Columbia University · Department of Applied Physics and Applied Mathematics, Columbia University · Department of Psychology, Northwestern University
Detecting Data Contamination in Large Language Models
Apr 21, 2026Juliusz Janicki, Savvas Chamezopoulos, Evangelos Kanoulas +1Membership Inference Attacks LLM Auditing
University of Amsterdam · Elsevier
Auditing LLMs for Algorithmic Fairness in Casenote-Augmented Tabular Prediction
Apr 21, 2026Xiao Qi Lee, Ezinne Nwankwo, Angela ZhouLLM Auditing Algorithmic Fairness
Department of Computer Science San Jose State University · Department of Computer Science University of California, Berkeley · Department of Data Science and Operations University of Southern California
An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models
Apr 20, 2026Hanrui Luo, Shreyank N GowdaLanguage Model Safety Evaluation LLM Auditing
School of Computer Science, University of Nottingham, Nottingham, United Kingdom.
Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
Apr 20, 2026Ruixuan Liu, David Evans, Li XiongLLM Auditing Privacy Leakage in Language Models
Emory University · University of Virginia
Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols
Apr 20, 2026Fernando ReitichLLM Evaluation LLM Auditing
Imens, LLC · Weston, Florida, USA
Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs
Apr 20, 2026Ziyang LiuLLM Auditing
Independent Researcher
ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding
Apr 20, 2026Zirui Wang, Yusen Hou, Shaofeng Liang +4LLM Auditing Language Model Fingerprinting
The Hong Kong University of Science and Technology (Guangzhou) · Deep Interdisciplinary Intelligence Lab (DI2Lab)
AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code
Apr 19, 2026William M. ParrisLLM Auditing Static Code Analysis
BDB Labs | Jurisprudential AI Governance Initiative
PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations
Apr 19, 2026Patrick KeoughHealthcare LLM Auditing
Independent Researcher Bremerton, Washington, USA
The Granularity Gap: A Multi-Dimensional Longitudinal Audit of Sycophancy in Gemini Models
Apr 19, 2026Patrick KeoughLLM Sycophancy LLM Alignment
Independent Researcher Bremerton, WA, USA
Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation
Apr 18, 2026Michelle Star, Andrew Aquilina, Yu-Ru LinLLM Evaluation Emotional Support Conversation
School of Computing and Information, University of Pittsburgh
One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them
Apr 18, 2026Ali Holmov, Paul Youssef, Nandi Schoots +1Knowledge Editing Transformer Interpretability
Technical University of Munich · Marburg University · University of Oxford
DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training
Apr 18, 2026Ziwen Pan, Zihan Liang, Jad Kabbara +1LLM Auditing Social Bias in Language Models
Emory University · MIT
Introspection Adapters: Training LLMs to Report Their Learned Behaviors
Apr 18, 2026Keshav Shenoy, Li Yang, Abhay Sheshadri +4LLM Auditing Language Model Introspection
Anthropic Fellows · Ashwood Centre on AI Science and Policy, University of Cambridge · Anthropic
Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases
Apr 17, 2026Eric Gan, Aryan Bhatt, Buck Shlegeris +2LLM Auditing
Redwood Research

Apr 26, 2026, cs.AIJ/K move · Enter open · S save

When AI reviews science: Can we trust the referee?

Jialiang Wang, Yuchen Liu, Hang Xu, Kaichun Hu+6

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, School of Computer Science and Engeering, Southeast University, Nanjing 210000, China · These authors contributed equally+2

LLM Auditing Adversarial Attacks on LLMs Prompt Injection Attacks Automated Peer Review

Open paper AskPDF arXiv

The volume of scientific submissions continues to climb, outpacing the capacity of qualified human referees and stretching editorial timelines. At the same time, modern large language models (LLMs) offer impressive capabilities in summarization, fact checking, and literature triage, making the integration of AI into peer review increasingly attractive -- and, in practice, unavoidable. Yet early deployments and informal adoption have exposed acute failure modes. Recent incidents have revealed that hidden prompt injections embedded in manuscripts can steer LLM-generated reviews toward unjustifiably positive judgments. Complementary studies have also demonstrated brittleness to adversarial phrasing, authority and length biases, and hallucinated claims. These episodes raise a central question for scholarly communication: when AI reviews science, can we trust the AI referee? This paper provides a security- and reliability-centered analysis of AI peer review. We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level. We instantiate this taxonomy with four treatment-control probes on a stratified set of ICLR 2025 submissions, using two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores. Together, this taxonomy and experimental audit provide an evidence-based baseline for assessing and tracking the reliability of AI peer review and highlight concrete failure points to guide targeted, testable mitigations.

Momentum

Related topics

Latest papers 453