LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Auditing Provenance Sensitivity in LLM Agent Action Selection
Jul 23, 2026Junchi LiaoAI Agent Auditing LLM Auditing
MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing
Jul 22, 2026Yu Liu, Zhiwei Yang, Diandian Guo +7LLM Auditing Materials Science
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China +2
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior
Jul 21, 2026Meena Jagadeesan, Tatsunori Hashimoto, Jon KleinbergAI-Generated Text Detection LLM Auditing
1Stanford University · University of Pennsylvania · 2Cornell University
HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks
Jul 21, 2026Haozhe JiaLLM Auditing Temporal Data Leakage
University College Dublin
Semi-Automated Detection of Gaps in LLM Security Knowledge
Jul 20, 2026Shufan Chai, Liangliang Sun, Jessica StaddonLLM Auditing LLMs for Cybersecurity
Northeastern University Oakland, CA, USA
PEARL: Auditable Repair for Scientific Reasoning Graph Extraction
Jul 20, 2026Bohan Su, Pengze Li, Yuchen Lu +1LLM Reasoning with Graphs LLM Auditing
School of Computer Science, Wuhan University · Artificial Intelligence Innovation and Incubation, Fudan University · Department of Computer Science, Faculty of Science, University of Bath +1
Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families
Jul 19, 2026Aleksander FafułaLLM Auditing LLM Refusal Behavior
independent researcher • funduszai.pl
Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat
Jul 19, 2026Pilsung KangLLM Auditing
Department of Software Science, Dankook University Yongin, South Korea
Auditing Retrieval-Augmented LLM Hypotheses for Longitudinal Cell Painting Morphology
Jul 17, 2026Gilchan Park, Guang Zhao, Byung-Jun Yoon +1Retrieval-Augmented Generation LLM Auditing
Brookhaven National Laboratory Upton, New York, USA
The Information Shadow: Measuring Structural Limits on What Language Models Can Learn
Jul 17, 2026Priyansh Srivastava, Romit ChatterjeeLLM Auditing LLM Training
Sirena Ai · Independent Researcher
Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies
Jul 16, 2026Filippos Vlahos, Guillaume Bied, Tijl De BieLLM Auditing Political Bias in Language Models
Ghent University, BE
How Artificial Intelligence LLM Engines Shape the Global Conflict Information Environment
Jul 15, 2026Jason MiklianGenerative Engine Optimization LLM Auditing
Centre for Global Sustainability, University of Oslo
Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
Jul 15, 2026Joonyong Park, David M. Chan, Yuki Saito +1Audio-Language Model Evaluation LLM Auditing
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Berkeley AI Research (BAIR), University of California, Berkeley, CA, USA
The Refusal Residue: When Probes Catch Alignment Faking and When They Don't
Jul 15, 2026Aman MehtaLLM Auditing Deception in Language Models
Snowflake AI Research
Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games
Jul 12, 2026Yuan Gao, Jiangyi Yang, Yao Zhao +1Imperfect-Information Games Social Deduction Games
University of Melbourne
ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm
Jul 11, 2026Kefan Song, Yanjun QiAI Agent Auditing LLM Auditing
University of Virginia, Charlottesville, VA, USA
One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions
Jul 11, 2026Tomas BrucknerLLM Answer Verification LLM Auditing
Faculty of Informatics and Statistics, Prague University of Economics and Business, Prague, Czech Republic
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
Jul 11, 2026Hong-In Won, Jinseok Jang, Hyoseop KimLLM Evaluation Moral Reasoning in Language Models
KITECH, Incheon, Republic of Korea · National Research Council of Science and Technology (NST), Sejong, Republic of Korea · KITECH, Daegu, Republic of Korea
Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
Jul 11, 2026Hironao NakamuraCoT Faithfulness LLM Grounding
Independent Researcher
Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
Jul 10, 2026Izumi Takahara, Teruyasu MizoguchiAI Agents for Scientific Discovery LLM Auditing
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan 153-8505
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Jul 9, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangLLM-as-a-Judge LLM Auditing
Dyson School of Design Engineering, Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom
Who Analyses the Analyser? Self-Validating LLM Hazard Analysis with Constitutional Meta-STPA
Jul 9, 2026Samuel Tetteh, Udip Shrestha, Joshua R. Waite +1LLM Auditing LLM Safety
Mechanical Engineering, Iowa State University · Electrical and Computer Engineering, Iowa State University · Translational AI Center, Iowa State University
From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Jul 9, 2026Joongho Ahn, Moonsoo KimLLM Auditing LLM Guardrails
AI Leadership Research Center
Alignment Plausibility: A New Standard for Assuring AI in Healthcare
Jul 8, 2026Gwydion Williams, Sara Zannone, Bilal A MateenLLM Alignment Healthcare
PATH, United Kingdom · UCL, United Kingdom
Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
Jul 8, 2026Mihnea C. Moldoveanu, Joel A. C. BaumLLM Auditing Deception in Language Models
University of Toronto
Length Penalties Make Chain-of-Thought Less Monitorable
Jul 8, 2026Bryce LittleCoT Faithfulness Overthinking in Language Models
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
Jul 8, 2026Silvia SantanoLLM Auditing Reasoning Consistency in Language Models
Online Data Selection Is Implicit Alignment
Jul 8, 2026Aoxiong Zeng, Yuxin Yang, Xiangquan YangSupervised Fine-Tuning LLM Alignment
East China Normal University · Shanghai University
Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation
Jul 7, 2026Niels Potters, Theo HofmanLLM Evaluation LLM Auditing
Eindhoven University of Technology (TU/e), Dept. of Mechanical Engineering, Control Systems Technology section, Engineering Systems Design group, P.O.Box 513, 5600 MB Eindhoven, The Netherlands.

Jul 23, 2026, cs.AIJ/K move · Enter open · S save

Auditing Provenance Sensitivity in LLM Agent Action Selection

Junchi Liao

AI Agent Auditing LLM Auditing LLM Agents LLM Agent Safety

Open paper AskPDF arXiv

LLM agents choose tools and arguments from context that mixes user requests, tool outputs, retrieved records, memory, and untrusted text. Evidence can be relevant without being authorized to determine a decision, so a correct action need not be grounded only in permitted evidence. We introduce a target-specific authorization audit that labels context factors separately for each tool and argument target. Its primary test holds the task, proposition, position, and policy fixed while changing only the proposition's source authority. We then test behavior when valid evidence is weakened and use context-subset interactions as a secondary localization diagnostic. Across 450 controlled next-action tasks and multiple open-weight LLM families, trusted and untrusted variants produce different actions in 5.4 percent of competing cases versus 1.7 percent of supporting cases. Under controlled degradation, unauthorized competition is retained in a full-correct, mixed-error, clean-correct pattern in 2.4 percent of comparisons, with a 95 percent confidence interval from 2.1 to 3.0 percent. These are controlled stress-set rates, not deployment prevalence. The models respond to textual source-authority cues, but this does not prevent untrusted evidence from influencing their actions.

Momentum

Related topics

Latest papers 453