LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
Jul 25, 2026Jeff OttersonAutomated Software Testing LLM Auditing
W. P. Carey School of Business, Arizona State University
Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science
Jul 24, 2026Davide Scarso, Hugo Noronha de Almeida, Joaquim PinaLLM Evaluation LLM Auditing
CIUHCT - Interuniversity Centre for the History of Science and Technology · MARE - Marine and Environmental Sciences Centre, ARNET - Aquatic Research Network Associate Laboratory
Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines
Jul 24, 2026Santhiya RajanLLM Auditing LLM Agent Reliability
Multiverse Computing
Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
Jul 24, 2026Yunting Song, Matthew Watson, Peter Grabowski +1LLM Safety Benchmarks LLM Auditing
Google Mountain View, CA 94043
Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
Jul 23, 2026Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda +1LLM Auditing Deception in Language Models
dreadnode, Principal AI Research Engineer · dreadnode, StaffAI Security Researcher · dreadnode, Distinguished Engineer +1
Code Monitor Red Teaming for Public-Test-Passing Code
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenLLM Auditing Code Generation Evaluation
University of Electronic Science and Technology of China Chengdu, China
Auditing Evidence Use in Medical LLM Diagnosis
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenLLM Auditing Medical Diagnosis
University of Electronic Science and Technology of China Chengdu, China
Auditing Provenance Sensitivity in LLM Agent Action Selection
Jul 23, 2026Junchi LiaoAI Agent Auditing LLM Auditing
MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing
Jul 22, 2026Yu Liu, Zhiwei Yang, Diandian Guo +7LLM Auditing Materials Science
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China +2
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior
Jul 21, 2026Meena Jagadeesan, Tatsunori Hashimoto, Jon KleinbergAI-Generated Text Detection LLM Auditing
1Stanford University · University of Pennsylvania · 2Cornell University
HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks
Jul 21, 2026Haozhe JiaLLM Auditing Temporal Data Leakage
University College Dublin
Semi-Automated Detection of Gaps in LLM Security Knowledge
Jul 20, 2026Shufan Chai, Liangliang Sun, Jessica StaddonLLM Auditing LLMs for Cybersecurity
Northeastern University Oakland, CA, USA
PEARL: Auditable Repair for Scientific Reasoning Graph Extraction
Jul 20, 2026Bohan Su, Pengze Li, Yuchen Lu +1LLM Reasoning with Graphs LLM Auditing
School of Computer Science, Wuhan University · Artificial Intelligence Innovation and Incubation, Fudan University · Department of Computer Science, Faculty of Science, University of Bath +1
Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families
Jul 19, 2026Aleksander FafułaLLM Auditing LLM Refusal Behavior
independent researcher • funduszai.pl
Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat
Jul 19, 2026Pilsung KangLLM Auditing
Department of Software Science, Dankook University Yongin, South Korea
Auditing Retrieval-Augmented LLM Hypotheses for Longitudinal Cell Painting Morphology
Jul 17, 2026Gilchan Park, Guang Zhao, Byung-Jun Yoon +1Retrieval-Augmented Generation LLM Auditing
Brookhaven National Laboratory Upton, New York, USA
The Information Shadow: Measuring Structural Limits on What Language Models Can Learn
Jul 17, 2026Priyansh Srivastava, Romit ChatterjeeLLM Auditing LLM Training
Sirena Ai · Independent Researcher
Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies
Jul 16, 2026Filippos Vlahos, Guillaume Bied, Tijl De BieLLM Auditing Political Bias in Language Models
Ghent University, BE
How Artificial Intelligence LLM Engines Shape the Global Conflict Information Environment
Jul 15, 2026Jason MiklianGenerative Engine Optimization LLM Auditing
Centre for Global Sustainability, University of Oslo
Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
Jul 15, 2026Joonyong Park, David M. Chan, Yuki Saito +1Audio-Language Model Evaluation LLM Auditing
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Berkeley AI Research (BAIR), University of California, Berkeley, CA, USA
The Refusal Residue: When Probes Catch Alignment Faking and When They Don't
Jul 15, 2026Aman MehtaLLM Auditing Deception in Language Models
Snowflake AI Research
Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games
Jul 12, 2026Yuan Gao, Jiangyi Yang, Yao Zhao +1Imperfect-Information Games Social Deduction Games
University of Melbourne
ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm
Jul 11, 2026Kefan Song, Yanjun QiAI Agent Auditing LLM Auditing
University of Virginia, Charlottesville, VA, USA
One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions
Jul 11, 2026Tomas BrucknerLLM Answer Verification LLM Auditing
Faculty of Informatics and Statistics, Prague University of Economics and Business, Prague, Czech Republic
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
Jul 11, 2026Hong-In Won, Jinseok Jang, Hyoseop KimLLM Evaluation Moral Reasoning in Language Models
KITECH, Incheon, Republic of Korea · National Research Council of Science and Technology (NST), Sejong, Republic of Korea · KITECH, Daegu, Republic of Korea
Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
Jul 11, 2026Hironao NakamuraCoT Faithfulness LLM Grounding
Independent Researcher
Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
Jul 10, 2026Izumi Takahara, Teruyasu MizoguchiAI Agents for Scientific Discovery LLM Auditing
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan 153-8505
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Jul 9, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangLLM-as-a-Judge LLM Auditing
Dyson School of Design Engineering, Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom
Who Analyses the Analyser? Self-Validating LLM Hazard Analysis with Constitutional Meta-STPA
Jul 9, 2026Samuel Tetteh, Udip Shrestha, Joshua R. Waite +1LLM Auditing LLM Safety
Mechanical Engineering, Iowa State University · Electrical and Computer Engineering, Iowa State University · Translational AI Center, Iowa State University

Jul 25, 2026, cs.SEJ/K move · Enter open · S save

Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

Jeff Otterson

W. P. Carey School of Business, Arizona State University

Automated Software Testing LLM Auditing Mutation Testing Code Generation Evaluation Automated Test Generation

Open paper AskPDF arXiv

Large language models increasingly write both code and the tests meant to check it; coverage records what ran, not what was verified. We study an adversarial test-hardening loop under a mechanical oracle: a Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically, so no model judges another's output. In Experiment 1, on five Python subjects (one same-lineage-loop cell could not be scored), the loop killed 105 mutants that one-shot generation missed and lost none, and the cross-lineage-Critic question returned a pre-declared null. The central finding was an autopsy: an earlier analysis reported a cross-lineage effect at p = 9.5e-66 that was an instrument artifact, an output cap silently truncating the verbose model, caught only by adversarial review of the completed analysis. Review then found a further confound, each arm resampling its own initial suite; Experiment 2 removes it. Under a pre-registered frozen-shared-round-0 design (five replicates on each of four subjects, seeds committed in advance), same-lineage Critic rounds killed 78% of the survivors the frozen initial suite left standing (mean incremental kill rate 0.783, 95% cluster-bootstrap interval [0.592, 0.935]), a within-replicate causal estimate; the cross-provider configuration showed a positive pilot difference (rate gap 0.178, 95% interval [0.039, 0.347]; magnitude dominated by a single replicate) at 5.5x lower arm cost. This compares two named model-provider-harness configurations, not an isolated lineage effect: part of the gap is one configuration's receipted operational failures, including truncation recurrences, now detected and scored rather than laundered. Cross-model comparisons can inherit the asymmetries of the harness that runs them. We release both protocols, all receipts, and the analysis code.

Momentum

Related topics

Latest papers 453