LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Causal Stories from Sensor Traces: Auditing Epistemic Overreach in LLM-Generated Personal Sensing Explanations
May 9, 2026Shanshan Zhu, Han Zhang, J. Doris Chi +2LLM Grounding LLM Auditing
University of Illinois Urbana-Champaign, USA · University of Chicago, USA · Yale University, New Haven, CT, USA +2
SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization
May 8, 2026Houjun Liu, Lisa Einstein, John Yang +5LLM Auditing Prompt Optimization
Stanford University
LLM hallucinations in the wild: Large-scale evidence from non-existent citations
May 8, 2026Zhenyue Zhao, Yihe Wang, Toby Stuart +3LLM Auditing LLM Hallucination Detection
Department of Information Science, Cornell University, Ithaca, NY, USA · Department of Sociology, University of California Los Angeles, Los Angeles, CA, USA · Department of Computer Science and Technology, Tsinghua University, Beijing, China +1
MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditing
May 8, 2026Yinsheng Yao, Jiehao Tang, Zhaozhen Yang +1LLM Auditing Multi-Agent Debate
Tongji University Shanghai, China
LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs
May 7, 2026Chacha Chen, Matthew Jörke, Adam Goliński +4Belief Revision LLM Auditing
1Apple · 2Stanford University · †Work done while an intern at Apple. +1
Dataset Watermarking for Closed LLMs with Provable Detection
May 7, 2026Pengrun Huang, Kamalika Chaudhuri, Yu-Xiang WangDataset Watermarking LLM Auditing
Preprint. Under review.
When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
May 7, 2026Sushant Gautam, Finn Schwall, Annika Willoch Olstad +6LLM Safety Benchmarks LLM Evaluation
1Simula Metropolitan Center for Digital Engineering · 2Oslo Metropolitan University · 4Simula Research Laboratory +2
DataDignity: Training Data Attribution for Large Language Models
May 7, 2026Xiaomin Li, Andrzej Banburski-Fahey, Jaron LanierLLM Auditing Information Retrieval
Microsoft
Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models
May 6, 2026Quintin Pope, Ajay Hayagreeve Balaji, Jacques Thibodeau +1Language Model Generation Evaluation LLM Auditing
Oregon State University · Independent Researcher
Distilling Bayesian Belief States into Language Models for Auditable Negotiation
May 6, 2026Zongqi Cui, Baihan LinMulti-Agent Negotiation Opponent Modeling
Emory University · Icahn School of Medicine at Mount Sinai
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
May 6, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Evaluation LLM Alignment
Department of Computer Science, University of Oxford
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
May 5, 2026David Gringras, Misha SalahshoorLLM Evaluation LLM Auditing
Harvard University · AISST, Cambridge, MA
Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks
May 5, 2026Tianze Han, Beining Xu, Hanbo Zhang +1Mental Health Counseling LLM Sycophancy
Shenzhen MSU-BIT University, Shenzhen, China
AI-Generated Smells: An Analysis of Code and Architecture in LLM and Agent-Driven Development
May 4, 2026Yuecai Zhu, Nikolaos Tsantalis, Peter C. RigbySoftware Engineering LLM Auditing
Department of Computer Science and Software Engineering Concordia University Montréal, Québec, Canada
Model Routing as a Trust Problem: Route Receipts for Adaptive AI Systems
May 3, 2026Vincent SchmalbachAI Accountability Adaptive Model Routing
Independent Researcher
Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models
May 2, 2026William Guey, Wei Zhang, Pierrick Bougault +4Multilingual Language Model Evaluation LLM Auditing
Department of Industrial Engineering, Tsinghua University, Beijing, China · School of Social Sciences, Tsinghua University, Beijing, China · Department of Industrial Engineering, Federal University of Rio de Janeiro, Brazil
When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI
May 1, 2026Alfredo Madrid-García, Miguel RujasHealthcare LLM Auditing
Escuela Técnica Superior de Ingenieros de Telecomunicación Universidad Politécnica de Madrid Avenida Complutense, 30, Madrid, 28040, Spain
How Frontier LLMs Adapt to Neurodivergence Context: A Measurement Framework for Surface vs. Structural Change in System-Prompted Responses
Apr 30, 2026Ishan Gupta, Pavlo BuryiLLM Alignment Language Model Safety Evaluation
Harrisburg University of Science and Technology
Test Before You Deploy: Governing Updates in the LLM Supply Chain
Apr 30, 2026Mohd Sameen Chishti, Damilare Peter Oyinloye, Jingyue LiLLM Evaluation LLM Auditing
Department of Computer Science NTNU Trondheim, Norway
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
Apr 30, 2026Renwei Meng, Bowen Zhang, Jian Wang +4LLM Evaluation LLM Auditing
Anhui University, Hefei, China
TRUST: A Framework for Decentralized AI Service v.0.1
Apr 29, 2026Yu-Chao Huang, Zhen Tan, Mohan Zhang +3AI Agent Reliability AI Accountability
UNITES Lab, University of North Carolina at Chapel Hill · Arizona State University · Columbia University
Persuadability and LLMs as Legal Decision Tools
Apr 29, 2026Oisin Suttle, David LillisLLM Auditing LLM Decision-Making
School of Law and Criminology Maynooth University Maynooth, Ireland · School of Computer Science University College Dublin Dublin, Ireland
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
Apr 28, 2026David Hartmann, Manuel Tonneau, Angelie Kraft +7LLM Evaluation LLM Auditing
Weizenbaum Institute · TU Berlin · University of Oxford +9
Authority Inversion in LLM-Mediated Ubiquitous Systems: When Models Trust Users Over Sensors
Apr 28, 2026Long Zhang, Zi-bo Qin, Wei-neng ChenKnowledge Conflicts in Language Models LLM Auditing
School of Computer Science and Engineering, South China University of Technology, Guangzhou, China
Making AI-Assisted Grant Evaluation Auditable without Exposing the Model
Apr 28, 2026Kemal BicakciLLM Auditing Privacy-Preserving ML
Informatics Institute, Istanbul Technical University, Istanbul, Türkiye · Securify Information Technology and Security Training Consulting Inc., Ankara, Türkiye
Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM
Apr 28, 2026Vinith M. Suriyakumar, Ayush Sekhari, Lena Stempfle +5LLM Auditing Language Model Probing
MIT · BU · Thorn
Evaluating whether AI models would sabotage AI safety research
Apr 27, 2026Robert Kirk, Alexandra Souly, Kai Fronsdal +2Language Model Safety Evaluation LLM Auditing
UK AI Security Institute · UK AISI Research Affiliate
A Multi-Dimensional Audit of Politically Aligned Large Language Models
Apr 27, 2026Lisa Korver, Mohamed Mostagir, Sherief RedaLLM Auditing Political Bias in Language Models
Department of Computer Engineering Brown University Providence, RI 02906 · Ross School of Business University of Michigan Ann Arbor, MI 48109
One Size Fits None: Heuristic Collapse in LLM Investment Advice
Apr 26, 2026Jillian Ross, Andrew W. LoFinancial Services LLM Auditing
Computer Science and Artificial Intelligence Laboratory · Massachusetts Institute of Technology · Cambridge, MA, USA +1

May 9, 2026, cs.HCJ/K move · Enter open · S save

Causal Stories from Sensor Traces: Auditing Epistemic Overreach in LLM-Generated Personal Sensing Explanations

Shanshan Zhu, Han Zhang, J. Doris Chi, Subigya Nepal+1

University of Illinois Urbana-Champaign, USA · University of Chicago, USA · Yale University, New Haven, CT, USA+2

LLM Grounding LLM Auditing

Open paper AskPDF arXiv

LLMs are increasingly used to explain personal sensing data, translating traces of activity and mood into natural-language accounts of why an anomalous day may have occurred. However, such explanations can sound coherent and personally meaningful even when the underlying evidence is sparse or missing. We introduce epistemic overreach (EO) as a measure for cases where a generated explanation implies more than the available sensing evidence can justify. To audit how often and in what forms EO occurs, we obtained anomalous-day scenarios from three longitudinal sensing datasets of college students: StudentLife, GLOBEM, and CollegeExperience. Across activity, sleep, and affect anomalies, we generated 14,922 explanations using three LLM families -- Llama, Qwen, and GPT -- under two prompting conditions: one minimally constrained prompt and another prompt explicitly instructing models to bound claims to the data. For each scenario, we varied the amount of behavioral evidence available to the model to examine whether more evidence reduces EO. We evaluated each explanation using a structured rubric, decomposing EO into the dimensions of unsupported causal attribution, unacknowledged data gaps, overconfident language, temporal inconsistency, and diagnostic inference. We find that LLMs routinely attribute anomalous days to causes without sufficient support from the data, and that this pattern replicates across datasets, anomaly types, and model families. Further, providing richer context does not reliably reduce EO; bounded prompting helps but does not eliminate it. These findings suggest that evidential grounding should be a first-order evaluation criterion for LLM-generated personal sensing explanations, alongside fluency and plausibility. We argue that personal sensing explanations require evidential discipline: systems must distinguish what is observed, what is inferred, and what remains unknown.

Momentum

Related topics

Latest papers 453