LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Provably Auditable and Safe LLM Agents from Human-Authored Ontologies
Jun 3, 2026Aaron SterlingAI Agent Auditing LLM Auditing
Thistleseeds
Thinking Through Signs: PEEL as a Semiotic Scaffolding for Epistemically Accountable AI-Enabled Research
Jun 2, 2026Clarisse de Souza, Gabriel Barbosa, Simone Diniz Junqueira Barbosa +3LLM Evaluation AI Accountability
PUC-Rio, PUC-Behring Institute of Artificial Intelligence, Brazil · PUC-Rio, Brazil
Covert Influence Between Language Models
Jun 2, 2026Avidan Shah, Jay Chooi, Jinghua Ou +1LLM Auditing
MATS · New York University · 1MATS 2New York University +5
Auditing LLM-Governed Social Robots with Culture-Specific Moral Gradients
Jun 2, 2026Carmen Ng, Gjergji KasneciMultilingual Language Model Evaluation Moral Reasoning in Language Models
Technical University of Munich, Munich, Germany
FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences
Jun 2, 2026Gurvan Richardeau, Gohar Dashyan, Erwan Le Merrer +1LLM Auditing Language Model Fingerprinting
PEReN (Paris, France) · Université de Rennes, Inria, CNRS/IRISA (Rennes, France) · LAAS, CNRS, (Toulouse, France)
The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertBenchmark Auditing LLM Auditing
NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland
Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation
Jun 2, 2026Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan +1LLM Auditing Software Vulnerability Detection
Universitas Gadjah Mada, Indonesia · Monash University, Indonesia
A Practice Auditing Framework for Large Language Model Use: Collective Empiricism, Pseudo-Rational Cognition, and Governance of AI-Generated Content
Jun 2, 2026Yang Zhao, Yingshuo Li, Zeyu ZhangLLM Auditing AI Governance
School of Information and Electronics, Beijing Institute of Technology
Auditing Asset-Specific Preferences in Financial Large Language Models: Evidence from Bitcoin Representations and Portfolio Allocation
Jun 1, 2026Wenbin WuLLM Auditing LLM Decision-Making
Cambridge Centre for Alternative Finance, University of Cambridge
Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims
Jun 1, 2026Siddhesh Milind Pawar, Sarah Masud, Haneul Yoo +2LLM Auditing Pragmatic Reasoning in Language Models
♠University of Copenhagen · ♣KAIST
Do Gender Cues Affect LLM Value Trade-offs? Evidence from a Controlled Decision Benchmark
Jun 1, 2026Yangyang Liu, Dong Yu, Pengyuan LiuGender Bias in Language Models Moral Reasoning in Language Models
Beijing Language and Culture University
The Ghost Couple: Correlated LLM Name Priors and Their Haunting of the Web and Academic Publishing
Jun 1, 2026Michał Brzozowski, Neo Christopher ChungLLM Auditing Hallucination in Language Models
Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland
CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models
Jun 1, 2026Swapnil ParekhLLM Auditing Data Contamination in Language Models
Intuit Inc.
From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication
May 31, 2026Máté Metzger, Nadnapang Phophichit, Hansa DhammahasoLLM Auditing Machine Translation
Independent Researcher, Hungary · Nibbana Meditation Centre, Thailand
A Registry-Bound LLM Pipeline for Evidence-Grounded Trait Extraction across Tropical Plants, Aquatic Species, and Exotic Pets
May 31, 2026Jeff WangData Provenance LLM Auditing
NEXLY LLC, United States
Which Institutional Frameworks Do Chatbots Assume? Auditing Jurisdictional Defaults in Multilingual LLMs
May 29, 2026Zhizhi Wang, Harini SureshMultilingual Language Model Evaluation LLM Auditing
Brown University
Bit-Exact AI Inference Verification Without Performance Tradeoffs
May 29, 2026Naci CankayaNeural Network Verification LLM Inference
Machine Intelligence Research Institute
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
May 29, 2026Tom Lucas, Alessio Buscemi, Alfredo Capozucca +2LLM Evaluation AI Accountability
Luxembourg Institute of Science and Technology (LIST), Luxembourg, University of Luxembourg, Luxembourg, and Scriptor Artis, France · Luxembourg Institute of Science and Technology (LIST), Luxembourg · University of Luxembourg, Luxembourg +1
Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language Models Multilingual Language Model Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
May 29, 2026Swastik Roy, Rajkumar Pujari, Tharindu Kumarage +5LLM-as-a-Judge LLM Auditing
Amazon AGI
CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs
May 28, 2026Ryan FaheyKV Caching LLM Auditing
Khoury College of Computer Sciences Northeastern University Boston, MA, USA
Auditing LLM Benchmarks with Item Response Theory
May 28, 2026Sander Land, Daniel M. BikelBenchmark Auditing LLM Auditing
Writer, Inc.
LLMSurgeon: Diagnosing Data Mixture of Large Language Models
May 28, 2026Yaxin Luo, Jiacheng Cui, Xiaohan Zhao +5LLM Auditing
1VILA Lab, MBZUAI · 2UCL
Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage
May 28, 2026Shahinul Hoque, Jinghuai Zhang, Jinyuan Sun +1LLM Auditing Black-Box Model Auditing
University of Tennessee, Knoxville · University of California, Los Angeles
NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models
May 28, 2026Anany KotawalaData Leakage LLM Evaluation
Princeton University, Princeton, NJ, USA.
Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs
May 28, 2026Zhibo Zhang, Yuxi Li, Zhen Ouyang +2Mixture-of-Experts Language Models LLM Auditing
Huazhong University of Science and Technology, Wuhan, China · Nanyang Technological University, Singapore
KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing
May 28, 2026Yijia Fang, Yiqing Feng, Bingyu Li +1LLM Auditing Language Model Fingerprinting
Beihang University China · Xidian University China · HKUST Hong Kong SAR, China
Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles
May 28, 2026Drishti Goel, Agam Goyal, Veda Duddu +8Healthcare LLM Auditing
University of Illinois Urbana-Champaign · University of Massachusetts Amherst · OSF HealthCare +1
Inferring the Size of Large Language Models From Popular Text Memorization
May 28, 2026Ivica NikolicParameter Estimation Memorization in Language Models
National University of Singapore
When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
May 27, 2026Aisha Najera, Alvin Moon, Vedant Srinivasan +1LLM Evaluation Annotator Disagreement
AI Lab, Princeton University · Engineering and Applied Sciences, RAND Corporation · Science, Technology, and International Affairs, Georgetown University

Jun 3, 2026, cs.LOJ/K move · Enter open · S save

Provably Auditable and Safe LLM Agents from Human-Authored Ontologies

Aaron Sterling

Thistleseeds

AI Agent Auditing LLM Auditing LLM Agent Safety LLM Agent Reliability

Open paper AskPDF arXiv

We introduce the LLM agent architecture Agentic Redux, intended for use with nontrivial problem domains that require linear auditability. Using the typed lambda calculus, we prove that, run on appropriate domains, Agentic Redux executions are semantically guaranteed to be correct, with all decisions recorded in an append-only ledger. We present two production-grade appropriate domains, in healthcare billing compliance, and security vulnerability disclosure. Working code for Agentic Redux run on both domains is available in a supporting code repository. We also introduce Ontology-First Agent Design, a methodology for creation of agent frameworks on a problem domain, in which a human expert ontologizes the problem domain with Basic Formal Ontology, and then assigns an LLM to derive roles that agents and humans-in-the-loop can fill, in order to work the problems in the domain.

Momentum

Related topics

Latest papers 453