LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Jul 9, 2026Joongho Ahn, Moonsoo KimLLM Auditing LLM Guardrails
AI Leadership Research Center
Alignment Plausibility: A New Standard for Assuring AI in Healthcare
Jul 8, 2026Gwydion Williams, Sara Zannone, Bilal A MateenLLM Alignment Healthcare
PATH, United Kingdom · UCL, United Kingdom
Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
Jul 8, 2026Mihnea C. Moldoveanu, Joel A. C. BaumLLM Auditing Deception in Language Models
University of Toronto
Length Penalties Make Chain-of-Thought Less Monitorable
Jul 8, 2026Bryce LittleCoT Faithfulness Overthinking in Language Models
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
Jul 8, 2026Silvia SantanoLLM Auditing Reasoning Consistency in Language Models
Online Data Selection Is Implicit Alignment
Jul 8, 2026Aoxiong Zeng, Yuxin Yang, Xiangquan YangSupervised Fine-Tuning LLM Alignment
East China Normal University · Shanghai University
Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation
Jul 7, 2026Niels Potters, Theo HofmanLLM Evaluation LLM Auditing
Eindhoven University of Technology (TU/e), Dept. of Mechanical Engineering, Control Systems Technology section, Engineering Systems Design group, P.O.Box 513, 5600 MB Eindhoven, The Netherlands.
Faithfulness to Refusal: A Causal Audit of Neuron Selectors
Jul 6, 2026Ananth Eswar, Pratinav Seth, Utsav Avaiya +1Transformer Interpretability Feature Attribution
Lexsi Labs · Work done while at Lexsi Labs.
Curated retrieval versus open web search in public AI information services: a coverage-trust trade-off
Jul 6, 2026Hafsteinn Einarsson, Hafsteinn Birgir Einarsson, Jón Gunnar Ólafsson +1LLM Auditing RAG Evaluation
Faculty of Industrial Engineering, Mechanical Engineering and Computer Science, University of Iceland, Reykjavík, Iceland · Faculty of Political Science, University of Iceland, Reykjavík, Iceland · The Icelandic Web of Science, University of Iceland, Reykjavík, Iceland
Detecting Answer-Driven Reasoning in LLM-Based Educational Tutors via Truncated Chain-of-Thought Auditing
Jul 6, 2026Bonan Shen, Dingyan Shang, Youting Wang +1CoT Faithfulness Intelligent Tutoring Systems
Independent Researcher United States · Northeastern University Boston, MA, USA · Syracuse University Syracuse, NY, USA
Revealing Hidden Model Behaviors with Task-Specific Self-Reports
Jul 3, 2026Taras Kutsyk, Bartosz ZielińskiLLM Auditing Language Model Introspection
1Jagiellonian University, Faculty of Mathematics and Computer Science · 2Jagiellonian University, Doctoral School of Exact and Natural Sciences
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
Jul 3, 2026Kaley Brauer, Claudio Mayrink Verdun, Samuel MarksLLM Auditing LLM Interpretability
Harvard University · 4Anthropic
Behind the Refusal: Determining Guardrail Activation via Behavioral Monitoring
Jul 2, 2026William Hackett, Peter GarraghanLLM Auditing LLM Guardrails
Mindgard · Lancaster University
AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations
Jul 2, 2026Javier Irigoyen, Roberto Daza, Francisco Jurado +5LLM Safety Benchmarks LLM Auditing
BiometricsAI, Universidad Aut´onoma de Madrid (UAM), Spain · GHIA, Universidad Aut´onoma de Madrid (UAM), Spain · Universidad de Las Palmas de Gran Canaria (ULPGC), Spain
Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map
Jul 2, 2026Gabriel HurtadoLLM Auditing
Moonsong Labs
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
Jul 1, 2026Michael Saldivar, Ben SlivinskiLLM Answer Verification LLM Auditing
Independent Researchers
Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
Jul 1, 2026Shayan Talaei, Abhinav Chinta, Devvrit Khatri +3LLM Auditing Language Model Distillation
1Stanford University · 3Foundation AI–Cisco Systems Inc. · University of Texas at Austin
Auditing Forgetting in Limited Memory Language Models
Jul 1, 2026Arya Raeesi, Hanna RoedMemory-Augmented Language Models LLM Auditing
University of California, Berkeley, Berkeley, California, United States of America.
The Moving Target: A Longitudinal Audit of Trustworthiness Drift Across Twelve Checkpoints of Open-Source Chat LLMs
Jul 1, 2026Zhichao Fan, Yanhang Li, Zexin Zhuang +2LLM Safety Benchmarks LLM Evaluation
University of Illinois Urbana-Champaign · Northeastern University · Southern Methodist University +2
Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law
Jun 30, 2026Noah Scharrenberg, Chang SunLLM-as-a-Judge LLM Auditing
Department of Advanced Computing Sciences, Maastricht University, Maastricht, The Netherlands. · Contractuo, Eindhoven, The Netherlands.
Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbed
Jun 30, 2026Zhichao Fan, Zexin Zhuang, Yanhang LiMemorization in Language Models LLM Auditing
University of Illinois Urbana-Champaign · Southern Methodist University · Northeastern University
SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models
Jun 29, 2026Shuaimin Li, Liyang Fan, Zeyang Li +9LLM Evaluation LLM Auditing
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Shenzhen University · Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology +5
Fuzzing Large Language Models to Elicit Hidden Behaviours
Jun 28, 2026Mohammed Abu Baker, Lakshmi Babu-SaheerLLM Backdoor Attacks LLM Auditing
School of Computing and Information Science Anglia Ruskin University
LLMography: Transforming Human-AI Conversations into Traceability, Oversight, and Auditability Indicators
Jun 28, 2026Mohammed BousmahLLM Auditing Human Oversight of AI
LTI Laboratory, National School of Applied Sciences, Chouaib Doukkali University, El Jadida, Morocco
When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis
Jun 28, 2026Hoyoung Lee, Suhwan Park, Seunghan Lee +15LLM Compression LLM Auditing
UNIST · LinqAlpha · LG AI Research +9
The Contagion Tensor: A Framework for Measuring Output-Distribution Coupling in Multi-Agent LLM Systems -- and Auditing the Claims It Enables
Jun 27, 2026Zewen LiuMulti-Agent LLM Systems LLM Auditing
Capability Gates Are Not Authorization: Confused-Deputy Failures in LLM Agent Frameworks
Jun 27, 2026David Mellafe ZuvicAI Agent Auditing LLM Auditing
Independent Security Research Chile
Auditing Framing-Sensitive Behavioral Instability in Large Language Models for Mental Health Interactions
Jun 25, 2026Abla Bedoui, Ashley L. Greene, Mohammed CherkaouiPrompt Sensitivity LLM Auditing
School of computer science, digital engineering and AI, Long Island University, Brooklyn, NY, USA · Department of Psychology, Long Island University, Brooklyn, NY, USA
Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform
Jun 25, 2026Manar Alsaid, Chimdumebi Nebolisa, Faris AbbasLLM Evaluation Automated Program Repair
Department of Computer Science and Information Systems, East Texas A&M University, 2600 W. Neal Street, Commerce, 75428, Texas, United States · Department of Dual Enrollment and P-16 Programs, Texas Woman’s University, 3042026 Administration Drive, Denton, 76204, Texas, United States
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Jun 24, 2026Akshay Paruchuri, Sanmi Koyejo, Ehsan AdeliMultimodal Robustness LLM Auditing
Stanford University

Jul 9, 2026, cs.AIJ/K move · Enter open · S save

From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

Joongho Ahn, Moonsoo Kim

AI Leadership Research Center

LLM Auditing LLM Guardrails LLM Agent Harnesses LLM Agent Reliability

Open paper AskPDF arXiv

Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context. Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces. We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture: deterministic behavior moves into code, manifests, schemas, and validation artifacts around a replaceable composition boundary, while source-backed claims remain the authority for runtime answers. We instantiate it on a public-data slice of five Korean corporate groups (25 listed companies) and evaluate three research questions. (1) The harness preserves its source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts across the fixed validation scenarios; a fault-injection control confirms the validators flag deliberately broken contracts. (2) The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded. (3) The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely. A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility. The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.

Momentum

Related topics

Latest papers 453