LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

CAPRI: Contract-Aware Proof Repair for Isabelle
Aug 13, 2026Jim Woodcock, Gabriel Leite, Augusto Sampaio +1LLM Auditing Interactive Theorem Proving
Southwest University, China · Aarhus University, Denmark · University of York, UK +2
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM Evaluation LLM Auditing
School of Statistics and Data Science, Southwestern University of Finance and Economics · School of Statistics and Data Science, Southwestern University of Finance and Economics Chengdu, China
Follow the Norm: Accounting for Fine-Tuning and Prompt Effects on Model Rationales
Aug 13, 2026Long Hoang Nguyen, Brice Valentin Kok-Shun, Guangyu Du +1Moral Reasoning in Language Models Normative Conformity in Language Models
1Technical University of Munich · University of Auckland
Causal Structure is Inducible but Functionally Decoupled: The Routing/Readout Boundary of a Typed Mechanism Library
Aug 12, 2026Xining XunCausal Representation Learning LLM Auditing
Tsingjiao Information Science (Beijing) Co., Ltd.
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study
Aug 12, 2026Simone MungariLLM Auditing Political Bias in Language Models
Revelis s.r.l.
Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning
Aug 11, 2026Zirui Song, Huaxing Liu, Xiang Wang +8LLM Auditing Machine Unlearning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · AMAP, Alibaba Group · Peking University
Most biomedical publications show signs of LLM-assisted writing
Aug 11, 2026Lena Holzwarth, Rita González-Márquez, Dmitry KobakAI-Generated Text Detection LLM Auditing
1Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Mathematics, Computer Science, and Statistics, Ghent University, Belgium · 3VIB Center for AI and Computational Biology, VIB, Ghent, Belgium
Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?
Aug 11, 2026Qingjie Zhang, Xingzhang Ren, Zixuan Chen +6Language Model Pretraining LLM Auditing
1Tsinghua University · 2Qwen Team, Alibaba Group · 3Alibaba Group
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
Aug 11, 2026Qingjie Zhang, Ziqi Tang, Jie Zhang +7LLM Auditing Data Contamination
Tsinghua University · SiliconProspect AI · Nanyang Technological University +1
Activation Probes Surface Code-Security Signals that the Model's Output Misses
Aug 10, 2026Ivan WiryadiLLM Auditing Software Vulnerability Detection
Independent.
Fair on the Surface? Benchmarking Hidden-Output Fairness Gaps in LLM Recommenders
Aug 8, 2026Chan Aristella Lu, Arya Fayyazi, Junhao Zhang +6LLM Auditing Language Model Bias Evaluation
University of Georgia · University of Southern California · Carnegie Mellon University +1
When Is a Steerable Concept Representation Real? Measurement Confounds in a Cross-Family Audit of Neuroscience Parallels in LLMs
Aug 8, 2026Yuqi Wu, Shengming Zhao, Jie ChenLanguage Model Steering LLM Auditing
College of Biomedical Engineering, Fudan University
TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance
Aug 8, 2026Yuqi Wu, Shengming Zhao, Jie ChenLLM Auditing Language Model Fingerprinting
College of Biomedical Engineering, Fudan University
Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness
Aug 7, 2026Srijith RavikumarLLM Auditing Language Model Calibration
Amazon.com LLC Seattle, Washington, USA
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
$A^2E$ : An End-to-End Agent Auditing Engine
Aug 7, 2026Haoning Wang, Mingxun Zhang, Chenyue Yu +4Tool-Use Evaluation AI Agent Auditing
Shanghai Artificial Intelligence Laboratory
Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints
Aug 7, 2026Paul-Peter ArslanLLM Auditing Language Model Bias Evaluation
Institute for Future Technologies
Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Models
Aug 7, 2026Zhuoheng Huang, Mukesh SinghLLM Auditing Sparse Mixture-of-Experts
Independent Researcher
OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents
Aug 6, 2026Xinying Cai, Minghao Guo, Jiahe Liu +7LLM Auditing LLM Agent Evaluation
Rutgers University · Technical University of Denmark · New Jersey Institute of Technology +4
Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts
Aug 6, 2026Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio +1LLM Sycophancy LLM Auditing
Université Côte d’Azur · Technical University Munich
HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards
Aug 6, 2026Zhuowen Liu, Bohan Cui, YinShang Guo +2Reward Hacking LLM Auditing
The Chinese University of Hong Kong · Fudan University · Nanjing University +2
Runtime Observability for Heterogeneous Attention Memory
Aug 6, 2026Fanzhe Wei, Li Liu, Ziyang Wang +1KV Caching LLM Auditing
Metask Lab
Subliminal Learning is Non-Semantic Distillation
Aug 6, 2026Ethan Hadley, Eren GultepeLLM Auditing Subliminal Learning
Department of Computer Science, Southern Illinois University Edwardsville, Edwardsville, IL, USA.
Item Response Theory for AI Safety
Aug 5, 2026Joshua Fonseca Rivera, Neil Shah, David Demitri Africa +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent · 1Independent · UK AI Security Institute +1
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
Aug 5, 2026Zhixiang Liang, Yifei Liu, Yidan Huang +5Agent Failure Analysis AI Agent Reliability
University of Illinois Urbana-Champaign · 2Joy Future Academy, JD
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
Aug 5, 2026Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto +3LLM Auditing Language Model Introspection
Institute of Science Tokyo · 2ZOZO Research · 3Stanford University +2
LatentGuard: Efficient and Inspectable Latent Reasoning for LLM Safeguards
Aug 4, 2026Zhinan Liu, Jie Li, Mingyu Kang +1LLM Auditing LLM Guardrails
1Xiamen University · University of Science and Technology of China
Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains
Aug 4, 2026Yanchao Li, Wanhao Liu, Jiaqing Xie +4LLM Evaluation LLM Auditing
Nanjing University · Shanghai Artificial Intelligence Laboratory · North University of China
Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
Aug 4, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation LLM Auditing
Department of Statistics and Data Science Northwestern University
Evading Chain-of-Thought Monitoring Through Model Poisoning
Aug 3, 2026Giorgio Severi, Shujaat Mirza, Blake Bullwinkel +1LLM Backdoor Attacks LLM Auditing
Microsoft

Aug 13, 2026, cs.SEJ/K move · Enter open · S save

CAPRI: Contract-Aware Proof Repair for Isabelle

Jim Woodcock, Gabriel Leite, Augusto Sampaio, Ran Wei

Southwest University, China · Aarhus University, Denmark · University of York, UK+2

LLM Auditing Interactive Theorem Proving LLM Reliability

Open paper AskPDF arXiv

We address the use of large language models (LLMs) to help discover Isabelle proofs. An Isabelle build establishes that the submitted theory is accepted, but not that an LLM changed only what the developer authorised. We present CAPRI, a contract-aware repair workflow in which Isabelle checks the proof and an independent checker enforces a machine-readable edit contract. Prompts, proposals, candidate repositories, diagnostics, verdicts, and hashes are retained for audit. We evaluate five workflows on twelve failed proofs from four developments, with three replicates per task and condition, giving 180 runs and 138 valid repairs. Of 144 terminal candidates accepted by Isabelle, six had modified protected text; all arose in iterative workflows that could edit a complete theory. A proof-body-only interface produced 29/36 valid repairs and no contract violations, compared with 31/36 for the corresponding full-theory workflow. One-shot repair produced 22/36, while a later prospectively frozen iterative workflow produced 32/36; these figures compare complete workflows rather than individual mechanisms. A separate post hoc OpenRouter campaign found no improvement in the designated Luna comparisons. A Sol configuration with matched demonstrations produced 33/36 repairs, compared with 29/36 in the frozen OpenAI Responses condition, but the difference was not statistically significant in a one-sided exact McNemar test (

p=0.0625

Momentum

Related topics

Latest papers 453