LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation
Sep 27, 2026Gowthamkumar NandakishoreLLM Evaluation Selective Prediction
PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations
Sep 24, 2026Luciano MaldonadoLLM Auditing Privacy Leakage in Language Models
West Virginia University, Morgantown, WV, USA.
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Sep 24, 2026Dipankar SarkarLLM Evaluation LLM Auditing
Skelf Research
Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt Sensitivity LLM-as-a-Judge
KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1
Claim-Gated Source-Risk Auditing for Generative Search
Sep 24, 2026Kainan Zhou, Chuhong Xu, Gangzhen Qian +1Data Provenance LLM Auditing
Google LLC Mountain View, USA · Sony Corporate of America San Jose, USA · Intuit Inc Mountain View, USA
How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
Sep 23, 2026Chen Yang, Xianyang Zhang, Jun ChenLLM Evaluation Selective Inference
Department of Statistics, Texas A&M University · Division of Computational Biology, Department of Quantitative Health Sciences, Mayo Clinic
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Sep 23, 2026Makar Ulesov, Vladislav Smirnov, Omar Ibrahim +1Benchmark Auditing LLM Auditing
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · School of Computing and Information, University of Pittsburgh
How Much Were You Told? Measuring External Information in Peer Reviews
Sep 23, 2026Matthieu Dubois, Pablo Piantanida, François YvonAI-Generated Text Detection LLM Auditing
Sorbonne Université, CNRS, ISIR · International Laboratory on Learning Systems (ILLS) Quebec AI Institute (MILA) CNRS, CentraleSupélec, Université Paris-Saclay
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing
Sep 21, 2026Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet +2Multilingual Language Model Evaluation LLM Auditing
Saarland University · German Research Center for Artificial Intelligence (DFKI) · Barcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain
Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)
Sep 21, 2026Amir Rafe, Subasish DasLLM Auditing Language Model Calibration
Ingram School of Engineering, Texas State University, San Marcos, TX, USA
Fingerprinting Multimodal Large Language Models
Sep 17, 2026Chao Huang, Meng Tong, Kejiang ChenLLM Auditing Language Model Fingerprinting
University of Science and Technology of China Anhui Province Key Laboratory of Digital Security Hefei, Anhui, China
Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression
Sep 17, 2026Guangzhe ZhangAgent Memory Counterfactual Evaluation
Independent AI Researcher
Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
Sep 17, 2026Hang Xiao, Chuhong Xu, Kainan Zhou +2Formal Hardware Verification LLM Auditing
Fortinet, Inc. Sunnyvale, CA, USA · Sony Corporate of America San Jose, CA, USA · Google LLC Mountain View, CA, USA
AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment
Sep 16, 2026Sai Sri Pushpa Jampani, Kshitij Mishra, Asif EkbalLLM Alignment LLM Auditing
Indian Institute of Technology Patna, Bihar, India
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward Hacking LLM Evaluation
Core contributors · Equal senior contribution
Faithful yet Collusive: Why Chain-of-Thought Monitoring Cannot Detect Collusion in LLM Pricing Agents under Oligopolistic Competition
Sep 16, 2026Dohun Lee, Hyunwoo ParkCoT Faithfulness LLM Auditing
Graduate School of Data Science, Seoul National University
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
Sep 16, 2026Zihan Chen, Di Zhu, Lei Zheng +1LLM-as-a-Judge LLM Auditing
Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA · Stony Brook University Stony Brook, NY, USA
OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network Verification LLM Auditing
Gensyn
Easy to Catch a Liar, Hard to Clear an Honest One: Language Models Diagnosing a Corrupted Reward Channel from a Verified Record
Sep 15, 2026Arman Nik KhahLLM Auditing Deception in Language Models
The University of Texas at Dallas
Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Sep 15, 2026Yuan Pu, Yewon Chang, Furong Jia +4LLM Evaluation LLM Auditing
Department of Computer Science, Duke University, 308 Research Drive, Durham, 27708, NC. · Department of Medicine, Duke University, 40 Duke Medicine Circle, Durham, 27710, NC. · Geriatrics and Extended Care, Durham VA Health System, 508 Fulton Street, Durham, 27705, NC. +2
Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures
Sep 14, 2026Yuhang WangLLM Auditing LLM Agent Security
Fudan University
Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML
Sep 14, 2026Sidi Chang, Peiying ZhuLLM-as-a-Judge LLM Auditing
Blossom AI San Francisco, CA, USA
PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews
Sep 14, 2026Miguel Zabaleta, Baihan LinLLM Auditing AI-Assisted Scientific Research
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
A decision-basis contract for auditable LLM-assisted medical billing verification: deterministic rules, verbatim evidence, and fail-closed abstention
Sep 14, 2026Jan Hölter, Kevin Geis, Benjamin Raab +1LLM Answer Verification LLM Auditing
Competence Center Artificial Intelligence, Aschaffenburg University of Applied Sciences, Würzburger Straße 45, Aschaffenburg, 63743, Bavaria, Germany.
Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model
Sep 12, 2026Hyojung HanLLM Alignment LLM Auditing
ThakiCloud
LLMs as Post-hoc Auditors of Physiological Plausibility in Symbolic Regression: A Clinician-Evaluated Case Study
Sep 12, 2026Jorge López-Varela, J. Ignacio Hidalgo, José-Manuel Muñoz +6LLM Auditing Explainability Evaluation
Computer Architecture Department, Universidad Complutense de Madrid, Spain. · Bioinspired Intelligence Ltd., Pozuelo de Alarc´on, Spain · Universidad Aut´onoma de Baja California, M´exico +7
OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows
Sep 10, 2026Aayam Bansal, Keertan BalajiAI for Science AI Agent Auditing
Auditable Emergency Triage for Maternal and Newborn Care in India
Sep 10, 2026Shobhit Jagga, Aman Dalmia, Niharika Priyadarshini +7Healthcare LLM Auditing
Noora Health · The Agency Fund
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
Sep 9, 2026Arman Nik KhahMemorization in Language Models LLM Auditing
The University of Texas at Dallas

Sep 27, 2026, cs.AIJ/K move · Enter open · S save

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Gowthamkumar Nandakishore

LLM Evaluation Selective Prediction LLM Auditing Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor that answers typed choice/noul/score questions over workflow state, is uniformly under-confident. The signed confidence-accuracy gap is

-0.214

, every occupied reliability bin's accuracy exceeds its confidence, and that sign uniformity collapses every binned ECE variant to the same value,

0.214

. The card frames the risk as over-confidence; the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails. A single disjointly fitted temperature (

T=0.469

, sharpening) removes most of the miscalibration (held-out ECE

0.204

0.037

) and outperforms the shipped per-option-count table. The frozen selection rule instead chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 is not supported. Re-running the released checkpoint on its full official test split reproduces the card's headline accuracy (

0.767

vs.

0.766

). The retrospective E1 reproduction preceded the analysis freeze; E2-E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at

q=0.05

(two descoped). The frozen gate beats random escalation but misses its 10% accepted-set error target on both tracks, an exploratory out-of-distribution probe finds no zero-shot transfer (accuracy

0.617

), and every score measures agreement with a synthetic teacher whose self-agreement ceiling (

0.735

) the specialist exceeds. Per-decision predictions, run manifests, and the frozen preregistration are in the ancillary files. The author has no affiliation with the model's publisher, the dataset's publisher, or TypeSafe.

Momentum

Related topics

Latest papers 453