LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

GANDR: Claim Auditing for Verifiable Legal Answer Generation
Sep 9, 2026Chen Qian, Yimeng Wang, Yu Chen +2LLM Answer Verification Legal Citation Verification
William & Mary · Anytime AI
Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States
Sep 9, 2026Marek Jeliński, Jan Dubiński, Maciej Chrabaszcz +1LLM Auditing Social Bias in Language Models
NASK - National Research Institute, Poland · Warsaw University of Technology, Poland · Gda´nsk University of Technology, Poland
What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores
Sep 8, 2026Dana Paquin, Riddhiman JainLLM Evaluation LLM Auditing
Stanford University
Deposon: An Auditable, Conservation-Guaranteed, Game-Theoretically Tested Scattering Layer over LLM Reasoning Paths
Sep 8, 2026Qihao YuanLLM Reasoning with Graphs LLM Auditing
School of Chemistry and Life Resources, Renmin University of China
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Sep 8, 2026Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah +1LLM Evaluation LLM Auditing
Qatar Computing Research Institute, HBKU
Audit Without Verification: When LLM Accountability Layers Relay Rather Than Check
Sep 7, 2026Paul-Peter ArslanAgent Failure Analysis AI Accountability
Institute For Future Technologies
A Removal Based Approach to Improve LLM Faithfulness at Test-Time
Sep 7, 2026Qinglan Luo, S M A Nahian, John Guttag +2Faithfulness of Language Model Explanations LLM Auditing
MIT CSAIL · MGH, HMS
Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
Sep 4, 2026Urja Pawar, Rajitha Ramanayake, Nabeel Kemal +6Faithfulness of Language Model Explanations LLM Auditing
BNY
Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit
Sep 4, 2026Dmitrij ŻatuchinLLM Auditing LLM Information Extraction
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia · Rankfor.AI, Tallinn, Estonia
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
Sep 3, 2026Dmitrij ŻatuchinLLM Auditing LLM Reliability
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis
Sep 2, 2026Ahmed Asaad, Amr Mohamed, Yang Zhang +1LLM Evaluation LLM Auditing
Durham University Business School · MBZUAI & Ecole Polytechnique · Ecole Polytechnique +1
Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity
Sep 1, 2026Sinjini Banerjee, Tim Marrinan, Anand D. SarwateSelective Prediction Ensemble Learning
Department of Electrical and Computer Engineering, Rutgers University · Pacific Northwest National Lab
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Sep 1, 2026Muhammed Saeed, Simon RazniewskiLLM Evaluation LLM Auditing
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
Sep 1, 2026Prakhar Gupta, Vaibhav GuptaRL for Language Models LLM Grounding
University of Michigan · University of Waterloo
Value Over Language Model: Detecting Original Contribution in Writing
Sep 1, 2026Vibhhu Sharma, Thorsten Joachims, Sarah DeanLLM Evaluation AI-Generated Text Detection
Cornell University
Self-Reports Are Not Verification: Environment-Grounded Auditing of LLM Operators in Evolutionary Search
Sep 1, 2026Enrong Pan, Ryan Zhou, Ting HuLLM Auditing Language Model Self-Assessment
Queen’s University
Validity-Aware Jailbreak Evaluation for Large Language Models
Aug 31, 2026Qilong Wu, Sahil Wadhwa, Pranab Mohanty +2Language Model Safety Evaluation LLM Auditing
University of Illinois Urbana-Champaign · Capital One
Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment
Aug 31, 2026Saad Mohammad Abrar, Eesha Kurella, Arnav Dadarya +3LLM Auditing Human Behavior Prediction
University of Maryland USA
Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries
Aug 31, 2026Phuong Anh Nguyen, Jill Noorily, Matthew Flathers +7LLM Evaluation LLM Auditing
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02446, USA · Psychiatry and Mental Health Department, School of Medicine, Pontificia Universidad Javeriana, Bogotá 110231, Colombia · Tufts University School of Medicine, Tufts University, Boston, MA 02111, USA
DIASENTINEL: An Auditable Multi-Agent System for Guideline-Grounded Diabetes Risk Screening
Aug 31, 2026Yung Wei Shueh, Zhi-Jie Chen, Chia-Hsuan Hsu +9Multi-Agent LLM Systems Medical Report Generation
Far Eastern Memorial Hospital, Taiwan · University of Michigan, USA · Rutgers University, USA +1
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Aug 31, 2026Adrians Skapars, Edoardo ManinoLanguage Model Steering Language Model Safety Evaluation
University of Manchester, United Kingdom
Commit-first LLM judging inherits the judge's own errors
Aug 31, 2026Idil GozelLLM-as-a-Judge LLM Auditing
Evaluator Integrity, London
Auditing MCQA Benchmarks through Probability Landscapes
Aug 31, 2026Minsoo Song, Chanjun ParkBenchmark Auditing LLM Auditing
Soongsil University
On the Recoverability of Private Information Unlearning in Large Language Models
Aug 30, 2026Shicheng Hu, Runzhi Tian, Ziqiao Wang +1LLM Auditing Privacy Leakage in Language Models
University of Illinois Urbana-Champaign · University of Ottawa · Tongji University
Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered
Aug 29, 2026Aryo Pradipta Gema, Neel Rajani, Rohit Saxena +2CoT Faithfulness LLM Auditing
University of Edinburgh · University of Edinburgh, Miniml.AI
Validating FKG.in: Soundness Assessment in LLM-Augmented Indian Food Knowledge
Aug 29, 2026Saransh Kumar Gupta, Armaan Shah, Lipika Dey +2LLM Auditing LLM Reliability
Department of Computer Science, Ashoka University, Sonipat, Haryana 131029, India · Mphasis AI and Applied Tech Lab, Ashoka University, Sonipat, Haryana 131029, India · Koita Centre for Digital Health, Ashoka University, Sonipat, Haryana 131029, India +1
Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification
Aug 28, 2026Cameron Wilding, Mina Shaker, Fatemeh GanjiLLM Auditing Privacy-Preserving ML
Worcester Polytechnic Institute Worcester, MA, USA
Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Aug 27, 2026Shuyi Fan, Boyuan Deng, Mengyu Xu +3Causal Inference LLM Auditing
The University of Chicago · Johns Hopkins University · The Hong Kong Polytechnic University, Hong Kong
The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion
Aug 26, 2026Chenglong Ma, Xinye Wanyan, Danula Hettiachchi +2LLM Auditing IR Evaluation
RMIT University Melbourne, Victoria, Australia
The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
Aug 24, 2026Taebong Kim, Youngsik Hong, Minsik Kim +3LLM Auditing Sequence Modeling
VIDRAFT AI Research · QuantumOS, Seoul, Republic of Korea

Sep 9, 2026, cs.CLJ/K move · Enter open · S save

GANDR: Claim Auditing for Verifiable Legal Answer Generation

Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu+1

William & Mary · Anytime AI

LLM Answer Verification Legal Citation Verification LLM Auditing Citation Verification Legal QA Evidence-Grounded Generation

Open paper AskPDF arXiv

In high-stakes domains such as legal practice, a language-model answer is only useful to the extent that a reader can verify each claim against the source the system cites. Current grounded-generation pipelines score the answer as a whole, so a correct conclusion can rest on fabricated or loosely matched citations and still score well. Closing this gap requires both a system built for per-claim verification and an evaluation that measures it. We introduce GANDR (Grounded ANswer DRafter), a two-agent system in which a Drafter writes an answer in a structured legal-reasoning format and a separate Critic, with the same view as a human verifier, audits each claim against its cited source and emits a per-claim audit trace on every round. We pair it with a strict correctness criterion requiring every citation to resolve to a passage the retriever returned. On a 185-item legal benchmark where all six systems share one backbone, one retrieval surface, and one citation instruction, GANDR ranks first on every primary metric, reaching 70.8% strict accuracy and leading the strongest baseline by 11.3 points (p<0.01). Reverting the protocol-anchored commit rule lowers strict accuracy by 22.7 points, and the strict lead stays positive on three further backbones, at +3.2 to +6.5 points. This lead traces to the Drafter configuration and the protocol-anchored commit, not to rewriting. Against two law-trained annotators the audit flags under-supported claims at F1 0.84 as a binary detector, while its four-way verdict labels agree only weakly and are advisory. Code is available upon request.

Momentum

Related topics

Latest papers 453