LLM Auditing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

59 papers in the last four weeks, up 79% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 453

All topics

Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoning
Jun 24, 2026Poojitha Thota, Shirin NilizadehLLM Auditing LLM Fine-Tuning
The University of Texas at Arlington Arlington, Texas, USA
Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients
Jun 23, 2026Zhihao Zhu, Hongyi Tang, Yi Yang +1LLM Auditing
Department of Information Systems, Business Statistics and Operations Management (ISOM), Hong Kong University of Science and Technology, Hong Kong, China. · Department of IT, Analytics, and Operations, University of Notre Dame, Notre Dame, Indiana, USA.
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Jun 23, 2026Federico Marcuzzi, Xuefei Ning, Roy Schwartz +1Pairwise Comparison LLM Auditing
INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Tsinghua University, China · The Hebrew University of Jerusalem, Israel +1
Natural Identifiers for Privacy and Data Audits in Large Language Models
Jun 23, 2026Lorenzo Rossi, Bartłomiej Marek, Franziska Boenisch +1Privacy Auditing LLM Auditing
CISPA Helmholtz Center for Information Security
When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs
Jun 23, 2026Hiroshi OkumuraLLM Auditing Pragmatic Reasoning in Language Models
Mitsubishi Research Institute, Inc. · Kobe University
Probing the Misaligned Thinking Process of Language Models
Jun 23, 2026Kaiwen Zhou, Constantin Venhoff, Jonathan Michala +2LLM Alignment LLM Auditing
1Anthropic Fellows · 2MATS · 3UCSC +2
Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs
Jun 22, 2026Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz WeinsteinLanguage Model Generation Evaluation LLM Auditing
Amazon Web Services
Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior
Jun 22, 2026Christopher J. Anders, Henrique Da Silva Gameiro, Nico Daheim +1LLM Auditing LLM Interpretability
RIKEN Center for Advanced Intelligence Project, Tokyo, Japan · Section of Computer Science, EPFL Lausanne, Switzerland · Ubiquitous Knowledge Processing Lab (UKP Lab),2026 Department of Computer Science, Technical University of Darmstadt +2
ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models
Jun 22, 2026Jun Zhang, Jiasheng Zheng, Boxi Cao +5LLM Auditing LLM Interpretability
Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Who Owns the AI Recommendation? A Multi-Industry Empirical Map of Brand Category Ownership Across Large Language Models
Jun 22, 2026Dmitrij ŻatuchinLLM Evaluation LLM Auditing
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models
Jun 22, 2026Yiran Zhu, Ziqi YangMembership Inference Attacks LLM Auditing
Zhejiang University
Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers
Jun 19, 2026Vatsal Ananthula, Adarsh KumarappanFaithfulness of Language Model Explanations LLM Auditing
University of California, Berkeley · 2California Institute of Technology
Reference-Based Distillation Detection in LLMs
Jun 19, 2026Rajat Rawat, Sizhe Chen, Akshay Anand +3LLM Auditing Teacher-Student Learning
University of California, Berkeley · University of Southern California · OpenAI
EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answering
Jun 19, 2026Fengchen Gu, Xiaotian Ren, Zhengyong Jiang +6Data Visualization LLM Answer Verification
School of AI and Advanced Computing, XJTLU Entrepreneur College (Taicang), Xi’an Jiaotong-Liverpool University, Suzhou, Jiangsu, China · ETSI de Telecomunicaci´on, Universidad Polit´ecnica de Madrid, Madrid, Spain
A Multi-Agent Audit Framework for High-Stakes Reasoning: Evaluation and Interpretability in Clinical Mental Health Screening
Jun 19, 2026Jingchen Ye, Yanpei Yu, Luyao ZhangLLM Auditing LLM Interpretability
Duke Kunshan University Suzhou, China · Social Science Division and Digital Innovation Research Center, Duke Kunshan University, No.8 Duke Ave., Kunshan, Jiangsu 215316, China.
Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families
Jun 18, 2026Abdul Rafay SyedLLM Auditing Emergent Misalignment in Language Models
Department of Computer Science Universität des Saarlandes Saarbrücken, Germany
AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing
Jun 18, 2026Zilong Zhang, Yi-Ting Hung, Weiyi He +3LLM-as-a-Judge LLM Auditing
Department of Mathematics and Statistics Georgia State University · Department of Probability and Statistics Department of Computer Science and Engineering Michigan State University · Department of Mathematics and Statistics Concordia University +1
Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning
Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Inverse Optimal Transport LLM-as-a-Judge
Department of Mathematics and Statistics Georgia State University · Department of Statistics University of Manitoba
PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning
Jun 16, 2026Bo Su, Ankit Shah, Thai LeLLM Auditing Machine Unlearning
Indiana University Bloomington, IN, USA
Self-CTRL: Self-Consistency Training with Reinforcement Learning
Jun 16, 2026Itamar Pres, Laura Ruis, Melat Ghebreselassie +2RL for Language Models Faithfulness of Language Model Explanations
MIT CSAIL
Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond
Jun 16, 2026Hobin Kim, Xiaoyuan Wu, Omer Akgul +2LLM Auditing LLM Reliability
Carnegie Mellon University · RSAC Labs
Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditing
Jun 16, 2026Kexin Chen, Yi Liu, Haonan Zhang +3LLM Auditing LLM Interpretability
Zhejiang University · Griffith University
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Jun 16, 2026Jeffrey G. Wang, Jason Wang, Marvin Li +1LLM Evaluation Membership Inference Attacks
Harvard University · Harvard Business School
Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
Jun 15, 2026Yanan LongLLM Evaluation Benchmark Auditing
StickFlux Labs
MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions
Jun 15, 2026Noor Islam S. Mohammad, Tamim SheikhLLM Safety Benchmarks LLM Auditing
Department of Computer Science, Informatics Institute, Istanbul Technical University, ˙Istanbul, Türkiye · Department of Computer Science and Engineering, Jashore University of Science and Technology, Bangladesh.
DoubtProbe: Black-Box Jailbreak Defense via Structural Verification and Semantic Auditing
Jun 15, 2026Xuanyu Yin, Yilin Jiang, Jun Zhou +3LLM Auditing LLM Jailbreak Attacks
East China Normal University Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · Institute of Information Engineering, Chinese Academy of Sciences Beijing, China
Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection
Jun 15, 2026Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Asher AliLLM Auditing Recommender Systems
Fandaqah, Al Khobar, Saudi Arabia. · Hamdard University, Karachi, Pakistan.
AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models
Jun 15, 2026Andreas Einwiller, Max Klabunde, Florian LemmerichLanguage Model Safety Evaluation LLM Auditing
University of Passau, Germany
Frame-Conditioned Moral Computation in LLaMA 3.1-8B-Instruct: A Mechanistic Interpretability Audit of Ethical Reasoning
Jun 13, 2026Ali Dasdan, Manan Shah, W. Russell Neuman +3LLM Alignment Moral Reasoning in Language Models
1KD Consulting, CA, USA · 2New York University, NY, USA
Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisation
Jun 12, 2026Arnesh Banerjee, Ayushi BhattacharjeeMathematical Reasoning Benchmarks LLM Auditing
Dept. of CSE (Data Science) Heritage Institute of Technology Kolkata, India

Jun 24, 2026, cs.CLJ/K move · Enter open · S save

Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoning

Poojitha Thota, Shirin Nilizadeh

The University of Texas at Arlington Arlington, Texas, USA

LLM Auditing LLM Fine-Tuning Text Summarization Data Poisoning Attacks LLM Unlearning

Open paper AskPDF arXiv

Training-time data poisoning during fine-tuning poses a significant threat to large language models (LLMs) deployed for abstractive text summarization, where small task-specific datasets exert disproportionate influence on model behavior. In this setting, adversaries manipulate fine-tuning data to induce persistent summarization failures, such as biased or harmful summaries, while preserving standard evaluation metrics. We present a unified post-hoc defense framework for detecting and remediating fine-tuning-stage poisoning in summarization models across the machine learning supply chain. Our experiments show that in white-box settings, poisoned document-summary pairs exhibit abnormally high training influence, enabling detection via influence-function analysis with semantic consistency checks. In black-box settings, poisoned models display two to three times greater sensitivity to semantics-preserving perturbations, enabling behavioral auditing without training data access. Beyond existing poisoning formulations, we introduce novel attacks targeting factual distortion and representational bias, showing that poisoning alters summarization behavior without triggering conventional alarms. Across nine architectures and six benchmark datasets under adaptive attacks, our defenses achieve 85-92% detection precision, while gradient-ascent unlearning restores up to 96% of original behavior with minimal utility loss (less than 0.6% ROUGE degradation). These results indicate that fine-tuning-time poisoning leaves persistent structural artifacts, enabling practical detection and post-deployment recovery without full retraining.

Momentum

Related topics

Latest papers 453