Algorithmic Auditing

Ask about this topic

Overview

Algorithmic auditing studies how to test automated decisions and the systems around them, especially when errors or disparities affect people. Common approaches compare subgroup outcomes, use counterfactual or causal tests, probe model behavior under perturbations, and apply statistical methods such as sequential testing and uncertainty analysis. Audits increasingly examine deployed pipelines, data and component interactions, strategic responses, and evidence trails—not just a model’s aggregate accuracy. A central challenge is making conclusions reliable and meaningful despite limited access, shifting conditions, incomplete data, and metrics that can be gamed or fail to capture the harms that matter.

Latest papers

CardsList

143 papers27 papers in the last four weeks, up 59% on the four weeks before. 0.2% of all new papers.

  1. FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents

    Oct 27, 2026Arther Tian, Alex Ding, Simon Wu +1AI Agent AuditingProcess-Based Agent Evaluation

    DGrid AI

  2. Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data

    Oct 26, 2026Kushagra Bhatnagar, Deekshith MarlaAlgorithmic AuditingData Provenance

  3. Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences

    Oct 23, 2026Fangyuan Lin, Noah Krever, Jakub Černý +1Algorithmic AuditingAdaptive Sampling

    Columbia University

  4. Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

    Oct 23, 2026John deVadossAI GovernanceAI Regulatory Compliance

    InterWork Alliance, Washington, DC

  5. Reliable AI-Assisted Decisions via Selective Auditing

    Oct 22, 2026Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1AI-Assisted Decision MakingHuman Oversight of AI

    Departments of Statistics and Management Science & Engineering, Stanford University

  6. Auditing the Hand-Written Inputs of Learned Access Control

    Oct 22, 2026Arsalan Jalil Sarghaleh, Mehdi HazratiAccess ControlAlgorithmic Auditing

    Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran · Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada

  7. The Price of Choice in Selective Prediction

    Oct 21, 2026Daniel Cieślak, Andrzej CzyżewskiSelective PredictionAlgorithmic Auditing

    Gdańsk University of Technology Gdańsk, Poland

  8. Better Policies, Worse Verifier Audits

    Oct 21, 2026Daniel Cieślak, Jacek ŁubińskiAlgorithmic AuditingPolicy Evaluation

    Gdańsk University of Technology, Gdańsk, Poland

  9. A Cue's Hidden Price: Auditing Seller Influence on AI Shoppers

    Oct 19, 2026Felix Buecheler, Gerardo Berbeglia, Mingyu GuoAI Agent AuditingGenerative Engine Optimization

    Adelaide University, Adelaide, Australia · Melbourne Business School, Melbourne, Australia

  10. Multi-Sensitive Distributional Fairness: Joint Wasserstein Audits and Shapley Allocation

    Oct 13, 2026Agathe Fernandes Machado, Charlotte Jamotton, Arthur CharpentierGroup FairnessIntersectional Fairness

  11. Finite-sample certification of coverage regret for budgeted document selection under adaptive stratified auditing

    Oct 13, 2026Nora SerdyukovaRegret MinimizationSelective Inference

    Department of Statistics, University of Concepcion, Chile

  12. Gaussian Process Bandits for AI System Optimization with Selectively Verified Feedback

    Oct 13, 2026Shuang WuGaussian Process BanditsMulti-Armed Bandits

    UCLA

  13. Transparent Information Flow Modeling for Role-Aware AI Act Assessment

    Oct 12, 2026Mattias Brännström, Themis Dimitra Xanthopoulou, Lili JiangAI GovernanceAlgorithmic Auditing

    Umeå University, Sweden

  14. Pre-Registered Genesis Accounting for Recursive Self-Improvement: First Audited Operator-Genesis Events in a Physics-Discovery Runtime

    Oct 10, 2026Wenbin Mao, Pei ZouRecursive Self-ImprovementAutonomous Scientific Discovery

    HoomanBio AI Lab, Huaman Biotechnology (Beijing) Co., Ltd.

  15. Actionable Meta-Requirements for AI Robustness Auditing

    Oct 9, 2026Joshua Owotogbe, Roya Nasiri, Indika Kumara +2AI AssuranceAlgorithmic Auditing

    Jheronimus Academy of Data Science, Netherlands · Tilburg University, Netherlands · University of Sannio, Italy

  16. Shared Canonical Component Removal Beyond Expert Views

    Oct 9, 2026Jose Roberto Tello Ayala, Akl C. Fahed, Roy Perlis +1Algorithmic AuditingConcept Erasure

    School of Engineering and Applied Sciences Harvard University · Heart and Vascular Institute Mass General Brigham · Center for Quantitative Health Mass General Brigham

  17. TRACE: A Governance Framework for Measuring Explainability Debt in Production AI Systems

    Oct 7, 2026Harish Kant PathakAI GovernanceAI Accountability

    Independent Researcher

  18. Robust Decentralized Fairness Auditing

    Oct 7, 2026Sayan Biswas, Jade Garcia Bourrée, Anne-Marie Kermarrec +2Algorithmic AuditingLLM Auditing

    EPFL, Switzerland

  19. Shared and structured inputs undermine collective random choice by reasoning AI agents

    Oct 7, 2026Takahiro Ezaki, Naoto Imura, Katsuhiro NishinariAI Agent EvaluationSelection Bias

    Research Center for Advanced Science and Technology, The University of Tokyo 4-6-1 Komaba, Meguro-ku, Tokyo 153-8904, Japan · Department of Aeronautics and Astronautics, School of Engineering, The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan

  20. Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging

    Oct 6, 2026Hamed Khosravi, Xiaoming HuoCorruption RobustnessAnytime-Valid Inference

    H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA, USA

  21. CHARTER: Auditing Reference Substitution in Hierarchical Compact-Evidence Evaluation for Computational Pathology

    Oct 6, 2026Hyun Do Jung, Jungwon Choi, Soojung Choi +2Computational PathologyAlgorithmic Auditing

    Yonsei University · KAIST · Hallym University Chuncheon Sacred Heart Hospital

  22. Can Power Draw Constrain Covert Compute? Limits of Analogue Verification for AI Governance

    Oct 5, 2026Tom Kimpson, Mauricio Baker, Emlyn GrahamAI GovernanceAlgorithmic Auditing

    School of Mathematics and Statistics, The University of Melbourne, Parkville, VIC, Australia · Machine Alignment, Transparency & Security (MATS) · Department of Engineering Science, University of Oxford, Oxford, UK

  23. ROUTEAUDIT: Interaction-Aware Identification for Budgeted Multi-Verifier Routing

    Oct 2, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +4Adaptive Model RoutingAlgorithmic Auditing

    School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

  24. Where Quantum Fourier Sampling Stops Short: A Three-Gate Audit Protocol for Delay-PUF Security Models

    Oct 2, 2026Owen Friedewald, Ali Shiri Sichani, Chi-Ren ShyuAlgorithmic Auditing

    Department of Electrical Engineering and Computer Science University of Missouri, Columbia, MO, USA

  25. Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening

    Oct 1, 2026Jie Tang, Chuanlong Xie, Lixing ZhuAlgorithmic AuditingAlgorithmic Fairness

    School of Statistics, Beijing Normal University, Zhuhai, Guangdong 519087, China

  26. A Rank Graduation metric for Algorithmic fairness

    Sep 30, 2026Dalia Atif, Paolo GiudiciAlgorithmic AuditingAlgorithmic Fairness

  27. Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

    Sep 29, 2026Kelly McConvey, Angelina Zhai, Rebecca Li +1Algorithmic AuditingAlgorithmic Fairness

    University of Toronto, Toronto, Ontario, Canada · Georgia Institute of Technology, Atlanta, Georgia, USA

  28. When Does Randomized Oversight Align AI Agents That Can Conceal?

    Sep 29, 2026Joshua S. Gans, Richard HoldenAlgorithmic AuditingAI Agent Auditing

    UNSW Business School

  29. VStress: Correlation-Aware Auditing and Adaptive Budget Allocation for Repeated Verifiers

    Sep 29, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +4Algorithmic AuditingBlack-Box Model Auditing

    School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

  30. Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification

    Sep 27, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +4Algorithmic AuditingReinforcement Learning with Verifiable Rewards

    School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China