cs.LGJul 30, 2025

Pulling Back the Curtain on Deep Networks

Authors: Maciej Satkiewicz, Roberto Corizzo, Marcin Pietroń

Organizations: Foundation · American University · AGH University of Krakow

Abstract

In linear models, visualizing a weight vector naturally reveals the model's preferred input direction, but extending this intuition to deep networks via gradients or GradientAscent often yields brittle or adversarial-looking features. We argue that deep networks are better understood as input-conditioned affine operators, whose natural adjoint action pulls a neuron's preferred direction back to input space. We further refine this representation through optional backward-only softening for hard-gated layers and propose pullbacks as an alternative backward operator for existing explanation procedures, including local attribution, smoothing, and iterative ascent. Across convolutional architectures (ResNet50, VGG) and transformer-based models (PVT), replacing gradients with pullbacks yields improvements in 46 out of 60 comparisons across five faithfulness, stability, and localisation metrics. In particular, PullbackAscent improves mean scores over GradientAscent across all five primary metrics on each architecture, reducing mean Infidelity by factors of 2.8 to 5.5. Strikingly, it also produces sharp, perceptually aligned and target-specific perturbations in only a few steps, without post-processing or explicit perceptual priors. More broadly, pullback-based methods achieve the best mean in 13 out of 15 architecture-metric combinations, outperforming established attribution baselines. Our results suggest that adjoint transport can serve as a reusable backward primitive, enhancing existing explanation procedures without architectural changes or elaborate attribution pipelines. We use "Semantic Pullbacks" (SP) to denote the resulting family of pullback-based explanations.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Playing the network backward: A Game Theoretic Attribution Framework

    May 7, 2026Jakob Paul Zimmermann, Jim Berend, Georg Loho +2Gradient-Based AttributionTransformer Interpretability

  2. Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI

    Oct 1, 2025Akchunya Chanchal, David A. Kelly, Hana ChocklerExplainable Artificial IntelligenceExplanation Stability

  3. Learn to Rank: Visual Attribution by Learning Importance Ranking

    Apr 7, 2026David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2Feature AttributionExplainable Image Classification