cs.CLNov 3, 2026

Whose Effect Is It? How Objectives Inflate Learned Subspace Interventions

Authors: Toygar Tanyel, Serkan Ayvaz

Organizations: Centre for Industrial Software University of Southern Denmark

Abstract

Interpretability often locates concepts by copying part of a language model's state from one input into another (activation patching). A larger output change counts as stronger evidence that this part carries the concept. Methods that learn which part to copy often change the output more than the whole state does. By that reading, learned parts isolate the concept more cleanly. Yet the whole contains the part. So whose effect is this excess, the network's or the method's? Copying the whole state learns nothing, so it shows what the network produces. We prove that, to first order, the effects of orthogonal parts add up to that of the whole. A part therefore exceeds the whole exactly when the rest moves the output back, which does not require isolating the concept more cleanly. Objectives that reward a larger effect select such parts. Distributed alignment search with cross-entropy, for example, exceeds the whole-state copy just before the output on almost all tasks, and inside the network at most sites. In both, changing only its objective to target what the network produces removes most of the excess. It therefore comes largely from the objective, so a larger learned effect alone is not stronger evidence.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Concept Subspaces Compute Beyond the Logit Lens: A Weights-Only Test for Locating Representations Upstream of Readout

    Sep 30, 2026Aojie Yuan, Zhiyuan Julian Su, Haiyue Zhang +1Transformer InterpretabilityRepresentation Geometry in Language Models

  2. Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models

    Oct 2, 2026Sankaran Vaidyanathan, Rafal Urbaniak, Emily Bunnapradist +2Circuits in Language ModelsMechanistic Interpretability