cs.LGMay 30, 2026

Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects

Authors: Hwiyeong LeeIngyu BangUiji HwangHyelim LimTaeuk Kim

Organizations: 1Hanyang University, Seoul, Republic of Korea.

Abstract

While sparse autoencoders provide features more interpretable than individual neurons, reliably characterizing them remains challenging. We propose Query Lens, which extends Logit Lens to enable more comprehensive and faithful interpretations of sparse features. By jointly considering encoder-side key features and decoder-side value features, we identify both the inputs that activate a feature and the outputs it promotes. We also account for indirect, module-mediated effects that arise when the feature is processed by downstream modules, going beyond the direct effect captured by Logit Lens. In experiments, we find that Query Lens yields coherent token signatures for features that remain uninterpretable under Logit Lens. Finally, we propose the Subspace Channel Hypothesis, suggesting that downstream modules read features through layer-specific subspaces.

Explore similar work

CardsList