From Tokens to Concepts: Leveraging SAE for SPLADE

Yuxuan Zong; Mathias Vast; Basile Van Cooten; Laure Soulier; Benjamin Piwowarski

cs.IRApr 23, 2026

From Tokens to Concepts: Leveraging SAE for SPLADE

Authors: Yuxuan Zong, Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski

Organizations: Sorbonne Université, CNRS, ISIR · Sinequa by ChapsVision · CNRS, Sorbonne Université, ISIR

Abstract

Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.

Explore similar work

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
Jun 23, 2026Jonas Klotz, Cassio F. Dantas, Pallavi Jain +2Improving Sparse Autoencoders Semantic Representations
Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Retrieval Layer Large Language Model Backbones
When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
Jul 9, 2026Weiduo Liao, Yunqiao Yang, Ying WeiImproving Sparse Autoencoders Visual Semantics

Jun 23, 2026 · cs.CVJ/K move · Enter open · S save

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos+1

The Berlin Institute for the Foundations of Learning and Data (BIFOLD) · Technische Universität Berlin, Germany · INRAE · Inria, EVERGREEN · UMR TETIS, Univ Montpellier, France

Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence. We present a human-grounded evaluation framework that quantifies alignment between SAE latents and human-annotated concepts, without requiring user studies, and validate this matching through targeted attribute perturbations. To enable this intervention-style evaluation in vision, we construct synCUB and synCOCO, synthetic benchmarks of paired images that differ in exactly one attribute. We introduce Fully-Binary Matching Pursuit (FBMP), a coalition-based matching procedure that supports many-to-one mappings between SAE latents and annotated concepts, and consistently outperforms one-to-one baselines. For functional validation, we propose a Targeted Attribute Perturbation Alignment Score (TAPAScore), which tests whether matched concepts respond selectively and in the expected direction under targeted image-level attribute perturbations. Under sanity checks, our matching and TAPAScore are the only evaluated metrics that reliably distinguish trained SAEs from untrained ones. Across SAEs trained on CLIP and DINOv2 embeddings, we find that increased overcompleteness can reduce perturbation alignment, indicating a reduction in interpretability. Our evaluation framework suggests that moderate dictionary sizes provide the best trade-off, yielding the most interpretable SAEs. Code and datasets are available at https://github.com/JonasKlotz/sae-concept-eval.

Improving Sparse Autoencoders Semantic Representations Model Interpretability Annotations

Open paper Ask about it PDF

Abstract

Explore similar work

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

Rescaling MLM-Head for Neural Sparse Retrieval

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities