Navigating the Concept Space of Language Models
Organizations: Adaption Labs, Curitiba, PR, Brazil · São Paulo State University (UNESP), Presidente Prudente, SP, Brazil
Abstract
Sparse autoencoders (SAEs) trained on large language model activations output thousands of features that enable mapping to human-interpretable concepts. The current practice for analyzing these features primarily relies on inspecting top-activating examples, manually browsing individual features, or performing semantic search on interested concepts, which makes exploratory discovery of concepts difficult at scale. In this paper, we present Concept Explorer, a scalable interactive system for post-hoc exploration of SAE features that organizes concept explanations using hierarchical neighborhood embeddings. Our approach constructs a multi-resolution manifold over SAE feature embeddings and enables progressive navigation from coarse concept clusters to fine-grained neighborhoods, supporting discovery, comparison, and relationship analysis among concepts. We demonstrate the utility of Concept Explorer on SAE features extracted from SmolLM2, where it reveals coherent high-level structure, meaningful subclusters, and distinctive rare concepts that are hard to identify with existing workflows.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Class Name | Feature Id | Explanation ( This neuron captures… ) |
|---|---|---|
| Programming, glyphs, and emojis | 9529 | short interface/command tokens highlighted with double-angle-bracket markup — i.e., keyboard keys, button or menu labels, quoted literal strings, and brief instruction verbs/affixes (like ”click”, ”Start”, ”Esc”, or ”un” in ”unplug”) used in step-by-step UI/instruction text. |
| Programming, glyphs, and emojis | 6967 | short inline label/prefix tokens used as decorative/attention badges at the start of headlines or UI items—e.g., emojis, arrows or symbols placed in <<…>>to mark categories, new items, or section headings. |
| Programming, glyphs, and emojis | 18379 | imperative UI-action tokens—especially the verb ”click”—used to mark step-by-step instructions in software/docs (often highlighted inside <<>>). |
| Programming, glyphs, and emojis | 5466 | short tokens used in instructional/UI contexts — especially single-letter key names and function-key labels (e.g., C, V, F, R), lone punctuation used in shortcut/list formatting (comma, colon, semicolon, arrow), and brief connective words that appear in commands or how‑to steps (like, for, until, become, eventually). |
| Programming, glyphs, and emojis | 11714 | short, standalone identifier-like tokens — e.g., file/config extensions and CLI keywords (yml, yaml, compile), brief abbreviations/initialisms (M in M.V.P., MX), and compact content words (eyes, dry, nasal). In other words, it responds to short label/identifier tokens that commonly appear in code/command contexts and short noun/adjective tokens in prose. |
| Programming, glyphs, and emojis | 16455 | UI/menu/list marker glyphs and decorative symbols — short tokens used as bullets, chevrons, arrows, play/expand icons or section labels that mark menus, list items, metadata headers, or navigation cues in text. |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 778 papers