Organizations: Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · CMU · EuroSafeAI · ELLIS Institute Tübingen
Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge.
Figures & tables
Figure 1: Overview of the CLT-Forge framework.
Figure 2: Activation caching and compression trade-offs. Left pair: Activation loading throughput for on-the-fly computation, cache generation, and cached reads. Cached reads achieve the highest throughput, while cache generation incurs a one-time serialization cost. Relative training time of caching compared to on-the-fly computation as a function of the average number of reuses per activation. Curves correspond to different levels of parallel cache generation since caching can easily be parallelized, while vertical markers indicate the break-even reuse count where caching becomes faster than recomputing activations. Right pair: Compression ratio achieved by different storage formats and compression methods, together with their corresponding save and load bandwidths. Int8 quantization reduces storage by up to 3.6× while introducing a 12.5% reconstruction error.
Figure 3: CLT-Forge training dynamics and scalability. Left pair: Training dynamics of a Cross-Layer Transcoder on Llama-3.2-1B activations with 65,536 features per layer (16 layers), trained on 300M OpenWebText tokens in 17 hours using a single 8 × H100 node. To our knowledge, this is the first open-source implementation for training CLTs on a 1B-scale language model. Right pair: Comparison of feature sharding and data parallelism as the total number of features increases. Feature sharding dramatically reduces per-GPU memory while improving throughput, enabling substantially larger dictionaries than conventional data-parallel training.
Figure 4: Sparse ( embedding_bag ) vs. dense decode runtime ratio at fixed k (bf16, din=2048 , cross-layer JumpReLU CLT). Below 1.0, sparse wins; the break-even k scales roughly linearly with expansion — ∼20 , 50 , 100 for 8× , 16× , 32× .
Figure 5: Example of a GPT-2 attribution-graph for the sentence ’The opposite of "large" is’ with replacement-score 0.77 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
GPT-2
Llama-3.2-1B
Dictionary size, dlatent
24,576
65,536
Expansion factor
32×
32×
Training tokens
209M
307M
Token batch size
8192
8192
Adam (β1,β2)
(0.9,0.999)
(0.9,0.999)
Peak learning rate
2×10−4
2×10−4
Appendix
Table 1: Optimal training configurations for the GPT-2 and Llama-3.2-1B Cross-Layer Transcoders (CLTs).
Figure 6: Training curves for the Llama-3.2-1B CLT over 307 M tokens. Left to right: L0-penalty coefficient λL0 (warmup schedule), L0 loss, dead-feature auxiliary loss, and dead-feature count. Log-y axes are clipped to the post-warmup range (from 5% of tokens) so the initial spike is off-frame.
Figure 7: Training curves for the GPT-2 CLT over 209 M tokens. Left to right: explained variance on the reconstruction, dead-feature count, average L0 per token, and overall training loss. As λL0 ramps up, L0 collapses by three decades and dead features grow, while explained variance peaks early and then degrades as the sparsity penalty tightens.
Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.
Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4
Rutgers University · University of California San Diego
Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.
Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail +7
Understanding the internal representations of large language models (LLMs) is a central challenge in interpretability research. Existing feature interpretability methods often rely on strong structural assumptions--such as linearity or sparsity--that may not hold in practice. In this work, we introduce InverseScope, an assumption-light and scalable framework for interpreting neural activations via input inversion. Given a target activation, InverseScope characterizes its encoded information by generating natural-language inputs that produce nearby activations, grounding abstract internal states in concrete language. To overcome the prohibitive cost of sampling in high-dimensional activation spaces, we propose a novel control-layer conditioning architecture that substantially improves sample efficiency compared to prior token-prepending approaches. We demonstrate that InverseScope reveals rich geometric structure in LLM representation spaces, including sentence-level linear analogies. The framework scales to state-of-the-art open-source models of up to 14B parameters and generalizes to out-of-distribution inputs, enabling systematic analysis of activation neighborhoods.
Yifan Luo, Zhennan Zhou, Bin Dong
School of Mathematical Science, Peking University, Beijing, China · School of Science, Westlake University, Hangzhou, China · Beijing International Center for Mathematical Research and the New Cornerstone Science Laboratory, Peking University, Beijing, China +1