Organizations: Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · CMU · EuroSafeAI · ELLIS Institute Tübingen
Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge.
Figures & tables
Figure 1: Overview of the CLT-Forge framework.
Figure 2: Activation caching and compression trade-offs. Left pair: Activation loading throughput for on-the-fly computation, cache generation, and cached reads. Cached reads achieve the highest throughput, while cache generation incurs a one-time serialization cost. Relative training time of caching compared to on-the-fly computation as a function of the average number of reuses per activation. Curves correspond to different levels of parallel cache generation since caching can easily be parallelized, while vertical markers indicate the break-even reuse count where caching becomes faster than recomputing activations. Right pair: Compression ratio achieved by different storage formats and compression methods, together with their corresponding save and load bandwidths. Int8 quantization reduces storage by up to 3.6× while introducing a 12.5% reconstruction error.
Figure 3: CLT-Forge training dynamics and scalability. Left pair: Training dynamics of a Cross-Layer Transcoder on Llama-3.2-1B activations with 65,536 features per layer (16 layers), trained on 300M OpenWebText tokens in 17 hours using a single 8 × H100 node. To our knowledge, this is the first open-source implementation for training CLTs on a 1B-scale language model. Right pair: Comparison of feature sharding and data parallelism as the total number of features increases. Feature sharding dramatically reduces per-GPU memory while improving throughput, enabling substantially larger dictionaries than conventional data-parallel training.
Figure 4: Sparse ( embedding_bag ) vs. dense decode runtime ratio at fixed k (bf16, din=2048 , cross-layer JumpReLU CLT). Below 1.0, sparse wins; the break-even k scales roughly linearly with expansion — ∼20 , 50 , 100 for 8× , 16× , 32× .
Figure 5: Example of a GPT-2 attribution-graph for the sentence ’The opposite of "large" is’ with replacement-score 0.77 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
GPT-2
Llama-3.2-1B
Dictionary size, dlatent
24,576
65,536
Expansion factor
32×
32×
Training tokens
209M
307M
Token batch size
8192
8192
Adam (β1,β2)
(0.9,0.999)
(0.9,0.999)
Peak learning rate
2×10−4
2×10−4
Appendix
Table 1: Optimal training configurations for the GPT-2 and Llama-3.2-1B Cross-Layer Transcoders (CLTs).
Figure 6: Training curves for the Llama-3.2-1B CLT over 307 M tokens. Left to right: L0-penalty coefficient λL0 (warmup schedule), L0 loss, dead-feature auxiliary loss, and dead-feature count. Log-y axes are clipped to the post-warmup range (from 5% of tokens) so the initial spike is off-frame.
Figure 7: Training curves for the GPT-2 CLT over 209 M tokens. Left to right: explained variance on the reconstruction, dead-feature count, average L0 per token, and overall training loss. As λL0 ramps up, L0 collapses by three decades and dead features grow, while explained variance peaks early and then degrades as the sparsity penalty tightens.
School of Mathematical Science, Peking University, Beijing, China · School of Science, Westlake University, Hangzhou, China · Beijing International Center for Mathematical Research and the New Cornerstone Science Laboratory, Peking University, Beijing, China +1