High-performing vision language models still produce incorrect answers, yet their failure modes are often difficult to explain. To make model internals more accessible and enable systematic debugging, we introduce VisualScratchpad, an interactive interface for visual concept analysis during inference. We apply sparse autoencoders to the vision encoder and link the resulting visual concepts to text tokens via text-to-image attention, allowing us to examine which visual concepts are both captured by the vision encoder and utilized by the language model. VisualScratchpad also provides a token-latent heatmap view that suggests a sufficient set of latents for effective concept ablation in causal analysis. Through case studies, we reveal three underexplored failure modes: limited cross-modal alignment, misleading visual concepts, and unused hidden cues. Project page: https://hyesulim.github.io/visual_scratchpad_projectpage/
Figures & tables
Figure 1: VisualScratchpad pipeline. A. During inference in a vision-language model, we extract the intermediate representation z from the vision encoder. B. A sparse autoencoder processes z to produce concept activations. The attention map from output text tokens to image tokens is applied at the patch level to weight these activations. Latents exhibiting similar activation patterns across output tokens are then clustered and visualized in a token–latent heatmap. C. The causal influence of these concepts on the model’s output can be evaluated through latent ablation.
Figure 2: Attention-based concept re-ranking . A. SAEs return latent activations for each image patch. B. Image-level activations can be computed by naïvely averaging activations across all patches, or C. by applying a weighted average where the text-to-image attention map serves as the weighting coefficient, promoting concepts relevant to the text tokens to the top of the ranking. The bottom row shows the top-ranked concept obtained from the corresponding method.
Figure 3: Case studies. A. Limited cross-alignment, B. misleading visual cues, and C. unused hidden cues can cause wrong answer.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Complexity of visual concepts across layers. Early layers capture simple visual components, including position and color, mid layers capture object-level shape and patterns, and late layers understand scene-level events.
Figure 5: Token-Latent heatmap visualization. Raw values are difficult to analyze, so we normalize in column-wise to show if the latent is specifically attended by certain tokens or by overall tokens. Moreover, we cluster and sort by activation correlation in column-wise, using hierarchical clustering.
Figure 6: Illustration on how SAE latent statistics are computed. After training the SAE, we pass the dataset through the model and compute each latent’s mean activation and sparsity, yielding a compact summary of its activation distribution. Right: Based on the mean activation and sparsity, we filter out latents in top k % of each axis treating them as noisy latents.
Figure 7: Causal analysis example. Given an image and a prompt to describe it, the VLM generates a caption and VisualScratchpad produces a token–latent heatmap. Latents are re-ranked using token attention and clustered based on activation similarity. Each cluster roughly corresponds to a main topic in the caption. By zeroing out the latents in each cluster and reconstructing the representations, we observe how the outputs change relative to the original. Ablating latent clusters that correspond to distinct semantic topics removes those topics from the generated caption, demonstrating their causal role in shaping the model’s predictions.
Figure 8: Three main components of VisualScratchpad include (1) exploring SAE latents, (2) model inference with SAE (vision-question answering using VLMs and image classification using CLIP), and (3) close-up observation on activated concepts.
Figure 9: Control pannels of VisualScratchpad. Users can adjust visualization settings for spatial attribution masks, configure options for activation-value plots, and customize heatmap displays.
Figure 10: Details for Case 2. We iterated for two turns to remove sitting-related concepts.