High-performing vision language models still produce incorrect answers, yet their failure modes are often difficult to explain. To make model internals more accessible and enable systematic debugging, we introduce VisualScratchpad, an interactive interface for visual concept analysis during inference. We apply sparse autoencoders to the vision encoder and link the resulting visual concepts to text tokens via text-to-image attention, allowing us to examine which visual concepts are both captured by the vision encoder and utilized by the language model. VisualScratchpad also provides a token-latent heatmap view that suggests a sufficient set of latents for effective concept ablation in causal analysis. Through case studies, we reveal three underexplored failure modes: limited cross-modal alignment, misleading visual concepts, and unused hidden cues. Project page: https://hyesulim.github.io/visual_scratchpad_projectpage/
Figures & tables
Figure 1: VisualScratchpad pipeline. A. During inference in a vision-language model, we extract the intermediate representation z from the vision encoder. B. A sparse autoencoder processes z to produce concept activations. The attention map from output text tokens to image tokens is applied at the patch level to weight these activations. Latents exhibiting similar activation patterns across output tokens are then clustered and visualized in a token–latent heatmap. C. The causal influence of these concepts on the model’s output can be evaluated through latent ablation.
Figure 2: Attention-based concept re-ranking . A. SAEs return latent activations for each image patch. B. Image-level activations can be computed by naïvely averaging activations across all patches, or C. by applying a weighted average where the text-to-image attention map serves as the weighting coefficient, promoting concepts relevant to the text tokens to the top of the ranking. The bottom row shows the top-ranked concept obtained from the corresponding method.
Figure 3: Case studies. A. Limited cross-alignment, B. misleading visual cues, and C. unused hidden cues can cause wrong answer.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Complexity of visual concepts across layers. Early layers capture simple visual components, including position and color, mid layers capture object-level shape and patterns, and late layers understand scene-level events.
Figure 5: Token-Latent heatmap visualization. Raw values are difficult to analyze, so we normalize in column-wise to show if the latent is specifically attended by certain tokens or by overall tokens. Moreover, we cluster and sort by activation correlation in column-wise, using hierarchical clustering.
Figure 6: Illustration on how SAE latent statistics are computed. After training the SAE, we pass the dataset through the model and compute each latent’s mean activation and sparsity, yielding a compact summary of its activation distribution. Right: Based on the mean activation and sparsity, we filter out latents in top k % of each axis treating them as noisy latents.
Figure 7: Causal analysis example. Given an image and a prompt to describe it, the VLM generates a caption and VisualScratchpad produces a token–latent heatmap. Latents are re-ranked using token attention and clustered based on activation similarity. Each cluster roughly corresponds to a main topic in the caption. By zeroing out the latents in each cluster and reconstructing the representations, we observe how the outputs change relative to the original. Ablating latent clusters that correspond to distinct semantic topics removes those topics from the generated caption, demonstrating their causal role in shaping the model’s predictions.
Figure 8: Three main components of VisualScratchpad include (1) exploring SAE latents, (2) model inference with SAE (vision-question answering using VLMs and image classification using CLIP), and (3) close-up observation on activated concepts.
Figure 9: Control pannels of VisualScratchpad. Users can adjust visualization settings for spatial attribution masks, configure options for activation-value plots, and customize heatmap displays.
Figure 10: Details for Case 2. We iterated for two turns to remove sitting-related concepts.
Vision-language models can describe an image with remarkable accuracy, yet a more fundamental question remains unanswered: what visual information actually drives their answers? In this work, we investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region. Extending the analysis to larger vision-language models reveals a similar pattern across models and corruption settings, suggesting that strong multimodal performance does not necessarily imply spatially localized causal representations. We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures. Together, our experiments expose a gap between seeing, using, and reasoning over visual structure, and provide a causal framework for studying how visual information is transformed, preserved, and ultimately used by modern vision-language models.
Naren Kumar S, Tirth Bhatt, Mayank Singh
LINGO Research Group, Indian Institute of Technology Gandhinagar, India
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.
Vision-language Models (VLMs), despite achieving strong performance on multimodal benchmarks, often misinterpret straightforward visual concepts that humans identify effortlessly, such as counting, spatial reasoning, and viewpoint understanding. Previous studies manually identified these weaknesses and found that they often stem from deficits in specific skills. However, such manual efforts are costly, unscalable, and subject to human bias, which often overlooks subtle details in favour of salient objects, resulting in an incomplete understanding of a model's vulnerabilities. To address these limitations, we propose a Reinforcement Learning (RL)-based framework to automatically discover the failure modes or blind spots of any ``candidate VLM'' on a given data distribution without human intervention. Our framework trains a questioner agent that adaptively generates queries based on the candidate VLM's responses to elicit incorrect answers. Our approach increases question complexity by focusing on fine-grained visual details and distinct skill compositions as training progresses, consequently identifying novel failure modes in which VLMs struggle. We demonstrate the broad applicability of our framework by showcasing its generalizability across various model combinations.
Kanishk Jain, Qian Yang, Shravan Nayak +3
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2