cs.LGMar 11, 2026

Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT

Authors: Sai V R Chereddy

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The paper explores how video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor: Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.

Figures & tables

Figure 1: Qualitative visualization of CLS Token attention (Layer 10, Head 8) for the (a) ’Strike’ and (b) ’Gutter’ videos. This head functions as one of the semantic ”outcome-detectors”.
Figure 2: Contribution of each layer towards the final logits of target class across frames for (a)strike and (b)gutter runs.
Figure 3: Token-wise heatmap across frames in ”strike run” indicating contribution of each token to the final predicted output class
Figure 4: Attention visualisation heatmap for [CLS] token at Layer 9, Head 8.
Figure 5: The average L2 norm of the activation delta between the ”strike” and ”gutter” runs. The sharp rise from layer 5 onwards shows the amplification cascade.
Prediction RankOriginal Prediction (Logit)Ablated Prediction (Logit)Change in Logits
1Bowling (16.9954)Bowling (16.6583)-0.3371
2Throwing Ball (9.0371)Throwing Ball (9.1341)0.0970
3Playing Cricket (6.2185)Playing Cricket (6.4343)0.2158
4High Kick (5.9146)High Kick (5.7234)-0.1912
5Dancing Ballet (5.7508)Dancing Ballet (5.7579)0.0072
Table 1: Comparison of logits before and after ablation in ”strike run”.
Prediction RankOriginal Prediction (Logit)Ablated Prediction (Logit)Change in Logits
1Bowling (16.5200)Bowling (16.5044)-0.0156
2Throwing Ball (6.3758)Throwing Ball (6.6308)0.2551
3Headbutting (4.8109)Headbutting (5.1653)0.3544
4Playing Cricket (4.6217)Playing Cricket (4.6968)0.0751
5Throwing Baseball (4.2289)Throwing Baseball (4.3045)0.0755
Table 2: Comparison of logits before and after ablation in ”gutter run”.
Figure 6: The layer-wise differentiating accuracy of the trained probe showed 100% accuracy in distinguishing between the ”strike” and ”gutter” runs, acting as a superficial fingerprint scanner.
LayerComponentLabel% Recovery
4AttentionL4 Attention54.41
4MLPL4 MLP60.17
5AttentionL5 Attention50.22
5MLPL5 MLP57.49
6AttentionL6 Attention43.62
6MLPL6 MLP49.11
Table 3: Patching Experiment Results showing causal effects per layer and component.
Figure 7: Causal effect of patching individual components on the layer 11 signal. MLP blocks are causally sufficient to create the outcome signal.
More in this topic

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

Explore similar work

CardsList
Apr 15, 2026

Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers

Transparency of neural networks' internal reasoning is at the heart of interpretability research, adding to trust, safety, and understanding of these models. The field of mechanistic interpretability has recently focused on studying task-specific computational graphs, defined by connections (edges) between model components. Such edge-based circuits have been defined in the context of large language models, yet vision-based approaches so far only consider neuron-based circuits. These tell which information is encoded, but not how it is routed through the complex wiring of a neural network. In this work, we investigate whether useful mechanistic circuits can be identified through computational graphs in vision transformers. We propose an effective method for Automatic Visual Circuit Discovery (Vi-CD) that recovers class-specific circuits for classification, identifies circuits underlying typographic attacks in CLIP, and discovers circuits that lend themselves for steering to correct harmful model behavior. Overall, we find that insightful and actionable edge-based circuits can be recovered from vision transformers, adding transparency to the internal computations of these models.

Nina Żukowska, Wolfgang Stammer, Bernt Schiele +1

Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany

Transformer InterpretabilityMechanistic InterpretabilityCircuit DiscoveryVision TransformerVLM Interpretability
AskPDF
Apr 20, 2026

Decision-Aware Attention Propagation for Vision Transformer Explainability

Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet their prediction process remains difficult to interpret because information is propagated through complex interactions across layers and attention heads. Existing attention based explanation methods provide an intuitive way to trace information flow. However, they rely mainly on raw attention weights, which do not explicitly reflect the final decision and often lead to explanations with limited class discriminability. In contrast, gradient based localization methods are more effective at highlighting class specific evidence, but they do not fully exploit the hierarchical attention propagation mechanism of transformers. To address this limitation, we propose Decision-Aware Attention Propagation (DAP), an attribution method that injects decision-relevant priors into transformer attention propagation. By estimating token importance through gradient based localization and integrating it into layer wise attention rollout, the method captures both the structural flow of attention and the evidence most relevant to the final prediction. Consequently, DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods. Extensive experiments across Vision Transformer variants of different model scales show that DAP consistently outperforms existing baselines in both quantitative metrics and qualitative visualizations, indicating that decision aware propagation is an effective direction for improving ViT interpretability.

Sehyeong Jo, Gangjae Jang, Haesol Park

Aarhus University · University of Colorado Boulder · KIST

Transformer InterpretabilityVision TransformerNeural Network Interpretability
AskPDF
Aug 28, 2025

ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering

Existing Causal-Why Video Question Answering (VideoQA) models often struggle with higher-order reasoning, relying on opaque, monolithic pipelines that entangle video understanding, causal inference, and answer generation. These black-box approaches offer limited interpretability and tend to depend on shallow heuristics. We propose a novel, modular paradigm that explicitly decouples causal reasoning from answer generation, introducing natural language causal chains as interpretable intermediate representations. Inspired by human cognitive models, these structured cause-effect sequences bridge low-level video content with high-level causal reasoning, enabling transparent and logically coherent inference. Our two-stage architecture comprises a Causal Chain Extractor (CCE) that generates causal chains from video-question pairs, and a Causal Chain-Driven Answerer (CCDA) that derives answers grounded in these chains. To address the lack of annotated reasoning traces, we introduce a scalable method for generating accurate causal chains from existing datasets. We construct human verified causal chains for 46K samples. We also propose CauCo, a new evaluation metric for causality-oriented captioning. Experiments on three large-scale benchmarks demonstrate that our approach not only outperforms state-of-the-art models, but also yields substantial gains in explainability, user trust, and generalization -- positioning the CCE as a reusable causal reasoning engine across diverse domains. Project page: https://paritoshparmar.github.io/chainreaction/

Paritosh Parmar, Eric Peh, Basura Fernando

Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

Video QAMultimodal CoT ReasoningCausal ReasoningCausal ExplanationCausal Reasoning in Videos
AskPDF