Causal Attention

Causal attention in machine learning focuses on improving model performance and interpretability by explicitly incorporating causal relationships between inputs and outputs, rather than relying solely on statistical correlations. Current research investigates how to leverage causal attention within various architectures, including transformers and recurrent neural networks, to mitigate biases, enhance generalization, and improve efficiency in tasks such as language modeling, image recognition, and time series imputation. This work is significant because it addresses limitations of traditional attention mechanisms, leading to more robust, reliable, and explainable AI systems with broader applicability across diverse domains.

Papers

September 18, 2023

Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesis
Tianyi Song, Jiuxin Cao, Kun Wang, Bo Liu, Xiaofeng Zhang
Text to Image Story Generation Visual Storytelling Causal Attention History Representation

June 1, 2023

Faster Causal Attention Over Large Sequences Through Sparse Flash Attention
Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, François Fleuret
Transformer Based Language Model Long Sequence Transformer Language Model Causal Attention SParse ATtention

January 22, 2023

Causality-based Dual-Contrastive Learning Framework for Domain Generalization
Zining Chen, Weiqiu Wang, Zhicheng Zhao, Aidong Men
Contrastive Learning Domain Generalization Causal Attention Dual Contrastive Learning

November 6, 2022

Knowledge is Power: Understanding Causality Makes Legal judgment Prediction Models More Generalizable and Robust
Haotian Chen, Lingwei Zhang, Yiran Liu, Fanchao Chen, Yang Yu
Real Power Knowledge Based Causal Pattern Legal Judgment Prediction Causal Knowledge Causal Attention Judgment Prediction

October 14, 2022

CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
Jun Zhang, Shuyang Jiang, Jiangtao Feng, Lin Zheng, Lingpeng Kong
Long Sequence Efficient Attention Causal Attention

October 9, 2022

Fine-Tuning Pre-trained Transformers into Decaying Fast Weights
Huanru Henry Mao
Language Model Fine Tuning Transformer Megatron Decepticons Autoregressive Transformer Token Generation Causal Attention

March 30, 2022

Transformer Language Models without Positional Encodings Still Learn Positional Information
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, Omer Levy
Transformer Language Model Positional Encoding Causal Language Positional Embeddings Positional Information Causal Attention Causal Attention Mask

December 30, 2021

Causal Attention for Interpretable and Generalizable Graph Classification
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, Tat-Seng Chua
Graph Classification Causal Effect Estimation Interpretable Way Causal Feature Causal Attention