Causal Transformer

Causal transformers are autoregressive models leveraging the transformer architecture to predict future elements in a sequence, based solely on past observations. Current research focuses on applying this framework to diverse sequential data, including robotic control, time-series analysis, and natural language processing, often employing variations like Chunking Causal Transformers or incorporating causal understanding modules to improve performance and generalization. This approach offers a powerful tool for modeling complex temporal dependencies and causal relationships, leading to advancements in various fields ranging from robotics and healthcare to cybersecurity and materials science.

Papers

November 27, 2023

Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation
Sunandini Sanyal, Ashish Ramayee Asokan, Suvaansh Bhambri, Pradyumna YM, Akshay Kulkarni, Jogendra Nath Kundu, R Venkatesh Babu
Source Free Domain Adaptation Causal Transformer Causal Factor

November 15, 2023

Striped Attention: Faster Ring Attention for Causal Transformers
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, Jonathan Ragan-Kelley
Causal Transformer Efficient Attention Causal Attention Memory Bottleneck

November 14, 2023

Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
Hongxuan Zhang, Zhining Liu, Yao Zhao, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, Guihai Chen
Future Reasoning Glance Annotation Top Two Answer Causal Transformer Context Window Parallel Decoding Autoregressive Decoding

October 5, 2023

Molecule Design by Latent Prompt Transformer
Deqian Kong, Yuhao Huang, Jianwen Xie, Ying Nian Wu
Molecular Design Latent Vector Molecular Generation Causal Transformer Prompt Transformer

October 3, 2023

CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery
Yuxiao Cheng, Ziqian Wang, Tingxiong Xiao, Qin Zhong, Jinli Suo, Kunlun He
Time Series Causal Discovery Causal Graph Benchmark Platform Causal Transformer Ground Truth Causal

April 17, 2023

Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning
Siyu Wang, Xiaocong Chen, Dietmar Jannach, Lina Yao
Reinforcement Learning Recommender System Offline Reinforcement Learning Causal Transformer Learning Based Recommender System

March 23, 2023

LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models
Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn, Jan Skoglund, Zalán Borsos, Neil Zeghidour, Marco Tagliasacchi
Causal Transformer Entropy Coding Residual Quantization Causal Convolution Low Bitrate

March 14, 2023

CAT: Causal Audio Transformer for Audio Classification
Xiaoyu Liu, Hanlin Lu, Jianbo Yuan, Xinyu Li
Audio Classification Attention Block Causal Transformer Attention Based Transformer Audio Modeling

March 6, 2023

Real-World Humanoid Locomotion with Reinforcement Learning
Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, Koushil Sreenath
Reinforcement Learning Humanoid Robot Model Free Reinforcement Learning Causal Transformer Humanoid Locomotion Classical Control

October 2, 2022

Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks
Yuxuan Li, James L. McClelland
Transformer Megatron Decepticons Latent Representation Attention Layer Transformer Network Layer Transformer Task Specification Causal Transformer Systematic Generalization Emergent Structure

September 22, 2022

PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, Ashish Kapoor
Robotics Domain Causal Transformer Autoregressive Neural Network Robot Data Transformer Based Generative Robot Representation

July 6, 2022

Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies
Zehan Li, Haoran Miao, Keqi Deng, Gaofeng Cheng, Sanli Tian, Ta Li, Yonghong Yan
Automatic Speech Recognition Causal Transformer Streaming End to End

April 14, 2022

Causal Transformer for Estimating Counterfactual Outcomes
Valentyn Melnychuk, Dennis Frauen, Stefan Feuerriegel
Time Varying Causal Transformer Counterfactual Feature

March 29, 2022

Treatment Learning Causal Transformer for Noisy Image Classification
Chao-Han Huck Yang, I-Te Danny Hung, Yi-Chieh Liu, Pin-Yu Chen
Image Classification Variational Inference Causal Transformer Real World Noisy Datasets

February 4, 2022

Generative Modeling of Complex Data
Luca Canale, Nicolas Grislain, Grégoire Lothe, Johan Leduc
Generative Modeling Benchmark Datasets Causal Transformer Synthetic Tabular Data Hierarchical Datasets

November 19, 2021

Generalized Decision Transformer for Offline Hindsight Information Matching
Hiroki Furuta, Yutaka Matsuo, Shixiang Shane Gu
Reinforcement Learning Decision Transformer Causal Transformer Hindsight Instruction Expressive Policy