Non Autoregressive Transformer

Non-autoregressive transformers (NATs) are a class of neural network models designed to accelerate sequence generation tasks by predicting all output tokens simultaneously, unlike autoregressive models which generate sequentially. Current research focuses on improving NAT performance, particularly addressing challenges like capturing long-range dependencies and handling multi-modal data distributions, through techniques such as adaptive generation policies, diffusion-based normalization, and contrastive learning. These advancements are significant because they offer substantial speed improvements in applications like machine translation, speech recognition, and image synthesis, while striving to maintain comparable accuracy to their autoregressive counterparts.

Papers

November 11, 2024

ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis
Zanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han, Jiayi Guo, Zhiyuan Liu, Yuan Yao, Gao Huang
Critical Synthesis Visual Token Deep Representation Token Representation Temporal Interaction EfficientNet Algorithm Latent Token Non Autoregressive Transformer

September 26, 2024

Paraformer-v2: An improved non-autoregressive transformer for noise-robust speech recognition
Keyu An, Zerui Li, Zhifu Gao, Shiliang Zhang
Speech Recognition Token Embeddings Attention Based Encoder Decoder Non Autoregressive Transformer Non Autoregressive Automatic Speech Recognition

August 31, 2024

AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation
Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, Gao Huang
Image Generation Reward Model Token Generation Adaptive Policy ImageNet E Non Autoregressive Transformer Adversarial Reward

June 8, 2024

Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis
Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao, Gao Huang
Diffusion Model Image Synthesis ImageNet E Non Autoregressive Transformer

May 22, 2024

DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation
Weiting Tan, Jingyu Zhang, Lingfeng Shen, Daniel Khashabi, Philipp Koehn
Speech to Speech Translation Speech Feature Adaptive Normalization Non Autoregressive Transformer

January 9, 2024

Masked Audio Generation using a Single Non-Autoregressive Transformer
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi
Audio Generation Non Autoregressive Text to Audio Generation Audio Token Non Autoregressive Transformer

October 23, 2023

Non-autoregressive Streaming Transformer for Simultaneous Translation
Zhengrui Ma, Shaolei Zhang, Shoutao Guo, Chenze Shao, Min Zhang, Yang Feng
Low Latency Simultaneous Machine Translation Simultaneous Translation Non Autoregressive Transformer

October 19, 2023

Non-Autoregressive Sentence Ordering
Yi Bin, Wenhao Shi, Bin Ji, Jipeng Zhang, Yujuan Ding, Yang Yang
Non Autoregressive Encoder Decoder Framework Non Autoregressive Transformer

May 24, 2023

Beyond Individual Input for Deep Anomaly Detection on Tabular Data
Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan
Anomaly Detection Tabular Data Anomaly Score Tabular Datasets Deep Anomaly Detection Non Autoregressive Transformer

May 23, 2023

Optimizing Non-Autoregressive Transformers with Contrastive Learning
Chenxin An, Jiangtao Feng, Fei Huang, Xipeng Qiu, Lingpeng Kong
Contrastive Learning Non Autoregressive Autoregressive Transformer Multimodal Distribution Non Autoregressive Transformer

March 31, 2023

Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation
Min Liu, Yu Bao, Chengqi Zhao, Shujian Huang
Knowledge Distillation Neural Machine Translation Non Autoregressive Transformer Sequence Level Knowledge Distillation Non Autoregressive Neural Machine Translation

March 14, 2023

RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer
Pei Guo, Yisheng Xiao, Juntao Li, Min Zhang
Neural Machine Translation Neural Machine Translation Model Translation Benchmark Literal Translation Non Autoregressive Transformer Non Autoregressive Neural Machine Translation

March 11, 2023

SPOTR: Spatio-temporal Pose Transformers for Human Motion Prediction
Avinash Ajit Nargund, Misha Sra
Human Motion Prediction SpOT Robot Spatio Temporal Representation Future Motion Non Autoregressive Transformer Spatio Temporal Transformer

February 10, 2023

PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction
Ziji Zhang, Zhehui Wang, Rajesh Kamma, Sharanya Eswaran, Narayanan Sadagopan
Automatic Speech Recognition Automatic Speech Recognition System Language Correction Automatic Speech Recognition Error Correction Non Autoregressive Transformer

September 15, 2022

STPOTR: Simultaneous Human Trajectory and Pose Prediction Using a Non-Autoregressive Transformer for Robot Following Ahead
Mohammad Mahdavian, Payam Nikdel, Mahdi TaherAhmadi, Mo Chen
Robot Person Human Motion Prediction Pose Prediction Non Autoregressive Transformer Pixel Trajectory

June 13, 2022

On the Learning of Non-Autoregressive Transformers
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, Minlie Huang
LeArning Abstract Total Correlation Text Generation Model Novel Training Non Autoregressive Transformer

June 7, 2022

DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation
Sajad Norouzi, Rasa Hosseinzadeh, Felipe Perez, Maksims Volkovs
Machine Translation Self Distillation Multi Step Distilled Model Translation Accuracy Non Autoregressive Transformer Effective Distillation

May 16, 2022

Directed Acyclic Transformer for Non-Autoregressive Machine Translation
Fei Huang, Hao Zhou, Yang Liu, Hang Li, Minlie Huang
Transformer Based Autoregressive Transformer Non Autoregressive Machine Translation Non Autoregressive Transformer