Autoregressive Model

Autoregressive models are a class of generative models that predict sequential data by probabilistically modeling the next element based on preceding ones. Current research focuses on enhancing their controllability for tasks like image and 3D shape generation, improving their efficiency through parallel processing and non-autoregressive techniques, and applying them to diverse domains including time series forecasting, speech recognition, and even scientific modeling of physical systems. These advancements are driving improvements in various applications, from more accurate weather forecasting and efficient speech recognition to novel approaches in drug discovery and materials science.

Papers

August 2, 2024

TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generation
Yicheng Lin, Dandan Zhang, Yun Liu
Reinforcement Learning Autoregressive Model TCR Sequence

July 25, 2024

Physics-informed nonlinear vector autoregressive models for the prediction of dynamical systems
James H. Adler, Samuel Hocking, Xiaozhe Hu, Shafiqul Islam
Human Prediction Physic Informed Neural Network Dynamical System Autoregressive Model Physic Informed Machine Learning Vector Autoregression Predator Prey

July 24, 2024

Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation
Yongqi Li, Hongru Cai, Wenjie Wang, Leigang Qu, Yinwei Wei, Wenjie Li, Liqiang Nie, Tat-Seng Chua
Faithful Generation Autoregressive Model Cross Modal Retrieval Visual Token Text to Image Retrieval

July 23, 2024

Bayesian Autoregressive Online Change-Point Detection with Time-Varying Parameters
Ioanna-Yvonni Tsaknaki, Fabrizio Lillo, Piero Mazzarisi
Time Series Autoregressive Model Time Varying Change Point Detection Regime Switching

July 22, 2024

Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
Yuchen Li, Alexandre Kirchmeyer, Aashay Mehta, Yilong Qin, Boris Dadachev, Kishore Papineni, Sanjiv Kumar, Andrej Risteski
Text Generation Autoregressive Model Masked Language Common Pitfall Autoregressive Language Model Practical Guide Non Autoregressive Promise Theoretical Framework

July 8, 2024

ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
Ethan Chern, Jiadi Su, Yan Ma, Pengfei Liu
Large Multimodal Model Autoregressive Model Multimodal Generation Interleaved Image Text Generation Multimodal Generation Capability

July 1, 2024

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
Qi Zhang, Tianqi Du, Haotian Huang, Yifei Wang, Yisen Wang
Autoregressive Model Content Generation Empirical Comparison Masked Pre Training

June 19, 2024

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference
Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum
Autoregressive Model LLM Inference Tree Specie Non Autoregressive Autoregressive Decoding Parallel Inference

June 17, 2024

Autoregressive Image Generation without Vector Quantization
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, Kaiming He
Autoregressive Model Vector Quantization Autoregressive Image Generation Diffusion Loss Vector Quantized

June 12, 2024

DocSynthv2: A Practical Autoregressive Modeling for Document Generation
Sanket Biswas, Rajiv Jain, Vlad I. Morariu, Jiuxiang Gu, Puneet Mathur, Curtis Wigington, Tong Sun, Josep Lladós
Autoregressive Model Layout Generation KG to Text Document Generation

June 11, 2024

DeformTime: Capturing Variable Dependencies with Deformable Attention for Time Series Forecasting
Yuxuan Shu, Vasileios Lampos
Time Series Forecasting Multivariate Time Series Autoregressive Model Multivariate Time Series Forecasting Deformable Attention

June 5, 2024

LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes
Trung Dang, David Aponte, Dung Tran, Kazuhito Koishida
Autoregressive Model Generative Language Model Neural Audio Zero Shot Text to Speech Audio Token Discrete Code

May 28, 2024

Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass
Ethan Shen, Alan Fan, Sarah M. Pratt, Jae Sung Park, Matthew Wallingford, Sham M. Kakade, Ari Holtzman, Ranjay Krishna, Ali Farhadi, Aditya Kusupati
Autoregressive Model N Gram Greedy Decoding K$ Draft Multiple Generation Expensive Language Model

May 23, 2024

Distributed Speculative Inference of Large Language Models is Provably Faster
Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon, David Harel
Large Language Model Scientific Inference Autoregressive Model Inference Algorithm Speculative Exploration

May 21, 2024

What Have We Achieved on Non-autoregressive Translation?
Yafu Li, Huajian Zhang, Jianhao Yan, Yongjing Yin, Yue Zhang
Natural Language Autoregressive Model Human Annotation Non Autoregressive Translation Autoregressive Translation

May 20, 2024

Transfer Learning for Spatial Autoregressive Models with Application to U.S. Presidential Election Prediction
Hao Zeng, Wei Zhong, Xingbai Xu
Transfer Learning Autoregressive Model Spatial Dependence Residual Bootstrap

May 18, 2024

LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions
Victor Agostinelli, Sanghyun Hong, Lizhong Chen
Autoregressive Model Autoregressive Language Model Efficient Transformer Dynamic Reweighting Linear Transformer Speech to Text Translation Logical Proportion

May 9, 2024

Multi-Scale Dilated Convolution Network for Long-Term Time Series Forecasting
Feifei Li, Suhan Guo, Feng Han, Jian Zhao, Furao Shen
Time Series Autoregressive Model Long Term Time Series Forecasting Dilated Convolutional Neural Network Forecasting Benchmark

May 7, 2024

VAEneu: A New Avenue for VAE Application on Probabilistic Forecasting
Alireza Koochali, Ensiye Tahaei, Andreas Dengel, Sheraz Ahmed
Autoregressive Model Probabilistic Forecast Forecasting Performance VAE Model Probabilistic Time Series Forecasting Predictive Distribution

May 6, 2024

Functional Latent Dynamics for Irregularly Sampled Time Series Forecasting
Christian Klötergens, Vijaya Krishna Yalavarthi, Maximilian Stubbemann, Lars Schmidt-Thieme
Time Series Forecasting Autoregressive Model Differential Equation Latent State Latent Dynamic

Autoregressive Model

Papers

TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generation

Physics-informed nonlinear vector autoregressive models for the prediction of dynamical systems

Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation

Bayesian Autoregressive Online Change-Point Detection with Time-Varying Parameters

Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines

ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference

Autoregressive Image Generation without Vector Quantization

DocSynthv2: A Practical Autoregressive Modeling for Document Generation

DeformTime: Capturing Variable Dependencies with Deformable Attention for Time Series Forecasting

LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes

Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

Distributed Speculative Inference of Large Language Models is Provably Faster

What Have We Achieved on Non-autoregressive Translation?

Transfer Learning for Spatial Autoregressive Models with Application to U.S. Presidential Election Prediction

LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions

Multi-Scale Dilated Convolution Network for Long-Term Time Series Forecasting

VAEneu: A New Avenue for VAE Application on Probabilistic Forecasting

Functional Latent Dynamics for Irregularly Sampled Time Series Forecasting