Autoregressive Transformer

Latest papers

33 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Generative Modeling for Cascading Extremes via Autoregressive Transformers
Nov 3, 2026Miguel de Carvalho, Myung Won Lee, Jordan Richards +1Generative Modeling Time Series Tokenization
School of Mathematics and Maxwell Institute for Mathematical Sciences University of Edinburgh · CIDMA–Department of Mathematics Universidade de Aveiro · National Technical University of Athens Athena/Archimedes Research Centre
When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?
Oct 29, 2026Kağan Akman, Jake Feldman Starosta, Naci Saldi +1Autoregressive Transformer Sequence Modeling
Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada
Multi-Solution Inverse Design of Pixelated Patch Antennas Using an Autoregressive Transformer
Oct 29, 2026Young-Mok Yun, Seung-Won Shin, Jaeyong Chung +1Inverse Design Inverse Problems
Department of System Semiconductor Engineering, Yonsei University, Seoul 03722, South Korea.
ORBIT: Autoregressive Time Series Generation via Next Frequency-Band Prediction
Oct 19, 2026Chunyi Hou, Xiangfei Qiu, Hanyin Cheng +5Time Series Generation Frequency-Domain Time Series Modeling
School of Data Science and Engineering, East China Normal University, Shanghai, China · Chang’an University, Xi’an, China
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Oct 16, 2026Peter Cho-Ho Lam, Ziyi Wang, Zirui ZhouLanguage Model Controllability Prefix Tuning
Huawei Technologies Canada Co., Ltd.
Sampling the XY model with an attention-based neural network
Oct 16, 2026Piotr Białas, Christian D. Morales, Tomasz Stebel +1Boltzmann Sampling Autoregressive Transformer
Institute of Applied Computer Science, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · V. N. Karazin Kharkiv National University, 6 Nezalezhnosti Ave, Kharkiv 61022, Ukraine · Institute of Theoretical Physics, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland +1
PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
Oct 13, 2026Hannah Lawrence, Bodhi P. Vani, Ji Won Park +5Autoregressive Transformer Molecular Optimization
Genentech, South San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · Prescient Design, Genentech, Basel, Switzerland +1
Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Sep 30, 2026Yilun Liu, Yi Zhang, Ganyu Wu +5Dynamical Systems Chaotic Dynamical Systems
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · Technical University of Munich
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
Sep 28, 2026Julian Boesch, Andrew Wee, Alexander StranzlIn-Context Retrieval Recurrent Transformers
Purdue University · Obit Research · State University of New York at Stony Brook.
Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
Industrial Kinematic Trajectory Model (IKTM): Coordinate-Free Autoregressive Generator
Aug 27, 2026Max Amiri, David EyersAutoregressive Transformer Trajectory Generation
School of Computing University of Otago Dunedin, New Zealand
Clearing the Underbrush: AI-Enhanced RF Interference Suppression
Aug 25, 2026Rahul Jain, Pierre Trepagnier, Rick Gentile +2Finite Scalar Quantization Autoregressive Transformer
MIT Lincoln Laboratory, Lexington, MA 02421, USA
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
Aug 19, 2026Wenbo Zhang, Xiang RenEfficient Transformer Inference Cross-Layer Attention
University of Southern California
Full-bandwidth transformer
Aug 9, 2026Xi Wang, Ziyang Cai, Zheng Zhan +5Autoregressive Language Modeling Recurrent Transformers
Johns Hopkins University · Princeton University · Microsoft
Maglev: Sliding Recurrent Memory
Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language Models Next-Token Prediction
The University of Texas at Austin
Multimodal Auto-regressive Transformer Surrogate for Modeling Variable Operations and Quantifying Uncertainty in Geological Carbon Storage
Jul 26, 2026Yifu Han, Louis J. DurlofskyNeural Surrogate Modeling Uncertainty Quantification
Department of Energy Science and Engineering, Stanford University, Stanford, CA, 94305, USA
CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics
Jul 25, 2026Xinhong Xu, Yimeng Zhang, Yuanlong ZhangTransformer Neural Decoding
Department of Automation, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Jul 16, 2026Ziyang Cai, Xingyu Zhu, Yihe Dong +2Recurrent Transformers LLM Reasoning
Princeton Language and Intelligence, Department of Computer Science Princeton University
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model Distillation Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Jun 2, 2026Zekun Qi, Xuchuan Chen, Dairu Liu +10Zero-Shot Learning Transformer
Tsinghua University · Galbot Inc. · Shanghai Jiao Tong University +2
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
UniT: Unified Geometry Learning with Group Autoregressive Transformer
May 20, 2026Haotian Wang, Yusong Huang, Zhaonian Kuang +4Transformer Autoregressive Transformer
Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China. · The National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, P.R.China. · Applied Science, Amazon.com, Inc., USA.
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeTransformer Expressivity Transformer
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · DEKE Lab, Renmin University of China, Beijing, China
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University
A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting
Apr 22, 2026Brooks Kinch, Xiaozhe Hu, Yilong Huang +6Neural Surrogate Modeling Time Series Transformers
Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Tufts University, Medford, MA, USA · Department of Mathematics and Cybernetics, SINTEF Digital, Oslo, Norway +3
Applied AI-Enhanced RF Interference Rejection
Apr 14, 2026Rahul Jain, Pierre Trepagnier, Rick Gentile +2Autoregressive Transformer Wireless Communications
MIT Lincoln Laboratory, Lexington, MA 02421, USA
MARS: Enabling Autoregressive Models Multi-Token Generation
Apr 8, 2026Ziqi Jin, Lei Wang, Ziwei Luo +1Multi-Token Prediction LLM Inference Acceleration
Nanyang Technological University · Singapore Management University · Uppsala University
Modeling Matches as Language: A Generative Transformer Approach for Counterfactual Player Valuation in Football
Mar 16, 2026Miru Hong, Minho Lee, Geonhee Jo +4Autoregressive Transformer Counterfactual Prediction
University of Seoul, Seoul, Republic of Korea · Saarland University, Saarbrücken, Germany · KAIST, Daejeon, Republic of Korea +2

Nov 3, 2026J/K move · Enter open · S save

Generative Modeling for Cascading Extremes via Autoregressive Transformers

Miguel de Carvalho, Myung Won Lee, Jordan Richards, Sotirios Sabanis

School of Mathematics and Maxwell Institute for Mathematical Sciences University of Edinburgh · CIDMA–Department of Mathematics Universidade de Aveiro · National Technical University of Athens Athena/Archimedes Research Centre

Generative Modeling Time Series Tokenization Extreme Value Theory Autoregressive Transformer

Open paper AskPDF arXiv

Extreme events often occur sequentially, from earthquake-tsunami cascades to successive extreme losses in financial markets. We introduce an extremal token representation that encodes multivariate observations as a categorical sequence. A quiet token records the absence of threshold exceedances; otherwise, the token identifies the exceeding component with the largest marginal probability-scale score. The representation is invariant to componentwise strictly increasing transformations. It preserves the identity and temporal ordering of dominant exceedances but discards their magnitudes and the identities of nondominant simultaneous exceedances. A decoder-only transformer estimates the history-dependent next-token kernel on this representation, and recursive sampling generates future token sequences from the induced joint predictive law. We amortize our generative model by pretraining across simulated process families with varying parameters and dependence persistence. Next-token predictions are evaluated against exact conditional probabilities for held-out categorical processes and particle filter references for heavy-tailed vector autoregressive processes. Applied to ten-second returns for five semiconductor stocks, the pretrained model supports zero-shot transfer, while a few epochs of adaptation improve agreement between observed and generated token frequencies and transition patterns. These results support task-specific tokenization and simulation-based pretraining for modeling which dimensions dominate multivariate extremes.