Autoregressive Transformer

Latest papers

28 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Sampling the XY model with an attention-based neural network
Oct 16, 2026Piotr Białas, Christian D. Morales, Tomasz Stebel +1Boltzmann Sampling Autoregressive Transformer
Institute of Applied Computer Science, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · V. N. Karazin Kharkiv National University, 6 Nezalezhnosti Ave, Kharkiv 61022, Ukraine · Institute of Theoretical Physics, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland +1
PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
Oct 13, 2026Hannah Lawrence, Bodhi P. Vani, Ji Won Park +5Autoregressive Transformer Molecular Optimization
Genentech, South San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · Prescient Design, Genentech, Basel, Switzerland +1
Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Sep 30, 2026Yilun Liu, Yi Zhang, Ganyu Wu +5Dynamical Systems Chaotic Dynamical Systems
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · Technical University of Munich
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
Sep 28, 2026Julian Boesch, Andrew Wee, Alexander StranzlIn-Context Retrieval Recurrent Transformers
Purdue University · Obit Research · State University of New York at Stony Brook.
Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
Industrial Kinematic Trajectory Model (IKTM): Coordinate-Free Autoregressive Generator
Aug 27, 2026Max Amiri, David EyersAutoregressive Transformer Trajectory Generation
School of Computing University of Otago Dunedin, New Zealand
Clearing the Underbrush: AI-Enhanced RF Interference Suppression
Aug 25, 2026Rahul Jain, Pierre Trepagnier, Rick Gentile +2Finite Scalar Quantization Autoregressive Transformer
MIT Lincoln Laboratory, Lexington, MA 02421, USA
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
Aug 19, 2026Wenbo Zhang, Xiang RenEfficient Transformer Inference Cross-Layer Attention
University of Southern California
Full-bandwidth transformer
Aug 9, 2026Xi Wang, Ziyang Cai, Zheng Zhan +5Autoregressive Language Modeling Recurrent Transformers
Johns Hopkins University · Princeton University · Microsoft
Maglev: Sliding Recurrent Memory
Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language Models Next-Token Prediction
The University of Texas at Austin
Multimodal Auto-regressive Transformer Surrogate for Modeling Variable Operations and Quantifying Uncertainty in Geological Carbon Storage
Jul 26, 2026Yifu Han, Louis J. DurlofskyNeural Surrogate Modeling Uncertainty Quantification
Department of Energy Science and Engineering, Stanford University, Stanford, CA, 94305, USA
CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics
Jul 25, 2026Xinhong Xu, Yimeng Zhang, Yuanlong ZhangTransformer Neural Decoding
School of Life Sciences Tsinghua University Beijing, China
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Jul 16, 2026Ziyang Cai, Xingyu Zhu, Yihe Dong +2Recurrent Transformers LLM Reasoning
Princeton Language and Intelligence, Department of Computer Science Princeton University
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model Distillation Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Jun 2, 2026Zekun Qi, Xuchuan Chen, Dairu Liu +10Zero-Shot Learning Transformer
Tsinghua University · Galbot Inc. · Shanghai Jiao Tong University +2
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
UniT: Unified Geometry Learning with Group Autoregressive Transformer
May 20, 2026Haotian Wang, Yusong Huang, Zhaonian Kuang +4Transformer Autoregressive Transformer
Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China. · The National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, P.R.China. · Applied Science, Amazon.com, Inc., USA.
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeTransformer Expressivity Transformer
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · DEKE Lab, Renmin University of China, Beijing, China
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University
A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting
Apr 22, 2026Brooks Kinch, Xiaozhe Hu, Yilong Huang +6Neural Surrogate Modeling Time Series Transformers
Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Tufts University, Medford, MA, USA · Department of Mathematics and Cybernetics, SINTEF Digital, Oslo, Norway +3
Applied AI-Enhanced RF Interference Rejection
Apr 14, 2026Rahul Jain, Pierre Trepagnier, Rick Gentile +2Autoregressive Transformer Wireless Communications
MIT Lincoln Laboratory, Lexington, MA 02421, USA
MARS: Enabling Autoregressive Models Multi-Token Generation
Apr 8, 2026Ziqi Jin, Lei Wang, Ziwei Luo +1Multi-Token Prediction LLM Inference Acceleration
Nanyang Technological University · Singapore Management University · Uppsala University
Modeling Matches as Language: A Generative Transformer Approach for Counterfactual Player Valuation in Football
Mar 16, 2026Miru Hong, Minho Lee, Geonhee Jo +4Autoregressive Transformer Counterfactual Prediction
University of Seoul, Seoul, Republic of Korea · Saarland University, Saarbrücken, Germany · KAIST, Daejeon, Republic of Korea +2
Thinking into the Future: Latent Lookahead Training for Transformers
Mar 3, 2026Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1Autoregressive Language Modeling Sequence Prediction
Apple
STAR-VAE: A Scalable Latent-Variable Transformer for Controllable Molecular Generation
Nov 4, 2025Bum Chul Kwon, Ben Shapira, Moshiko Raboh +7Molecular Optimization Transformer
IBM Research, Cambridge, MA, USA · IBM Research, Haifa, Israel · IBM Research, Yorktown Heights, NY, USA
Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
Oct 31, 2025Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai +3Autoregressive Language Modeling Sequence Prediction
National Taiwan University

Oct 16, 2026J/K move · Enter open · S save

Sampling the XY model with an attention-based neural network

Piotr Białas, Christian D. Morales, Tomasz Stebel, Dawid Zapolski

Institute of Applied Computer Science, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · V. N. Karazin Kharkiv National University, 6 Nezalezhnosti Ave, Kharkiv 61022, Ukraine · Institute of Theoretical Physics, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland+1

Boltzmann Sampling Autoregressive Transformer Phase Transitions

Open paper AskPDF arXiv

We present a transformer-based variational autoregressive network for sampling lattice systems with continuous degrees of freedom and apply it to the two-dimensional XY model near the Berezinskii-Kosterlitz-Thouless transition. The architecture is combined with physics-informed ingredients: locality-informed attention and approximate probabilities. The resulting sampler significantly outperforms existing neural samplers. We train the model on lattices up to

96\times96

, verify with a mode-dropping estimator that no parts of the configuration space are missed, and reproduce the energy, magnetization and helicity modulus obtained with the traditional Markov Chain Monte Carlo algorithm.