Autoregressive Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 157% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 91

All topics

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2Process-Level Supervision Autoregressive Language Models
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Maps
Sep 30, 2026Yeongmin Kim, Arnaud Doucet, Andrew Campbell +3Flow Map Autoregressive Language Models
Google DeepMind · Work done as a Student Researcher at Google DeepMind Amsterdam
Uncovering Uncontrolled Repetition through Residual Stream Dynamics
Sep 30, 2026Yuanhe Zhang, Xinyao Zhou, Haoran Gao +5Repetition Autoregressive Language Models
Beijing University of Posts and Telecommunications · JIUTIAN Research · Nanyang Technological University +1
FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matching
Sep 28, 2026Thanh-Long V. Le, Steven Walton, Seunghyun Yoon +4Image Editing Multi-Reference Image Generation
Adobe Research · KAIST
NavJev: Efficient Vision-Language Navigation via Action-Centric Visual Compression and Discriminative Action-Semantic Memory
Sep 28, 2026Kai Sheng, Liuyi Wang, Jinlong Li +3Vision-Language Navigation Diffusion-Based Vision-Language-Actions
College of Electronic and Information Engineering, Tongji University, Shanghai, China
SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining
Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Batch Large Language Model Pretraining
Peking University · Nanjing University
Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues
Sep 24, 2026Wenxuan Wu, Shuhan Zhang, Shuai Wang +1Target Speaker Extraction Audio-Visual Consistency
Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents
Sep 21, 2026Dongming Jiang, Yi Li, Bingzhe LiAgentic Memory Artificial Intelligence Agents
Department of Computer Science, The University of Texas at Dallas
Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
Sep 17, 2026Leonid Sinev, Ilya Koziev, Vladislav LeshchukAutoregressive Language Models Diffusion Language Models
M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models
Sep 16, 2026Chunpu Xu, Zhixuan Liang, Yuhao Zhang +6Discrete Action Tokenizers Diffusion-Based Vision-Language-Actions
The Hong Kong Polytechnic University, HongKong SAR, China · Shanghai AI Laboratory, Shanghai, China · The University of Hong Kong, HongKong SAR, China +2
When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models
Sep 15, 2026Julio C. Amador Diaz LopezConfidence Calibration Autoregressive Language Models
SiftyML, LTD London, United Kingdom
Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning
Sep 14, 2026Sophia Tang, Shiyi WangDiffusion Language Models Autoregressive Language Models
University of Pennsylvania · Harvard University · Kempner Institute +1
Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching
Sep 14, 2026Jiayang Gu, Zheng Fang, Lichaun Xiang +3Timestep Autoregressive Language Models
University of Warwick · Bytedance
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinLLM Inference Optimization Autoregressive Language Models
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
CARTS: Contextual Autoregressive Rank Transcoding Steganography for Full-Capacity Keyed Text Encoding
Sep 12, 2026Wissam Ghantous, Alexander V. MantzarisSteganography Passage Encoding
University of Central Florida Orlando, Florida, USA
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
Sep 11, 2026NCP Team, Jiaqi Cao, Chiyu Chen +25Autoregressive Language Models Latent Space
Shanghai AI Lab; LUMIA Lab, Shanghai Jiao Tong University
Separating Stream Stability from Long-Term Recall in Language Models
Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Large Language Model Memory Efficient Long-Context Inference
School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology
Line-Coupled Language Model
Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Autoregressive Language Models Language Modeling
University of California, Riverside Riverside, CA, USA
ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Dataset Distillation Knowledge Distillation
VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1
Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
Aug 31, 2026Tongfei Guo, Lili SuAutonomous Driving Autoregressive Language Models
Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA.
AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning
Aug 30, 2026Hanjun Luo, Qiushi Liu, Jingya Zhang +8LLM Reasoning Strategies Autoregressive Language Models
New York University · New York University Abu Dhabi · University of Washington Seattle +4
StepKV: Step-Aware KV Cache Compression for LLM Agents
Aug 26, 2026Boyu Feng, Jiahong Liu, Yifan Li +7Key-Value Cache Compression Cache
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd
DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
Aug 13, 2026Tianyi Li, Yaxin Luo, Xinyi Shang +1Speculative Decoding Drafts
VILA Lab, MBZUAI
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanSpeculative Decoding LLM Inference Optimization
Kyungpook National University Daegu, Republic of Korea
Reducing Pretraining-Generation Mismatch in Diffusion Language Models
Aug 10, 2026Xiaocheng Lu, Huabin Liu, Song Guo +1Diffusion Language Models Large Language Model Pretraining
Inclusion AI · The Hong Kong University of Science and Technology
Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression
Aug 4, 2026Angelo Nardone, Paolo FerraginaLarge Language Model Compression Data Compression Methods
University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy
Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs
Aug 1, 2026Zhenhong Sun, Hanqing Zhao, Yatao Bian +7Mathematical Reasoning Diffusion Language Models
Australian National University Canberra, Australia · Nanyang Technological University Singapore · National University of Singapore Singapore +2
From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Jul 29, 2026Seunggeun Kim, Jaeyeon Kim, Taekyun Lee +4Masked Diffusion Language Models Autoregressive Language Models
University of Texas at Austin · Harvard University
UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
Jul 27, 2026Xiaoyi Jiang, Jingyuan Li, Yixuan Jiang +4Autoregressive Language Models Autoregressive Model
Tsinghua University · MathonAI · Beijing Institute of Mathematical Sciences and Applications +1
Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning
Jul 23, 2026Rogerio Guimaraes, Pietro PeronaDiffusion Models Inference-Time
California Institute of Technology, Pasadena CA 91125, USA
Total Variation Distance Estimation in Autoregressive Models
Jul 21, 2026Eric Price, Kevin Tian, Zhiyang Xun +1Autoregressive Language Models Kullback-Leibler Divergence
University of Texas at Austin
Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
Jul 19, 2026Libo Sun, Po-Wei Harn, Zewei Zhang +2Key-Value Cache Quantization Regularization
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL 36830, USA · Department of Information Management, National Central University, Taoyuan 320317, Taiwan
Constraint-Anchored Reasoning Traces
Jul 18, 2026Zehua Cheng, Wei Dai, Jiahao SunLLM Reasoning Strategies Multimodal Large Language Models
Department of Computer Science University of Oxford United Kingdom · FLock.io United Kingdom
Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models
Jul 17, 2026Wenjie Fan, Bin Ma, Dong LiAutoregressive Language Models Repetition
Yotta Labs
A Better Start for Language Models: Domain-Conditional Position Offsets
Jul 17, 2026Ye QiaoPretraining Autoregressive Language Models
University of California, Irvine
Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model
Jul 13, 2026Xinghang Li, Jun Guo, Qiwei Li +21Indoor Scene Generation Human Motion Generation
Xiaomi Robotics
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding Autoregressive Language Models
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation
Jul 9, 2026Ingrid Petrova, Luan VejsiuIterative Self-Correction Autoregressive Generation
European University of Tirana
A Practical Investigation of Training-free Relaxed Speculative Decoding
Jul 9, 2026Guoxuan Xia, Luka Ribar, Paul BalancaSpeculative Decoding Multi-Token Prediction
Graphcore
Trees from Marginals: Autoregressive drafting with factorized priors
Jul 7, 2026Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy +1Autoregressive Decoding Drafts
Artur Chakhvadze1
Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
Jul 7, 2026Yonggan Fu, Lexington Whalen, Abhinav Garg +23Autoregressive Language Models Language Modeling
NVIDIA · Georgia Tech · Work done during internships at NVIDIA +3
Depth Exploration for LLM Decoding
Jun 28, 2026Weisi Yang, Zipeng Sun, Stephen XiaLarge Language Model Decoding Autoregressive Language Models
Northwestern University · MILA – Quebec AI Institute · McGill University
Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks
Jun 25, 2026Yunqi Xue, Zhijiang Li, Philip Torr +1Autoregressive Image Generation Autoregressive Generation
School of Information Management, Wuhan University, Wuhan, China · Torr Vision Group, University of Oxford, Oxford, United Kingdom
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
Jun 25, 2026Muhammad AhmedKv-Cache Management Large Language Model Serving
Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection
Jun 23, 2026Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-ZakenMachine-Generated Text Detection Natural Language Processing
Dream Security Ltd. Tel Aviv, Israel
AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation
Jun 22, 2026Huadai Liu, Kaicheng Luo, Wen Wang +4Modern Generative Audio Models Acoustic Latent Space
Hong Kong University of Science and Technology (HKUST) · Tongyi Fun Team, Alibaba Group
Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
Jun 16, 2026Wujian Peng, Lingchen Meng, Yuxuan Cai +7Visual Tokenizers Autoregressive Language Models
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-Of-Experts Large Language Models Mixture-Of-Experts
AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1
Circuit Tracing in Autoregressive Protein Language Models
Jun 14, 2026Darin Tsui, William Deinzer, Daniel Saeedi +1Mechanistic Interpretability Molecular Generation
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atltanta, GA.
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengFull-Duplex Speech Models Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling
Jun 9, 2026Zhiwei Tang, Yuanyu He, Yizheng Han +4Autoregressive Language Models Autoregressive Generation
1DAMO Academy, Alibaba Group · 2Hupan Lab · 3Zhejiang University +1
End-to-End Training for Discrete Token LLM based TTS System
Jun 8, 2026Changfeng Gao, Yong Ren, Jun Yuan +3Flow-Matching Text-To-Speech Text-To-Speech Synthesis
Tencent Yuanbao, China
De novo molecular generation with optical property preconditioning at the token level
Jun 6, 2026Haozhe Huang, Manuel Gonzalez Lastre, Hyun Suk Park +3Molecular Generation Autoregressive Language Models
Department of Computer Science, University of Toronto, 40 St George St., Toronto, ON M5S 2E4, Canada · 2Vector Institute for Artificial Intelligence, W1140-108 College St., Schwartz Reisman Innovation Campus, Toronto, ON M5G 0C6, Canada · 3Departamento de Física Teórica de la Materia Condensada, Universidad Autónoma de Madrid, 28049, Madrid, Spain. +6
Latent Reasoning with Normalizing Flows
Jun 4, 2026Guancheng Tu, Xiangjun Fu, Suhao Yu +5Efficient Latent Reasoning Chain-of-Thought Reasoning
University of Pennsylvania · UC San Diego · Meta
Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance
Jun 4, 2026Gizem Yüce, Giorgos Nikolaou, Nicolas FlammarionLarge Language Model Unlearning Autoregressive Language Models
Theory of Machine Learning Lab, EPFL
Data Attribution in Large Language Models via Bidirectional Gradient Optimization
Jun 3, 2026Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar +1Large Language Model Training Autoregressive Language Models
ETH Zurich
ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
May 29, 2026Rodney Lafuente-MercadoCredit Assignment Large Language Model Reinforcement Learning
Scale AI
Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement
May 29, 2026Riju Marwah, Ritvik Garimella, Vishal Pallagani +3Large Language Models Fail Autoregressive Language Models
Guru Gobind Singh Indraprastha University, India · Artificial Intelligence Institute, University of South Carolina, USA · Indian Institute of Technology, Kanpur, India +1
SuperVoxelGPT: Adaptive and Ordered 3D Tokenization for Autoregressive Shape Generation
May 28, 2026Yuan Li, Congyi Zhang, Xifeng Gao +1Voxel Autoregressive Generation
University of Texas at Dallas, USA · LightSpeed, USA
$D^2$ -Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signals
May 25, 2026Aoxi Liu, Yupeng Chen, James Oldfield +5Large Language Model Safety Diffusion Language Models
Torr Vision Group, University of Oxford · The Chinese University of Hong Kong, Shenzhen

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

Andreea Dutulescu, Stefan Ruseti, Mihai Masala, Traian Rebedea+1

National University of Science and Technology POLITEHNICA Bucharest · NVIDIA

Process-Level Supervision Autoregressive Language Models

Open paper AskPDF arXiv

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter

β

decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Momentum

Related topics

Latest papers 91