Autoregressive Language Modeling

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 81

All topics

CNet: A Complex-Valued Deep Learning Framework with Wirtinger Autodifferentiation and FFT--Hadamard Convolution
Oct 6, 2026Marcel CrasmaruComplex-Valued Neural Networks Autoregressive Language Modeling
Generative modeling of intrinsically disordered protein regions by reinforcing sparse autoencoder features
Oct 1, 2026Jason X. Liu, Sebastian Ibarraran, Frank Hu +7Autoregressive Language Modeling Sparse Autoencoders
Stanford University
PassGPT+: Leveraging Linguistic Priors for Password Modeling
Sep 30, 2026Rajneesh Anand, Neeraj Lakshmanan, Masoud YariAutoregressive Language Modeling Diffusion Language Models
Department of Chemical and Biomolecular Engineering, Lehigh University, Bethlehem, PA, 18015, USA · Department of Computer Science & Design, Singapore University of Technology & Design, Singapore · Department of Computer Science & Engineering, Lehigh University, Bethlehem, PA, 18015, USA
Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
Log-Depth Recurrent Language Modeling
Sep 23, 2026Yiqin Wang, Nuri Cingillioglu, Charles PertLong-Context Language Modeling Autoregressive Language Modeling
Department of Computing Imperial College London London, United Kingdom
Variational Quantum Transformer Architecture for Synthetic Language Generation
Sep 16, 2026Julian Hager, Michael Kölle, Gerhard Stenzel +3Quantum Machine Learning Autoregressive Language Modeling
Institute of Informatics, LMU Munich, Oettingenstraße 67, 80538 Munich, Germany
Structural priors for data-efficient language learning
Sep 12, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model Pretraining Neural Network Initialization
University of Göttingen, Germany
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
Sep 11, 2026NCP Team, Jiaqi Cao, Chiyu Chen +25Language Model Pretraining Autoregressive Language Modeling
Shanghai AI Lab; LUMIA Lab, Shanghai Jiao Tong University
RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding
Sep 9, 2026Fang LiAutoregressive Language Modeling Language Modeling
Department of Computer Science Oklahoma Christian University Edmond, USA
Line-Coupled Language Model
Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Multi-Token Prediction Language Model Decoding
University of California, Riverside Riverside, CA, USA
ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model Pretraining Autoregressive Language Modeling
VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1
Graph Machine: Towards Better Pretraining via Edges
Sep 2, 2026Lintai HouTransformer Autoregressive Language Modeling
Iter Labs
Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining
Sep 1, 2026Francois MeyerLanguage Model Pretraining Autoregressive Language Modeling
Department of Computer Science University of Cape Town
Probabilistic Model Checking of Autoregressive Neural Sequence Models
Sep 1, 2026Helge Spieker, Dennis Gross, Arnaud GotliebNeural Network Verification Autoregressive Language Modeling
Simula Research Laboratory, Oslo, Norway
One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
Aug 26, 2026Muge Zhang, Aaron Jencks, Krishna Badikela +2Language Model Pretraining Multilingual Language Models
Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington
TANGO: Treating Tokens as Operators
Aug 22, 2026Joshua NunleyTransformer FFNs Transformer
Department of Informatics Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
Full-bandwidth transformer
Aug 9, 2026Xi Wang, Ziyang Cai, Zheng Zhan +5Autoregressive Language Modeling Recurrent Transformers
Johns Hopkins University · Princeton University · Microsoft
Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
Aug 5, 2026George FountzoulasLanguage Model Generation Evaluation Autoregressive Language Modeling
Department of Computer Engineering & Informatics · Frederick University, Nicosia, Cyprus
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
Aug 3, 2026Avni Mittal, Avinash Anand, Ashutosh Kumar +7Neural Cellular Automata Autoregressive Language Modeling
Independent Researcher · Singapore Institute of Technology, Singapore · IIIT Delhi, India +2
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangAttention Residual Connections Autoregressive Language Modeling
Kimi Team
Total Variation Distance Estimation in Autoregressive Models
Jul 21, 2026Eric Price, Kevin Tian, Zhiyang Xun +1LLM Evaluation LLM Inference
University of Texas at Austin
Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models
Jul 17, 2026Wenjie Fan, Bin Ma, Dong LiLanguage Model Decoding Autoregressive Language Modeling
Yotta Labs
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Jul 11, 2026Petros Karypis, Sean O'Brien, Shreyas Kadekodi +2Language Model Pretraining Rotary Positional Embeddings
UC San Diego · Independent Researcher
Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
Jul 7, 2026Yonggan Fu, Lexington Whalen, Abhinav Garg +23Autoregressive Diffusion Autoregressive Language Modeling
NVIDIA · Georgia Tech · Work done during internships at NVIDIA +3
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
Jul 6, 2026Archie ChaudhuryLong-Context Language Model Inference Long-Context Language Modeling
Axionic Labs
A Unified Framework for In-Context Learning with Causal and Masked Language Models
Jul 5, 2026Chenrui Liu, Chuanlong Xie, Falong Tan +2Masked Language Modeling Autoregressive Language Modeling
Beijing Normal University at Zhuhai Zhuhai, China · Hunan University Changsha, China · Sun Yat-sen University Shenzhen, China
How Token Influence Decays with Distance: A Green-Function View of Trained Language Models
Jun 28, 2026Matthias Brändel, Stephan Köhler, Oliver RheinbachTransformer Interpretability Autoregressive Language Modeling
Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, Freiberg, Germany
From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond
Jun 26, 2026Paul DuboisLLM World Models World Models
MultiHashFormer: Hash-based Generative Language Models
Jun 26, 2026Huiyin Xue, Atsuki Yamaguchi, Nikolaos AletrasLLM Compression Decoder-Only Language Models
School of Computer Science, University of Sheffield, United Kingdom
The Context-Ready Transformer
Jun 25, 2026Mahesh GodavartiEfficient Transformer Inference Transformer Inference
A Carrot, Inc

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

CNet: A Complex-Valued Deep Learning Framework with Wirtinger Autodifferentiation and FFT--Hadamard Convolution

Marcel Crasmaru

Complex-Valued Neural Networks Autoregressive Language Modeling Convolutional Neural Networks

Open paper AskPDF arXiv

CNet is a C++/CUDA framework for building deep complex-valued neural networks (CVNNs) and optimizing complex functions by gradient descent with Wirtinger derivatives. Complex models are underexplored yet natural where data is intrinsically complex -- RF/IQ communications, MRI k-space, radar/SAR, audio spectra -- and phase carries information real networks discard. CNet is physics-native: a network is a cascade of complex (often unitary) operations on an amplitude vector, and classification is a Born-rule measurement p_k = |z_k|^2/||z||^2 rather than a softmax over real logits. Its library of complex layers makes conv(x,k) = IFFT(FFT(x).FFT(k)) learnable via signal-processing primitives -- spectral-padding local kernels (Pad), the inverse DFT, a magnitude nonlinearity |z|^2 (CModulus2), holomorphic powers z^M (CPower), and mean pooling (MeanPool) -- with GPU kernels for every layer, a GPU true-Adam optimizer, and a reduced-memory inference mode. Three studies. (1) A fully complex FNet-style causal language model, built on a new O(N log N) causal Fourier mixer, matches a param-matched real-valued causal FNet on tiny-shakespeare in under half the steps. We introduce Born-rule attention: a content-based causal mixer whose weights are quantum-measurement probabilities |<Q_k,K_j>|^2 of one token against another -- the first attention mechanism built on the Born rule. With rotary position embeddings and per-token complex normalization, a fully complex Born-rule model reaches 1.52 nats/char on tiny-shakespeare, matching softmax attention and beating the Fourier mixer by 0.23. (2,3) Two bottleneck analyses -- radio-modulation classification (RML2016.10a) and the Fourier phase problem of coherent-diffraction imaging -- isolate where CVNNs need new operators. The complex machinery learns the correct structure; open problems are concrete and operator-level. Code: https://github.com/crasmarum/CNet

Momentum

Related topics

Latest papers 81