Fast Weights

Follow this topic Ask about this topic

Latest papers

8 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Looped Transformers as Optimizers
Sep 29, 2026Yulong Huang, Chen Jiang, Zhanpeng Zhou +5Fast Weights Recurrent Transformers
HKUST (GZ) · StepFun · SJTU +2
TERN: A Delta-rule Memory with a Seasonal Reference and Online Adaptation for Epidemic Forecasting
Sep 16, 2026Shunya Nagashima, Yuta FunayamaEpidemic Forecasting Time Series Forecasting
LTS, Inc., Japan
Modular TTT: Rethinking Test-Time Training as Composable Modules
Aug 7, 2026Bohao Tang, Zhen Qin, Yuqi Pan +3Test-Time Training Fast Weights
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Stable Self-Modulating Quantum Fast-Weight Programmers with Bounded Memory Gates
Jul 2, 2026Kuo-Chung Peng, Jiun-Cheng Jiang, Chun-Hua Lin +8Time Series Forecasting Variational Quantum Circuits
National Taiwan University, Taiwan · Stevens Institute of Technology, NJ, USA · Seoul National University, Korea +3
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Self-Attention
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA
FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation
May 6, 2026Guangsheng Bao, Hongbo Zhang, Han Cui +4Efficient Neural Network Inference Test-Time Adaptation
Zhejiang University · Westlake University · Shanghai Polytechnic University +1
Where to Bind Matters: Hebbian Fast Weights in Vision Transformers for Few-Shot Character Recognition
Apr 22, 2026Gavin Money, Sindhuja Penchala, Jiacheng Li +1Hebbian Learning Vision Transformer
Department of Computer Science, The University of Alabama, Tuscaloosa, AL, USA
Reinforced Fast Weights with Next-Sequence Prediction
Feb 18, 2026Hee Seung Hwang, Xindi Wu, Sanghyuk Chun +1Fast Weights Long-Context Language Modeling
Princeton University

Sep 29, 2026J/K move · Enter open · S save

Looped Transformers as Optimizers

Yulong Huang, Chen Jiang, Zhanpeng Zhou, Hongtao Zhang+4

HKUST (GZ) · StepFun · SJTU+2

Fast Weights Recurrent Transformers Deep Learning Optimization

Open paper AskPDF arXiv

Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models have likewise highlighted the value of scaling test-time computation through longer computation trajectories. However, the principles for designing effective loop transitions remain poorly understood. We view the looped hidden state as a fast weight that is updated throughout the depth. We formulate loop transitions as local gradient-based updates, with recurrent blocks predicting implicit targets at each depth. Our framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. Mapping representative loop transitions into this framework reveals mismatches between their transitions and projections. We first align the input maps of existing transitions. We then derive OperLoop, which combines explicit weight decay, adaptive step size and a delta objective. The aligned variants reduce training loss and improve average commonsense accuracy. OperLoop improves average generative performance over the compared looped and non-looped baselines under matched training FLOPs. These results support the framework's usefulness for loop design. We extend the analysis to additional loop models and outline a roadmap for future loop transition design.