Transformers

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

Focused topic

Transformer Architectures

Within Transformers

684 papersExplore

Focused topic

Quadratic Attention

Within Transformers

144 papersExplore

Focused topic

Dynamic Sparse Attention

Within Transformers

109 papersExplore

Focused topic

Transformer Attention

Within Transformers

89 papersExplore

Focused topic

Transformer Encoder

Within Transformers

80 papersExplore

Focused topic

Transformer Layers

Within Transformers

80 papersExplore

Focused topic

Softmax Attention

Within Transformers

74 papersExplore

Focused topic

Unified Transformer

Within Transformers

40 papersExplore

Focused topic

Multi-Head Self-Attention

Within Transformers

35 papersExplore

Focused topic

Autoregressive Transformers

Within Transformers

33 papersExplore

Focused topic

Global Attention Mechanism

Within Transformers

20 papersExplore

1,208 papers

Latest in Transformers

Browse all topics →

Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Date pendingArman Adibi, Alireza Jafari, Mohammad Ghavamzadeh +1Samplers Transformer Architectures
Tracing Computation Density in LLMs
Date pendingCorentin Kervadec, Iuliia Lysova, Iuri Macocco +2Large Models Transformer Layers
UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation
Date pendingZeyang Liu, Le Wang, Sanping Zhou +4Layouts Large-Scale Image Generation
Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
Date pendingVarvara Arzt, Allan Hanbury, Terra BlevinsEnglish Unified Transformer
On the Residual Scaling of Looped Transformers: Stability and Transferability
Date pendingShaowen Wang, Bingrui Li, Ge Zhang +3Residual Connections Learning Rates
Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packaging
Date pendingPhat Nguyen, Xue Geng, Kaixin Xu +3Vision Transformer Sparse Deep Neural Network Accelerators
Musical Attention Transformer: Music Generation Using a Music-Specific Attention Model
Date pendingShinnosuke Takasuka, Hideo MukaiMusic Generation Transformer Architectures
SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
Date pendingJongbeom Lee, Hyunwoo Yu, Jincheol Yang +2Dynamic Sparse Attention Spatial Attention

Date pending · cs.LGJ/K move · Enter open · S save

Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh, Hadi Daneshmand

A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.

Samplers Transformer Architectures In-Context Learning

Open paper Ask about it PDF