Transformer Variant

Follow this topic Ask about this topic

Recent momentum

-70%

3 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

1 new paper

A weekly snapshot of new work published in Transformer Variant.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Transformer Variant.

70 papers

Latest in Transformer Variant

Open your feed →

Length Generalization for Transformers via Compression
Sep 8, 2026Georg Zetzsche, Hongjian Jiang, Andy Yang +4Transformer Variant Generalization Bounds
From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs
Sep 1, 2026Jie Chen, Xiangqian Yu, Yanchao Lian +9Transformer Variant Recommendation
Transformer-Based Flow Shop Scheduling Using MILP-Generated Training Data
Aug 30, 2026Roderich WallrathFlexible Job-Shop Scheduling Adaptive Scheduling
Algebraic Decomposition Theory for Transformer Length Generalization
Aug 13, 2026Andy Yang, Blerta Veseli, Corentin Barloy +5Transformer Variant Length
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
Aug 6, 2026Omar Coser, Antonio Orvieto, Paolo Soda +1Time-Series Classification Time Series
CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization
Aug 3, 2026Chuyan Chen, Peng Sun, Kun YuanDiffusion Transformers Muon
Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory
Aug 3, 2026Kaustubh Kapil, Kishor P. UplaVisual Geometry Grounded Transformer Vision Transformer
At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference
Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Tensor Programs Vectorization
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3Transformer Variant Adam
Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs
Jul 24, 2026Ilia Sobakinskikh, Paul Alexander BilokonTransformer Variant Time Series
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Jul 23, 2026Zahra Yousefijamarani, Alaa AlameldeenTransformer Variant In-Memory Computing
KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
Jul 23, 2026Yann Bouquet, Alireza Khodamoradi, Kristof Denolf +1Post-Training Quantization Quantization
Agree on the Model, Verify the Inference: GKR Protocols for HND-Based Transformer Inference
Jul 23, 2026Xiaolong Liang, Juanjuan Li, Rui Qin +1Transformer Variant Theorem
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
Jul 19, 2026Zhihua LiangTransformer Variant Diffusion Transformers
RepTran: Search-Based Repair of Transformer Models
Jul 13, 2026Yuta Ishimoto, Paolo Arcaini, Fuyuki Ishikawa +3Transformer Variant Diffusion Transformers
Incremental Transformer for Surrogate-Based Inverse Design of Geopolymer Mixtures
Jul 12, 2026Giansalvo Cirrincione, Filippo GrassiaGenerative Inverse Design Blended Cement Concrete
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
Jul 7, 2026Ho-Lam Chung, Ke-Han Lu, Yi-Cheng Lin +3Speech Encoder Speaker Identity
Structure-Specific Representational Priors Causally Control the Grokking Delay
Jul 5, 2026Gunner Levi HoweGrokking Representation Learning
FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity
Jul 4, 2026Shuai Li, Qinglin Wang, Ping Luo +8Federated Fine-Tuning Fedavg
Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits
Jul 2, 2026Zhiren Gong, Zihao Zeng, Chau Yuen +1Ablation Transformer Variant
The State-Prediction Separation Hypothesis
Jul 1, 2026Giovanni Monea, Nathan Godey, Kianté Brantley +1Transformer Variant \Emph{State-Prediction Separation Hypothesis
Exploring the Cryptographic Limits of Transformer Networks
Jun 28, 2026Stefan Domunco, Andis Draguns, Philip Torr +2Transformer Variant Diffie-Hellman
The Context-Ready Transformer
Jun 25, 2026Mahesh GodavartiTransformer Variant Diffusion Transformers
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Jun 22, 2026Mansour Zoubeirou a MayakiEnergy Based Models Transformer Variant
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarTransformer Variant Mixture-Of-Experts Layer
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Jun 16, 2026Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin +5Transformer Variant Recursive Reasoner
AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformers
Jun 16, 2026Kaijian Wang, Yuanyuan Xu, Fanjiang Ye +5Video Diffusion Models Accelerator
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Transformer Variant Curvature Prediction Quality
NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam
Jun 15, 2026Evgeny UkladchikovTransformer Variant Field-Programmable Gate Arrays
Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components
Jun 12, 2026Yifeng Guo, Jin-Hong Du, Xiang ChenTransformer Variant Importance
Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0
Jun 12, 2026Ali Asaria, Tony Salomone, Deep GandhiDiffusion Transformers Systematic Fp16 Divergence
Zero-shot generalization of transformer neural operators to larger domains
Jun 12, 2026Armand de Villeroché, Sibo Cheng, Vincent Le Guen +5Neural Operators Transformer Variant
Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer
Jun 11, 2026Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6Spiking Neural Networks Binary Spikes
Boundary-Protection W8A8 HiFloat8 Quantization for Large-Scale Text-to-Video Diffusion Transformers
May 31, 2026Yiming ZhaoVideo Diffusion Models Diffusion Transformers
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
May 28, 2026Anej Svete, William Merrill, Ryan Cotterell +1Transformer Variant Transformer Attention
Brain-IT-VQA: From Brain Signals to Answers
May 28, 2026Roman Beliy, Matias Cosarinsky, Oliver Heinimann +2Textvqa Brain
Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability
May 24, 2026David N. Olivieri, Antonio F. Pérez RodríguezTransformer Variant Autoregressive Transformers
The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth
May 24, 2026James HenryTransformer Variant Concept Bottleneck Models
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuTransformer Variant Batch Normalization
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3Transformer Variant Parameter-Efficient Fine-Tuning Methods
How Many Different Outputs Can a Transformer Generate?
May 21, 2026Maxime Meyer, Mario Michelessa, Caroline Chaux +1Transformer Variant Length
Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
May 20, 2026Nandan Kumar Jha, Brandon ReagenOptimizer Design Scaling Laws
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Variant Gradient Descent
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzSparsity Transformer Variant
Markovian Circuit Tracing for Transformer State Dynamic
May 20, 2026Abdullah XTransformer Variant Hidden Markov Models
Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
May 20, 2026Yang Zhao, Jiahao Lu, Bin Huang +2Transformer Variant Downstream Reasoning
Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
May 19, 2026Xinzhe Yuan, Xiang Peng, Bin Gu +1Spiking Neural Networks Binary Spikes
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
May 19, 2026Han Guo, Jack Zhang, Arjun Menon +4Graphics Processing Unit Kernels Matrix Multiplication
SNLP: Layer-Parallel Inference via Structured Newton Corrections
May 18, 2026Ligong Han, Kai Xu, Hao Wang +1Transformer Variant Transformer Residual Streams
Propagation of Chaos in Contextual Flow Maps
May 16, 2026Shi Chen, Zhengjiang Lin, Kaizhao Liu +1Transformer Variant Wasserstein Gradient Flows
Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
May 14, 2026Mahdi Naser Moghadasi, Faezeh GhaderiTransformer Variant Large Language Model Scale
What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization
May 12, 2026Ryota Yoshihashi, Masahiro Kada, Satoshi Ikehata +2Vision Transformer Localization
PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$ -Norm Steepest DescentPowerStep: Memory-Efficient Adaptive Optimization via \ell_p-Norm Steepest Descent
May 11, 2026Yao Lu, Dengdong Fan, Shixun Zhang +1Adam Transformer Variant
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
May 6, 2026Zhengyi Li, Yakai Wang, Kang Yang +6Transformer Variant Defence
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliTransformer Variant Weight Decay
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Variant Representational Geometry
Gated Subspace Inference for Transformer Acceleration
May 4, 2026Stephen J. ThomasTransformer Variant Low-Rank Factorization
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTensor Programs Tensor
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
Apr 28, 2026Oliver Kraus, Yash Sarrof, Yuekun Yao +2Transformer Variant Chain-Of-Thought
Distance-Misaligned Training in Graph Transformers and Adaptive Graph-Aware Control
Apr 24, 2026Qinhan Hou, Jing TangMultiplex Graph Transformers Training-Inference Mismatch

Sep 8, 2026 · cs.LGJ/K move · Enter open · S save

Length Generalization for Transformers via Compression

Georg Zetzsche, Hongjian Jiang, Andy Yang, Pascal Bergsträßer+3

Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a formalized version of the so-called RASP-l conjecture) posits that transformers length-generalize on a task if and only if a solution is expressible in the C-RASP language. While this hypothesis has strong empirical validation, theoretical problems arise from the fact that no computable length generalization bounds exist for C-RASP, alongside the discovery of seemingly contradictory experiments. To address these problems, we refine the C-RASP hypothesis utilizing the recently-proposed fragments C-RASP+ and C-RASP1. These fragments have computable length generalization bounds, though in the worst case requiring an extremely large (double exponential) sample size. It is an open question whether these sample size bounds are tight. In this paper, we resolve this open question by providing an exponentially tighter bound. In doing so, we show a polynomial length generalization bound for transformers if we adopt compressed strings, via a novel connection to power words. As an application, we show how this yields a fine-grained analysis of the C-RASP conjecture that resolves contradicting experimental evidence against it.

Transformer Variant Generalization Bounds Length Model and Context Compression Sample Size

Open paper Ask about it PDF