Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

Symmetry-Aware Transformer Training for Automated Planning
Aug 11, 2025Markus Fritzsche, Elliot Gestrin, Jendrik SeippClassical Planning Transformer Architectures
Linköping University, Sweden
Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation
Aug 7, 2025Kartar Kumar, Rajesh Kumar, Nikesh LagunLarge Language Model Workflows Reaction
Department of Computer Science, Institute of Business Administration (IBA), Karachi, Pakistan · International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Modelling Adjectival Modification Effects on Semantic Plausibility
Jul 29, 2025Anna Golub, Beate Zywietz, Annerose EichelPhysical Plausibility Transformer Architectures
Institute for Natural Language Processing, University of Stuttgart, Pfaffenwaldring 5b, Stuttgart, 70569, Baden-Württemberg, Germany.
A Systematic Analysis of Hybrid Linear Attention
Jul 8, 2025Dustin Wang, Rui-Jie Zhu, Steven Abreu +9Kimi Delta Attention Transformer Architectures
1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training Self-Supervised Vision Transformers
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1Large Language Model Quantization Transformer Architectures
Qualcomm AI Research, Amsterdam, Netherlands
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Compressed Model Decentralized Learning
Pluralis Research
LlamaSeg: Image Segmentation via Autoregressive Mask Generation
May 26, 2025Jiru Deng, Tengjin Weng, Tianyu Yang +3Visual Autoregressive Models Autoregressive Generation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2
Freqformer: Image-Demoiréing Transformer via Effective Frequency Decomposition
May 25, 2025Xiaoyang Liu, Bolin Qiu, Zheng Chen +5Image Restoration Wavelets
Shanghai Jiao Tong University
Quick ViTs: Speeding up Vision Transformers through Equivariance
May 21, 2025David Nordström, Johan Edstedt, Fredrik Kahl +1Self-Supervised Vision Transformers Vision Transformer
Chalmers University of Technology · Linköping University · University of Amsterdam
Predicting Estimated Times of Restoration for Electrical Outages Using Longitudinal Tabular Transformers
May 1, 2025Bogireddy Sai Prasanna Teja, Valliappan Muthukaruppan, Carls BenjaminRestoration Transformer Architectures
Exelon Corporation, Chicago, IL, USA
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Dynamic Sparse Attention Transformer Attention
University of Edinburgh · Cohere · Meta
Numerical stability analysis of large language models
Mar 13, 2025Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1Transformer Architectures Full-Precision Performance
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies
Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting
Mar 10, 2025Jianqi Zhang, Yuchan Liu, Zeen Song +2Forecasting Backbone Transformer Architectures
National Key Laboratory of Space Integrated Information Systems, Institute of Software, Chinese Academy of Sciences, Beijing, China 1 · University of Chinese Academy of Sciences, Beijing, China 2 · Digital-Intelligent Operations Center, Guizhou Power Grid Co., Ltd., Guizhou, China 3 +1
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
Feb 20, 2025Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4Efficient Long-Context Inference Token-Level Uncertainty
AIRI · Skoltech · Lomonosov Moscow State University +1
Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications
Feb 17, 2025Li Qiao, Mahdi Boloursaz Mashhadi, Zhen Gao +3Semantic Communications Token Compression
School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China · 5GIC & 6GIC, Institute for Communication Systems (ICS), University of Surrey, Guildford, United Kingdom · Centre for Wireless Communications, University of Oulu, 90014 Oulu, Finland +1
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Transformer Architectures Language Modeling
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University
3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation
Jan 28, 2025Yueen Ma, Zenglin Xu, Irwin KingDiffusion-Based Vision-Language-Actions Action Generation
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR
CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image
Dec 17, 2024Wonseok Roh, Hwanhee Jung, Jong Wook Kim +63D Reconstruction 3D Gaussian
Korea University · Google · Purdue University
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9Single-Token Output Distributions Distributional Information
Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Architectures Pretraining
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.
Tuning Language Models by Mixture-of-Depths Ensemble
Oct 16, 2024Haoyan Luo, Lucia SpeciaLarge Language Model Fine-Tuning Layer-Wise
University of Cambridge · Imperial College London
On Generalisation Error Bounds for Transformers
Oct 15, 2024Lan V. TruongGeneralization Bounds Transformer Architectures
Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), Vietnam
Multi-view Hand Reconstruction with a Point-Embedded Transformer
Aug 20, 2024Lixin Yang, Licheng Zhong, Pengxiang Zhu +43D Hand Motion Capture
School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, Shanghai 200230, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China +1
A Survey of Transformer-based Language Models with Focus on Efficiency
May 15, 2024Wazib Ansar, Saptarsi Goswami, Amlan ChakrabartiTransformer Architectures Efficiency
A. K. Choudhury School of IT University of Calcutta, Kolkata, India · Department of Computer Science Bangabasi Morning College Kolkata, India
Efficient Concertormer for Image Deblurring and Beyond
Apr 9, 2024Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien +1Gaussian Deblurring Transformer Attention
AdvMT: Adversarial Motion Transformer for Long-term Human Motion Prediction
Jan 10, 2024Sarmad Idrees, Seokman Sohn, Jongeun ChoiHuman Motion Prediction Long-Horizon Forecasting
School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea · Power Generation Lab, Korea Power Research Institute, 105, Munji-Ro, Yuseong-Gu, Daejeon, 34056, South Korea
Introduction to Transformers: an NLP Perspective
Nov 29, 2023Tong Xiao, Jingbo ZhuTransformer Architectures Natural Language Processing
NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China
VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder
Jul 3, 2023Yueen Ma, Dafeng Chi, Jingjing Li +3Conditional Variational Autoencoder Transformer Architectures
The Chinese University of Hong Kong · Huawei Noah’s Ark Lab
Generalizing Adam to Manifolds for Efficiently Training Transformers
May 26, 2023Benedikt BrantnerAdam Manifolds
Max-Planck-Institut für Plasmaphysik Boltzmannstraße 2, 85748 Garching, Deutschland · Technische Universität München, Zentrum Mathematik Boltzmannstraße 3, 85748 Garching, Deutschland
Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models
Date pendingArthur Negrão, Pedro Silva, Vander L. S. Freitas +2Transformer Architectures Post-Training
Postgraduate Program in Computer Science Federal University of Ouro Preto · Computing Department Federal University of Ouro Preto
On the Residual Scaling of Looped Transformers: Stability and Transferability
Date pendingShaowen Wang, Bingrui Li, Ge Zhang +3Residual Networks Transformer Architectures
Tsinghua University · ByteDance Seed · M-A-P
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Date pendingArman Adibi, Alireza Jafari, Mohammad Ghavamzadeh +1Accelerated Sampling Transformer Architectures
School of Computer and Cyber Sciences, Augusta University · Department of Computer Science, University of Virginia · Qualcomm AI Research
Debunking Grad-ECLIP: A Comprehensive Study on Its Incorrectness and Fundamental Principles for Model Interpretation
Date pendingYongjin Cui, Xiaohui FanGradient-Weighted Class Activation Mapping Cvpr
Zhejiang University
UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation
Date pendingZeyang Liu, Le Wang, Sanping Zhou +4Layouts High-Fidelity Conditional Generation
College of Artificial Intelligence, Xi’an Jiaotong University · Amazon.com, Inc. · Pixocial Technology
BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classification
Date pendingJingtao Zhang, Yi Liu, Qi Shen +1Hybrid Transformer Time-Series Classification
School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University · College of Medicine and Biological Information Engineering, Northeastern University
Tracing Computation Density in LLMs
Date pendingCorentin Kervadec, Iuliia Lysova, Iuri Macocco +2Transformer Architectures Traces
Universitat Pompeu Fabra · ICREA
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Transformer Architectures Scaling Laws
Tsinghua University · ByteDance Seed · M-A-P
Musical Attention Transformer: Music Generation Using a Music-Specific Attention Model
Date pendingShinnosuke Takasuka, Hideo MukaiTransformer Architectures Metadata
Department of Computer Science, School of Science and Technology, Meiji University, 1-1-1 Higashimita, Tama-ku, Kawasaki, Kanagawa 214-8571, Japan
Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packaging
Date pendingPhat Nguyen, Xue Geng, Kaixin Xu +3Vision Transformer Token Compression
Singapore University of Technology and Design (SUTD) · Agency for Science, Technology, and Research (A*STAR)

Aug 11, 2025, cs.AIJ/K move · Enter open · S save

Symmetry-Aware Transformer Training for Automated Planning

Markus Fritzsche, Elliot Gestrin, Jendrik Seipp

Linköping University, Sweden

Classical Planning Transformer Architectures Planning Transformer Encoder Symmetry Data Generation+1

Open paper AskPDF arXiv

While transformers excel in many settings, their application in the field of automated planning is limited. Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to hard planning problems. This in turn stems from problem symmetries: planning tasks can be represented with arbitrary variable names that carry no meaning beyond being identifiers. This causes a combinatorial explosion of equivalent representations that pure transformers cannot efficiently learn from. We propose a novel contrastive learning objective to make transformers symmetry-aware and thereby compensate for their lack of inductive bias. Combining this with architectural improvements, we show that transformers can be efficiently trained for either plan-generation or heuristic-prediction. Our results across multiple planning domains demonstrate that our symmetry-aware training effectively and efficiently addresses the limitations of PlanGPT.

Momentum

Related topics

Latest papers 1,120