Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

Aug 11, 2025cs.AI

Symmetry-Aware Transformer Training for Automated Planning

While transformers excel in many settings, their application in the field of automated planning is limited. Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to hard planning problems. This in turn stems from problem symmetries: planning tasks can be represented with arbitrary variable names that carry no meaning beyond being identifiers. This causes a combinatorial explosion of equivalent representations that pure transformers cannot efficiently learn from. We propose a novel contrastive learning objective to make transformers symmetry-aware and thereby compensate for their lack of inductive bias. Combining this with architectural improvements, we show that transformers can be efficiently trained for either plan-generation or heuristic-prediction. Our results across multiple planning domains demonstrate that our symmetry-aware training effectively and efficiently addresses the limitations of PlanGPT.

Markus Fritzsche, Elliot Gestrin, Jendrik Seipp

Linköping University, Sweden

Classical Planning Transformer Architectures Planning Transformer Encoder Symmetry Data Generation+1

Aug 7, 2025cs.AI

Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation

Large language models (LLMs) are beginning to reshape how organic-synthesis workflows are represented, queried, planned, and connected to experimental automation. Assessing their contribution is not straightforward because reaction-specific transformers, chemistry-adapted LLMs, tool-using agents, optimizers, and autonomous laboratories are often discussed under the same broad terminology despite operating at different levels of the synthesis workflow. This Review traces the progression from reaction prediction and retrosynthetic planning to reaction development, literature-to-protocol translation, and robotic execution, with representative numerical claims verified against primary sources available through 4 September 2026. We examine where language models provide genuine added value, where established specialist methods remain the principal source of performance, and how retrieval, uncertainty-aware optimization, deterministic chemistry tools, robotics, sensing, and human oversight alter system behavior. The evidence shows a clear transition from isolated language-model demonstrations toward modular scientific systems that combine broad language priors with specialized computation and experimentally grounded feedback. Emerging work on multimodal reaction understanding, standardized tool interoperability, provenance-aware execution, and programmable laboratory infrastructure extends this transition further. Despite this progress, reliable deployment continues to depend on generalization beyond historical reaction corpora, calibrated uncertainty, transparent component attribution, safety controls, reproducible system state, and prospective experimental validation.

Kartar Kumar, Rajesh Kumar, Nikesh Lagun

Department of Computer Science, Institute of Business Administration (IBA), Karachi, Pakistan · International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China

Large Language Model Workflows Reaction Large Language Model Generation Synthesis Automated Transformer Architectures

Jul 29, 2025cs.CL

Modelling Adjectival Modification Effects on Semantic Plausibility

While the task of assessing the plausibility of events such as "news is relevant" has been addressed by a growing body of work, less attention has been paid to capturing changes in plausibility as triggered by event modification. Understanding changes in plausibility is relevant for tasks such as dialogue generation, commonsense reasoning, and hallucination detection, as it allows to correctly model, for example, "false news is relevant", which is of lower relevance but higher concern due to potential disinformation. In this work, we tackle the Adept challenge benchmark (Emami et al. 2021) consisting of 16K English sentence pairs differing by exactly one adjectival modifier (e.g., false.) Our modeling experiments provide a conceptually novel method using sentence transformers and reveal that sentence transformers struggle despite their conceptual alignment with the task at hand, underperforming in comparison to transformers like RoBERTa. Finally, we discuss our findings in relation to prior work and present a detailed error analysis to shed light on potential sources for ST underperformance, highlighting advantages and shortcomings of the examined methods for balancing out train and test data.

Anna Golub, Beate Zywietz, Annerose Eichel

Institute for Natural Language Processing, University of Stuttgart, Pfaffenwaldring 5b, Stuttgart, 70569, Baden-Württemberg, Germany.

Physical Plausibility Transformer Architectures Relevance Impact Analysis

Jul 8, 2025cs.CL

A Systematic Analysis of Hybrid Linear Attention

Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored. We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized. To enable this comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six linear attention variants across five hybridization ratios. Benchmarking on standard language modeling and recall tasks reveals that superior standalone linear models do not necessarily excel in hybrids. While language modeling remains stable across linear-to-full attention ratios, recall significantly improves with increased full attention layers, particularly below a 3:1 ratio. Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. We recommend architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 to achieve Transformer-level recall efficiently. Our models are open-sourced at https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.

Dustin Wang, Rui-Jie Zhu, Steven Abreu +9

1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3

Kimi Delta Attention Transformer Architectures Discontinuous Gating

Jun 13, 2025cs.CV

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

Vision Transformers (ViTs) are essential in computer vision but are computationally intensive, too. Model quantization, particularly to low bit-widths like 4-bit, aims to alleviate this difficulty, yet existing Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) methods exhibit significant limitations. PTQ often incurs substantial accuracy drop, while QAT achieves high accuracy but suffers from prohibitive computational costs, limited generalization to downstream tasks, training instability, and lacking of open-source codebase. To address these challenges, this paper introduces General, Practical, and Lightning Quantization (GPLQ), a novel framework designed for efficient and effective ViT quantization. GPLQ is founded on two key empirical insights: the paramount importance of activation quantization and the necessity of preserving the model's original optimization basin'' to maintain generalization. Consequently, GPLQ employs a sequential activation-first, weights-later'' strategy. Stage 1 keeps weights in FP32 while quantizing activations with a feature mimicking loss in only 1 epoch to keep it stay in the same ``basin'', thereby preserving generalization. Stage 2 quantizes weights using a PTQ method. As a result, GPLQ is 100x faster than existing QAT methods, lowers memory footprint to levels even below FP32 training, and achieves 4-bit model performance that is highly competitive with FP32 models in terms of both accuracy on ImageNet and generalization to diverse downstream tasks, including fine-grained visual classification and object detection. We will release an easy-to-use open-source toolkit supporting multiple vision tasks.

Guang Liang, Xinyao Liu, Jianxin Wu

State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China

Quantization-Aware Training Self-Supervised Vision Transformers Vision Transformer Imagenet Transformer Architectures

Jun 5, 2025cs.LG

FPTQuant: Function-Preserving Transforms for LLM Quantization

Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces three novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, and (3) a cheap, dynamic per-token scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model's function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9X over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off -- it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.

Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1

Qualcomm AI Research, Amsterdam, Netherlands

Large Language Model Quantization Transformer Architectures

Jun 2, 2025cs.LG

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2

Pluralis Research

Compressed Model Decentralized Learning Data Compression Methods Transformer Architectures Bottlenecks

May 26, 2025cs.CV

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

We present \textbf{LlamaSeg}, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the \textbf{SA-OVRS} dataset, which contains \textbf{2M} segmentation masks annotated with over \textbf{5,800} open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance (

d_{\mathrm{AHD}}

) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.

Jiru Deng, Tengjin Weng, Tianyu Yang +3

Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2

Visual Autoregressive Models Autoregressive Generation Transformer Architectures Next-Token Prediction

May 25, 2025cs.CV

Freqformer: Image-Demoiréing Transformer via Effective Frequency Decomposition

Image demoiréing remains a challenging task due to the complex interplay between texture corruption and color distortions caused by moiré patterns. Existing methods, especially those relying on direct image-to-image restoration, often fail to disentangle these intertwined artifacts effectively. While frequency-aware approaches offer a promising direction, their potential is hindered by the discrete transform (e.g., Haar wavelet or block-based DCT), which may suffer from spatial discontinuity, channel redundancy, and further cause error accumulation during their fixed inverse processes. In this paper, we present Freqformer, a Transformer-based framework specifically designed for image demoiréing through targeted frequency separation. Our method performs an effective frequency decomposition that splits moiré patterns into high-frequency spatially-localized textures and low-frequency scale-robust color distortions, which are then handled by a dual-branch architecture and an asymmetric training scheme tailored to their distinct characteristics. We further propose a learnable Frequency Composition Transform (FCT) module to adaptively fuse the frequency-specific outputs, enabling consistent and high-fidelity reconstruction. To better aggregate the spatial dependencies and the inter-channel complementary information, we introduce a Spatial-Aware Channel Attention (SA-CA) module that refines moiré-sensitive regions without incurring high computational cost. Extensive experiments on various demoiréing benchmarks demonstrate that Freqformer achieves state-of-the-art performance with a compact model size. The code will be made publicly available at https://github.com/xyLiu339/Freqformer.

Xiaoyang Liu, Bolin Qiu, Zheng Chen +5

Shanghai Jiao Tong University

Image Restoration Wavelets High-Frequency Reconstruction Transformer Architectures

May 21, 2025cs.CV

Quick ViTs: Speeding up Vision Transformers through Equivariance

Natural images exhibit strong geometric regularities: local structures, such as edges, corners, and textures, appear in many orientations and mirror configurations. Since Vision Transformers (ViTs) operate on square image patches, these transformations naturally correspond to the dihedral symmetry group

\mathrm{D}_8

, also known as the octic group. Recent work has shown that ViTs can be made reflection equivariant and more efficient than standard ViTs simultaneously by implementing the linear layers in the Fourier domain of the reflection group. In this work, we extend the equivariance to reflections and rotations and analyze the scalability of the resulting networks. Our Quick ViTs, based on octic equivariant linear layers, achieve 5.33x reductions in FLOPs and up to 8x reductions in memory compared to ordinary linear layers. By analyzing the arithmetic intensity of these layers, we identify theoretical limits on how much the FLOP savings translate into throughput improvements on modern GPUs. However, these limitations disappear as the embedding dimensions increase. Enabled by their computational efficiency, we conduct a broader empirical evaluation of equivariant ViTs than in previous work. Upon training supervised (DeiT-III) and self-supervised (DINOv2) on ImageNet-1K, we find that our Quick ViTs match or exceed baseline accuracy while at the same time providing substantial efficiency gains.

David Nordström, Johan Edstedt, Fredrik Kahl +1

Chalmers University of Technology · Linköping University · University of Amsterdam

Self-Supervised Vision Transformers Vision Transformer Equivariant Neural Networks Imagenet Equivariance Transformer Architectures

May 1, 2025cs.LG

Predicting Estimated Times of Restoration for Electrical Outages Using Longitudinal Tabular Transformers

Utilities publish Estimated Times of Restoration (ETRs) for customer-facing storm outages, and their accuracy governs whether customers can make sound decisions about food, medical equipment, and relocation. Prior work treats ETR as static tabular regression in which each outage contributes one record, discarding the fact that every development of an outage, from crew assignment through dispatch, suspension, damage assessment and partial restoration, is recorded as a revision. We reformulate ETR prediction as longitudinal tabular regression and introduce a Longitudinal Tabular Transformer (LTT), an axial-attention model that consumes the revisions preceding a prediction and issues a refined estimate at every one. On 242{,}928 storm-attributed outages from a cohort of 526{,}468 filtered events and 10.0 million revisions at six operating companies, LTT reduces customer-weighted asymmetric error at all six, by a median of 36.9,% against the estimates the utilities published during the same storms and 11.3,% against the strongest learned baseline at each. It is the only method improving on the incumbent's satisfaction impact at all six companies while also reducing root mean squared error at all six. Stratification by revision index shows that LTT error is largest at the first revision, where no history is available, and falls monotonically as revisions accumulate.

Bogireddy Sai Prasanna Teja, Valliappan Muthukaruppan, Carls Benjamin

Exelon Corporation, Chicago, IL, USA

Restoration Transformer Architectures

Apr 24, 2025cs.CL

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of comprehensive evaluation. We address this gap with the largest-scale empirical analysis to date of training-free sparse attention, evaluating six methods across multiple model families and sizes, sequences up to 128K tokens, and sparsity levels up to 0.95 (i.e.,

1/20

attention budget) on nine diverse tasks. We first organise the rapidly evolving landscape of sparse attention methods into a taxonomy along four design axes. Our analysis then yields actionable insights: 1) sparse attention is effective: larger sparse models outperform smaller dense ones at equivalent cost, improving the Pareto frontier; 2) for the training-free methods we study, fine-grained per-query importance estimation during prefilling remains impractical-due to both the cost of estimation and the lack of sparse kernels that translate fine-grained sparsity into wall-clock gains-forcing a task-dependent choice between global-to-token and block-to-block selection. Instead, during decoding, token-to-page selection becomes feasible, enabling better generalisation and higher sparsity tolerance; 3) longer sequences tolerate higher sparsity, suggesting that fixed-budget methods in production are suboptimal. Together, these findings provide practical guidance for deploying sparse attention and methodological recommendations for future evaluations. Our code is available at https://github.com/PiotrNawrot/sparse-frontier.

Piotr Nawrot, Robert Li, Renjie Huang +3

University of Edinburgh · Cohere · Meta

Dynamic Sparse Attention Transformer Attention Transformer Architectures Efficient Long-Context Inference Importance

Mar 13, 2025math.NA

Numerical stability analysis of large language models

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.

Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1

Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies

Transformer Architectures Full-Precision Performance Residual Stream Outliers

Mar 10, 2025cs.AI

Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting

Time series forecasting (TSF) is important in real-world applications. Recently, Transformer-based methods have achieved strong performance by modeling token dependencies through attention mechanisms. However, existing methods are usually trained mainly with prediction error losses, which may cause models to exploit both critical and redundant token dependencies. Such redundant dependencies can introduce irrelevant information and weaken generalization. To address this issue, we propose a simple yet effective token dependency selection strategy. Specifically, by jointly introducing the attention entropy constraint and prediction error constraint, the model can identify fewer but more critical inter-token dependencies and perform forecasting based on them, thus avoiding the interference of redundant dependencies. The proposed method can be easily extended to various Transformer-based TSF models. Experiments on multiple TSF datasets demonstrate its effectiveness.

Jianqi Zhang, Yuchan Liu, Zeen Song +2

National Key Laboratory of Space Integrated Information Systems, Institute of Software, Chinese Academy of Sciences, Beijing, China 1 · University of Chinese Academy of Sciences, Beijing, China 2 · Digital-Intelligent Operations Center, Guizhou Power Grid Co., Ltd., Guizhou, China 3 +1

Forecasting Backbone Transformer Architectures Loss Function

Feb 20, 2025cs.CL

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

We introduce methods to quantify how Large Language Models (LLMs) encode and store contextual information, revealing that tokens often seen as minor (e.g., determiners, punctuation) carry surprisingly high context. Notably, removing these tokens -- especially stopwords, articles, and commas -- consistently degrades performance on MMLU and BABILong-4k, even if removing only irrelevant tokens. Our analysis also shows a strong correlation between contextualization and linearity, where linearity measures how closely the transformation from one layer's embeddings to the next can be approximated by a single linear mapping. These findings underscore the hidden importance of filler tokens in maintaining context. For further exploration, we present LLM-Microscope, an open-source toolkit that assesses token-level nonlinearity, evaluates contextual memory, visualizes intermediate layer contributions (via an adapted Logit Lens), and measures the intrinsic dimensionality of representations. This toolkit illuminates how seemingly trivial tokens can be critical for long-range understanding.

Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4

AIRI · Skoltech · Lomonosov Moscow State University +1

Efficient Long-Context Inference Token-Level Uncertainty Transformer Architectures Long-Range Coherence Peak Memory Microscopy

Feb 17, 2025cs.MM

Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications

In this paper, we introduce token communications (TokCom), a large model-driven framework to leverage cross-modal context information in generative semantic communications (GenSC). TokCom is a new paradigm, motivated by the recent success of generative foundation models and multimodal large language models (GFM/MLLMs), where the communication units are tokens, enabling efficient transformer-based token processing at the transmitter and receiver. In this paper, we introduce the potential opportunities and challenges of leveraging context in GenSC, explore how to integrate GFM/MLLMs-based token processing into semantic communication systems to leverage cross-modal context effectively at affordable complexity, present the key principles for efficient TokCom at various layers in future wireless networks. In a typical image semantic communication setup, we demonstrate a significant improvement of the bandwidth efficiency, achieved by TokCom by leveraging the context information among tokens. Finally, the potential research directions are identified to facilitate adoption of TokCom in future wireless networks.

Li Qiao, Mahdi Boloursaz Mashhadi, Zhen Gao +3

School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China · 5GIC & 6GIC, Institute for Communication Systems (ICS), University of Surrey, Guildford, United Kingdom · Centre for Wireless Communications, University of Oulu, 90014 Oulu, Finland +1

Semantic Communications Token Compression Cross-Modal Transformer Architectures

Feb 13, 2025cs.LG

You Do Not Fully Utilize Transformer's Representation Capacity

In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the hidden state from the previous layer to represent the entire context. We show that this design creates pressure toward representation collapse and can degrade performance. To address this issue, we introduce Layer-Integrated Memory (LIMe), a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers. Across language modeling, synthetic reasoning, and deep architectures, LIMe improves perplexity per FLOP in the studied regimes and yields strong gains on synthetic tasks while preserving higher value-vector entropy and token separability. Finally, learned routing weights reveal systematic reuse of local and long-distance features, showing how LIMe enriches attention-time memory without increasing hidden-state size. Code is available at https://github.com/corl-team/lime.

Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2

1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University

Transformer Architectures Language Modeling Representation Collapse Representation Quality Capacity

Jan 28, 2025cs.CL

3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation

Spatial intelligence, encompassing 3D perception and reasoning, is the essential next frontier of AI. Scaling current 3D vision-language models (VLMs) that rely on dense Transformers for spatial tasks incurs prohibitive computational costs. In this paper, we introduce 3D-MoE, a 3D VLM leveraging an efficient mixture-of-experts architecture with a modality- and spatial-context-aware probabilistic routing scheme, stably cultivated by a novel routing curriculum. To seamlessly extend 3D-MoE to embodied AI, we integrate a diffusion-based action head, Pose-DiT, transforming 3D-MoE into a 3D vision-language-action (VLA) model. By employing a rectified flow framework, Pose-DiT generates precise 6D pose actions in a single sampling step. Extensive experiments demonstrate that 3D-MoE achieves superior performance on diverse 3D vision-language benchmarks with drastically fewer activated parameters and yields higher success rates while enabling real-time inference for robot manipulation tasks.

Yueen Ma, Zenglin Xu, Irwin King

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR

Diffusion-Based Vision-Language-Actions Action Generation Mixture-Of-Experts Transformer Architectures

Dec 17, 2024cs.CV

CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image

Recently, generalizable feed-forward methods based on 3D Gaussian Splatting have gained significant attention for their potential to reconstruct 3D scenes using finite resources. These approaches create a 3D radiance field, parameterized by per-pixel 3D Gaussian primitives, from just a few images in a single forward pass. However, unlike multi-view methods that benefit from cross-view correspondences, 3D scene reconstruction with a single-view image remains an underexplored area. In this work, we introduce CATSplat, a novel generalizable transformer-based framework designed to break through the inherent constraints in monocular settings. First, we propose leveraging textual guidance from a visual-language model to complement insufficient information from a single image. By incorporating scene-specific contextual details from text embeddings through cross-attention, we pave the way for context-aware 3D scene reconstruction beyond relying solely on visual cues. Moreover, we advocate utilizing spatial guidance from 3D point features toward comprehensive geometric understanding under single-view settings. With 3D priors, image features can capture rich structural insights for predicting 3D Gaussians without multi-view techniques. Extensive experiments on large-scale datasets demonstrate the state-of-the-art performance of CATSplat in single-view 3D scene reconstruction with high-quality novel view synthesis.

Wonseok Roh, Hwanhee Jung, Jong Wook Kim +6

Korea University · Google · Purdue University

3D Reconstruction 3D Gaussian Cross-Attention Transformer Architectures Geometric Guidance

Dec 14, 2024cs.CL

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Tokenization is a necessary component within the current architecture of many language mod-els, including the transformer-based large language models (LLMs) of Generative AI, yet its impact on the model's cognition is often overlooked. We argue that LLMs demonstrate that the Distributional Hypothesis (DH) is sufficient for reasonably human-like language performance (particularly with respect to inferential lexical competence), and that the emergence of human-meaningful linguistic units among tokens and current structural constraints motivate changes to existing, linguistically-agnostic tokenization techniques, particularly with respect to their roles as (1) vehicles for conveying salient distributional patterns from human language to the model and as (2) semantic primitives. We explore tokenizations from a BPE tokenizer; extant model vocabularies obtained from Hugging Face and tiktoken; and the information in exemplar token vectors as they move through the layers of a RoBERTa (large) model. Besides creating suboptimal semantic building blocks and obscuring the model's access to the necessary distributional patterns, we describe how tokens and pretraining can act as a backdoor for bias and other unwanted content, which current alignment practices may not remediate. Additionally, we relay evidence that the tokenization algorithm's objective function impacts the LLM's cognition, despite being arguably meaningfully insulated from the main system intelligence. Finally, we discuss implications for architectural choices, meaning construction, the primacy of language for thought, and LLM cognition. [First uploaded to arXiv in December, 2024.]

Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9

Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3

Single-Token Output Distributions Distributional Information Transformer Architectures Generative Artificial Intelligence

Oct 31, 2024cs.LG

A Mechanistic Study of Transformers Training Dynamics

Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the training dynamics of transformers in a controlled and interpretable setting. On the sparse modular addition task, we demonstrate that specialized attention circuits, called clustering heads, can be implemented during gradient descent to solve the problem. Our experiments show that such pathways naturally emerge during training. By monitoring the evolution of tokens via a visual sandbox, we uncover a two-stage learning and the occurrences of loss spikes due to the high curvature of normalization layers. Our findings provide several insights into patterns observed in more practical settings, such as the pretraining of large language models.

Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes

Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Transformer Architectures Pretraining Gradient Descent

Oct 16, 2024cs.CL

Tuning Language Models by Mixture-of-Depths Ensemble

Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the predictive power embedded in late layers. Interpretability tools such as the logit lens show that late-layer representations already carry largely formed, task-relevant predictions; here we ask whether that observation can be turned into an actionable training signal. We find that focusing tuning effort on these layers can yield losses comparable to those of the final layer, with complementary test-time behaviour. Building on this, we introduce a tuning framework, Mixture-of-Depths Ensemble (MoDE), which treats the late layers as an ensemble that contributes to the final logits through learned routing weights. MoDE can be applied on top of any existing tuning method (e.g., LoRA) and, in our experiments, modestly improves reasoning performance at a small parameter overhead. We present MoDE as a mechanism study showing that late-layer logits can be made directly useful for tuning, and that they can substitute for substantially larger trainable modules with comparable performance.

Haoyan Luo, Lucia Specia

University of Cambridge · Imperial College London

Large Language Model Fine-Tuning Layer-Wise Tuning Transformer Architectures Logit Lens Ensemble

Oct 15, 2024stat.ML

On Generalisation Error Bounds for Transformers

In this paper, we establish a collection of covering number bounds for linear function classes under various norm constraints on the inputs and matrices. We then combine these results with existing covering number bounds to derive improved estimates and, based on these estimates, develop generalization error bounds for single-layer Transformers. The resulting generalization bounds improve upon several existing results in the literature and, in particular, are independent of the input sequence length. Moreover, our generalization error bound decays at the rate

O(1/\sqrt{n})

, where

n

denotes the sample size, thereby improving upon existing bounds that scale as

O((\log n)/\sqrt{n})

. Furthermore, our covering number analysis explicitly incorporates rank constraints on the underlying matrix classes, allowing us to characterize how low-rank structures affect the metric entropy and, consequently, the resulting generalization bounds for Transformer architectures.

Lan V. Truong

Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), Vietnam

Generalization Bounds Transformer Architectures Low-Rank Structure

Aug 20, 2024cs.CV

Multi-view Hand Reconstruction with a Point-Embedded Transformer

This work introduces a novel and generalizable multi-view Hand Mesh Reconstruction (HMR) model, named POEM, designed for practical use in real-world hand motion capture scenarios. The advances of the POEM model consist of two main aspects. First, concerning the modeling of the problem, we propose embedding a static basis point within the multi-view stereo space. A point represents a natural form of 3D information and serves as an ideal medium for fusing features across different views, given its varied projections across these views. Consequently, our method harnesses a simple yet effective idea: a complex 3D hand mesh can be represented by a set of 3D basis points that 1) are embedded in the multi-view stereo, 2) carry features from the multi-view images, and 3) encompass the hand in it. The second advance lies in the training strategy. We utilize a combination of five large-scale multi-view datasets and employ randomization in the number, order, and poses of the cameras. By processing such a vast amount of data and a diverse array of camera configurations, our model demonstrates notable generalizability in the real-world applications. As a result, POEM presents a highly practical, plug-and-play solution that enables user-friendly, cost-effective multi-view motion capture for both left and right hands. The model and source codes are available at https://github.com/JubSteven/POEM-v2.

Lixin Yang, Licheng Zhong, Pengxiang Zhu +4

School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, Shanghai 200230, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China +1

3D Hand Motion Capture Multi-View Hand Transformer Architectures Poetry

May 15, 2024cs.CL

A Survey of Transformer-based Language Models with Focus on Efficiency

The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a survey on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles, the efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been performed on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.

Wazib Ansar, Saptarsi Goswami, Amlan Chakrabarti

A. K. Choudhury School of IT University of Calcutta, Kolkata, India · Department of Computer Science Bangabasi Morning College Kolkata, India

Transformer Architectures Efficiency Survey

Apr 9, 2024cs.CV

Efficient Concertormer for Image Deblurring and Beyond

The Transformer architecture has achieved remarkable success in natural language processing and high-level vision tasks over the past few years. However, the inherent complexity of self-attention is quadratic to the size of the image, leading to unaffordable computational costs for high-resolution vision tasks. In this paper, we introduce Concertormer, featuring a novel Concerto Self-Attention (CSA) mechanism designed for image deblurring. The proposed CSA divides self-attention into two distinct components: one emphasizes generally global and another concentrates on specifically local correspondence. By retaining partial information in additional dimensions independent from the self-attention calculations, our method effectively captures global contextual representations with complexity linear to the image size. To effectively leverage the additional dimensions, we present a Cross-Dimensional Communication module, which linearly combines attention maps and thus enhances expressiveness. Moreover, we amalgamate the two-staged Transformer design into a single stage using the proposed gated-dconv MLP architecture. While our primary objective is single-image motion deblurring, extensive quantitative and qualitative evaluations demonstrate that our approach performs favorably against the state-of-the-art methods in other tasks, such as deraining and deblurring with JPEG artifacts. The source codes and trained models will be made available to the public.

Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien +1

Gaussian Deblurring Transformer Attention Diffusion Transformers Linear Attention Transformer Architectures

Jan 10, 2024cs.CV

AdvMT: Adversarial Motion Transformer for Long-term Human Motion Prediction

Human motion prediction is a crucial capability for advanced robotic systems that interact with humans. In facilities with dynamic human-robot collaboration settings, robots must anticipate human movements to ensure safety, prevent collisions, and optimize cooperative tasks. Traditionally, motion forecasting is treated as a sequential modeling problem using historical pose data, but achieving long-term accuracy and physical realism remains challenging. We present Adversarial Motion Transformer (AdvMT), a novel approach that integrates a Transformer-based motion encoder with a temporal continuity discriminator to address these challenges. The Transformer captures rich spatio-temporal dependencies across human joints, while adversarial training with a continuity discriminator enforces smooth, natural motion trajectories that adhere to biomechanical constraints. Our training scheme includes a bone-length consistency term and adversarial loss to reduce common artifacts like pose freezing or unnatural transitions. In experiments on the Human3.6M motion dataset, AdvMT achieves state-of-the-art long-horizon prediction accuracy while also delivering robust short-term predictions. These improvements strengthen the prediction foundation for physical AI in manufacturing and human-robot collaboration, where anticipating human motion is a prerequisite for safe and efficient robot coordination.

Sarmad Idrees, Seokman Sohn, Jongeun Choi

School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea · Power Generation Lab, Korea Power Research Institute, 105, Munji-Ro, Yuseong-Gu, Daejeon, 34056, South Korea

Human Motion Prediction Long-Horizon Forecasting Human-Robot Collaboration Transformer Architectures Artificial Intelligence Systems

Nov 29, 2023cs.CL

Introduction to Transformers: an NLP Perspective

Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.

Tong Xiao, Jingbo Zhu

NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

Transformer Architectures Natural Language Processing Deep Learning

Jul 3, 2023cs.CL

VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder

The natural language generation domain has witnessed great success thanks to Transformer models. Although they have achieved state-of-the-art generative quality, they often neglect generative diversity. Prior attempts to tackle this issue suffer from either low model capacity or over-complicated architectures. Some recent methods employ the VAE framework to enhance diversity, but their latent variables fully depend on the input context, restricting exploration of the latent space. In this paper, we introduce VOLTA, a framework that elevates generative diversity by bridging Transformer with VAE via a more effective cross-attention-based connection, departing from conventional embedding concatenation or summation. Additionally, we propose integrating InfoGAN-style latent codes to enable input-independent variability, further diversifying the generation. Moreover, our framework accommodates discrete inputs alongside its existing support for continuous inputs. We perform comprehensive experiments with two types of Transformers on six datasets from three different NLG tasks to show that our approach can significantly improve generative diversity while maintaining generative quality.

Yueen Ma, Dafeng Chi, Jingjing Li +3

The Chinese University of Hong Kong · Huawei Noah’s Ark Lab

Conditional Variational Autoencoder Transformer Architectures Diversity

May 26, 2023cs.LG

Generalizing Adam to Manifolds for Efficiently Training Transformers

One of the primary reasons behind the success of neural networks has been the emergence of an array of new, highly-successful optimizers, perhaps most importantly the Adam optimizer. It is widely used for training neural networks, yet notoriously hard to interpret. Lacking a clear physical intuition, Adam is difficult to generalize to manifolds. Some attempts have been made to directly apply parts of the Adam algorithm to manifolds or to find an underlying structure, but a full generalization has remained elusive. In this work a new approach is presented that leverages the special structure of the manifolds which are relevant for optimization of neural networks, such as the Stiefel manifold, the symplectic Stiefel manifold and the Grassmann manifold: all of these are homogeneous spaces and as such admit a global tangent space representation. This is a common vector space, often called the Lie subspace, that makes the generalization of all steps in the Adam optimizer (as well as other optimizers) possible. It is thus possible to extend the Adam optimizer to manifolds without a projection step, something that was not possible before. The resulting algorithm is then applied to train transformers and a symplectic autoencoder for which orthogonality constraints are enforced up to machine precision and we conclusively demonstrate the advantage of the proposed optimizer over existing methods.

Benedikt Brantner

Max-Planck-Institut für Plasmaphysik Boltzmannstraße 2, 85748 Garching, Deutschland · Technische Universität München, Zentrum Mathematik Boltzmannstraße 3, 85748 Garching, Deutschland

Adam Manifolds Equivariant Neural Networks Transformer Architectures Subspace Tangent Space+1

Date pendingcs.LG

Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

Post-training quantization (PTQ) is a practical way to reduce the memory footprint of large language models, but low-bit quantization is sensitive to mismatches between the quantization codebook and the empirical weight/activation distributions. We revisit Benford-like leading-digit statistics as a lightweight diagnostic of scale-broad behavior in transformer tensors. Across several model families, we observe a consistent functional dichotomy: transformational nn.Linear weights tend to be Benford-like, whereas LayerNorm parameters systematically deviate. Motivated by this observation, we propose BenQ, a data-free PTQ codebook that uses a simple log-spaced grid as a proxy for scale-broad distributions and applies it selectively to transformational layers while keeping stability-critical parameters in higher precision. In 4-bit group-wise PTQ, BenQ consistently improves over uniform RTN and trades wins with NF4 across architectures and tasks, while remaining substantially simpler than optimization-based methods. We additionally report dynamic activation quantization as an exploratory stress test: the results show that log-spaced grids can reduce RTN failures in some families, but also reveal that outlier handling remains essential for reliable low-bit activation PTQ. Code is available at https://github.com/ufopcsilab/benford-quant.

Arthur Negrão, Pedro Silva, Vander L. S. Freitas +2

Postgraduate Program in Computer Science Federal University of Ouro Preto · Computing Department Federal University of Ouro Preto

Transformer Architectures Post-Training Prior Knowledge Integration

Date pendingcs.LG

On the Residual Scaling of Looped Transformers: Stability and Transferability

Looped (weight-tied) Transformers apply a shared residual block

N

times (

h \leftarrow h + \varepsilon\,f(h)

, same

f

at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe

\varepsilon = 1/\!\sqrt{L}

for depth-

L

residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling

\varepsilon = 1/N

. For multi-layer blocks (

L

unique layers looped

N

times), we derive a factored parameterization

\varepsilon = \lambda/(N\!\sqrt{L})

that separates the two sources of growth:

1/N

controls the within-layer loop correlation, and

1/\!\sqrt{L}

controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers

L

, not on the loop count

N

, enabling direct hyperparameter transfer from small to large

N

without retuning. Experiments on looped Transformers confirm that

1/N

scaling improves trainability and yields better loss than

1/\!\sqrt{N}

scaling across loop counts.

Shaowen Wang, Bingrui Li, Ge Zhang +3

Tsinghua University · ByteDance Seed · M-A-P

Residual Networks Transformer Architectures Batch

Date pendingcs.LG

Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.

Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh +1

School of Computer and Cyber Sciences, Augusta University · Department of Computer Science, University of Virginia · Qualcomm AI Research

Accelerated Sampling Transformer Architectures In-Context Learning In-Context

Date pendingcs.CV

Debunking Grad-ECLIP: A Comprehensive Study on Its Incorrectness and Fundamental Principles for Model Interpretation

Grad-ECLIP is published at ICML 2024 and represents a new Transformer interpretation technical route (intermediate features-based). First, this paper demonstrates that the intermediate features-based technical route is not a novel one. Based on the existing attention-based route, we have developed Attention-ECLIP, which is completely equivalent to Grad-ECLIP but with simpler computation. Both through formal derivation and experimental validation, we prove that the intermediate feature-based route represented by Grad-ECLIP is actually an equivalent variant of the attention-based route. Next, this paper demonstrates that the Grad-ECLIP method is flawed. The model interpretation results obtained by Grad-ECLIP are not those of the original model, and the interpretation results are misaligned with the model's performance. We analyze the causes of Grad-ECLIP's flaws and propose, or rather, explicitly emphasize two fundamental principles that model interpretation should adhere to in order to avoid similar errors.

Yongjin Cui, Xiaohui Fan

Zhejiang University

Gradient-Weighted Class Activation Mapping Cvpr Transformer Architectures Systematic Study

Date pendingcs.CV

UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation

Content-aware layout generation is a critical task in graphic design automation, focused on creating visually appealing arrangements of elements that seamlessly blend with a given background image. The variety of real-world applications makes it highly challenging to develop a single model capable of unifying the diverse range of input-constrained generation sub-tasks, such as those conditioned by element types, sizes, or their relationships. Current methods either address only a subset of these tasks or necessitate separate model parameters for different conditions, failing to offer a truly unified solution. In this paper, we propose UniLayDiff: a Unified Diffusion Transformer, that for the first time, addresses various content-aware layout generation tasks with a single, end-to-end trainable model. Specifically, we treat layout constraints as a distinct modality and employ Multi-Modal Diffusion Transformer framework to capture the complex interplay between the background image, layout elements, and diverse constraints. Moreover, we integrate relation constraints through fine-tuning the model with LoRA after pretraining the model on other tasks. Such a schema not only achieves unified conditional generation but also enhances overall layout quality. Extensive experiments demonstrate that UniLayDiff achieves state-of-the-art performance across from unconditional to various conditional generation tasks and, to the best of our knowledge, is the first model to unify the full range of content-aware layout generation tasks.

Zeyang Liu, Le Wang, Sanping Zhou +4

College of Artificial Intelligence, Xi’an Jiaotong University · Amazon.com, Inc. · Pixocial Technology

Layouts High-Fidelity Conditional Generation Diffusion Transformers Transformer Architectures Schema

Date pendingcs.LG

BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classification

The proliferation of Internet-of-Things (IoT) devices has led to an unprecedented volume of multivariate time series (MTS) data, requiring efficient and accurate processing for timely decision-making in resource-constrained edge environments. Hyperdimensional (HD) computing, with its inherent efficiency and parallelizability, has shown promise in classification tasks but struggles to capture complex temporal patterns, while Transformers excel at sequence modeling but incur high computational and memory overhead. We introduce BiHDTrans, an efficient neurosymbolic binary hyperdimensional Transformer that integrates self-attention into the HD computing paradigm, unifying the representational efficiency of HD computing with the temporal modeling power of Transformers. Empirically, BiHDTrans outperforms state-of-the-art (SOTA) HD computing models by at least 14.47% and achieves 6.67% higher accuracy on average than SOTA binary Transformers. With hardware acceleration on FPGA, our pipelined implementation leverages the independent and identically distributed properties of high-dimensional representations, delivering 39.4 times lower inference latency than SOTA binary Transformers. Theoretical analysis shows that binarizing in holographic high-dimensional space incurs significantly less information distortion than directly binarizing neural networks, explaining BiHDTrans's superior accuracy. Furthermore, dimensionality experiments confirm that BiHDTrans remains competitive even with a 64% reduction in hyperspace dimensionality, surpassing SOTA binary Transformers by 1-2% in accuracy with 4.4 times less model size, as well as further reducing the latency by 49.8% compare to the full-dimensional baseline. Together, these contributions bridge the gap between the expressiveness of Transformers and the efficiency of HD computing, enabling accurate, scalable, and low-latency MTS classification.

Jingtao Zhang, Yi Liu, Qi Shen +1

School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University · College of Medicine and Biological Information Engineering, Northeastern University

Hybrid Transformer Time-Series Classification Transformer Architectures Multivariate Time-Series Neuro-Symbolic Framework

Date pendingcs.CL

Tracing Computation Density in LLMs

Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their full capacity for all inputs. We introduce the s-Trace method to efficiently estimate a subgraph of size s that approximates a full model output. With this method, we find the computation in a variety of LLMs to be organized in two distinct phases. A small subgraph mostly composed of early-layer nodes can reconstruct the head of the full model output distribution. Adding further nodes, mostly located in later layers and increasingly consisting of attention heads, leads to incremental refinements in approximating the full output distribution. We find moreover that the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics, such as unigram frequency. Overall, our results suggest a consistent modular organization in effective LLM computation, with a sparse early-layer core providing a rough prediction that is further refined through denser computations in later layers.

Corentin Kervadec, Iuliia Lysova, Iuri Macocco +2

Universitat Pompeu Fabra · ICREA

Transformer Architectures Traces

Date pendingcs.LG

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.

Shaowen Wang, Ge Zhang, Kairong Luo +6

Tsinghua University · ByteDance Seed · M-A-P

Transformer Architectures Scaling Laws

Date pendingcs.SD

Musical Attention Transformer: Music Generation Using a Music-Specific Attention Model

This study aims to enhance the quality of music generation using Transformers by incorporating meta-information. While Transformer-based approaches are effective at capturing long-term dependencies in musical compositions, the music they generate often suffers from issues such as excessive repetition or duplication of notes, leading to unnatural melodies. To address these limitations, we propose Musical Attention, a mechanism that incorporates meta-information such as bar numbers, key, signatures, and tempos into the attention process. Musical Attention explicitly leverages both the structural properties of music and its associated metadata, enabling the Transformer's attention mechanism to operate more effectively and thereby improving the quality of the generated output. In our framework, each musical note is represented as a combination of five events-pitch, bar number, onset, duration, and velocity in addition to the three metadata elements. The attention mechanism is then modified to reflect the correlations among these eight features, allowing the model to better capture the inherent characteristics of musical composition. Experimental results demonstrate that the model incorporating Musical Attention outperforms prior methods, such as Full Attention and Strided Attention, in terms of musical coherence, variation, and overall quality. Notably, it significantly reduces repetition and enhances the model's ability to generate diverse, harmonically consistent melodies. Musical Attention thus represents a meaningful advancement in AI-driven music generation, facilitating the creation of more natural and expressive compositions.

Shinnosuke Takasuka, Hideo Mukai

Department of Computer Science, School of Science and Technology, Meiji University, 1-1-1 Higashimita, Tama-ku, Kawasaki, Kanagawa 214-8571, Japan

Transformer Architectures Metadata

Date pendingcs.CV

Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packaging

Vision Transformers (ViTs) have achieved strong visual recognition performance, yet their deployment in resource-constrained industrial environments remains limited. The main challenges are their high computational cost, memory requirements, and energy consumption. While individual efficiency techniques such as neural architecture search (NAS), token compression, and low-precision inference have been extensively studied, most prior work targets only a single optimization axis, limiting overall deployment gains while preserving accuracy. In this paper, we present one of the first holistic frameworks that jointly optimizes three complementary axes: architecture, token, and bit-width. Specifically, the framework identifies compact backbones via Neural Architecture Search (AutoFormer), reduces information processing via token merging (ToMe), and accelerates per-operation execution via fp16 mixed-precision inference. In our study, we analyze accuracy-efficiency trade-offs on ImageNet-1K under aggressive compression. We then apply the selected configurations to a real-world in-house 3D X-ray semiconductor defect classification dataset for IC chip packaging inspection. Results show that the proposed multi-axis framework achieves more than

10\times

improvement in throughput along with over

10\times

reductions in parameter count, FLOPs, and energy consumption, while maintaining the required accuracy on the downstream industrial task. To the best of our knowledge, this is among the earliest works to jointly optimize architecture, token, and bit-width dimensions in ViTs and the first such resource-efficient, deployment-focused study for semiconductor manufacturing.

Phat Nguyen, Xue Geng, Kaixin Xu +3

Singapore University of Technology and Design (SUTD) · Agency for Science, Technology, and Research (A*STAR)

Vision Transformer Token Compression Neural Architecture Search Semiconductor Manufacturing Transformer Architectures