Transformer

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 44% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 351

All topics

SPADE: Split-and-Delay Embeddings for Autoregressive High-Granularity Calorimeter Simulation
Jun 9, 2026Joschka Birk, Frank Gaede, Anna Hallin +3Transformer Point Cloud Learning
Institute for Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany · Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany
GRAFT: Gain-Recalibrated Adapters for Transformer-Based Neural Population Activity Modeling
Jun 9, 2026Xiangsheng Ge, Yang XieBrain-Computer Interfaces Transformer
Shanghai Jiao Tong University
Tight Sample Complexity of Transformers
Jun 8, 2026Chenxiao Yang, Nathan Srebro, Zhiyuan LiTransformer Expressivity Transformer
Toyota Technological Institute at Chicago (TTIC)
A Universal Dense Football Event Representation Based on TabTransformer
Jun 8, 2026Weiran Yang, Daniel Memmert, Maximilian Klemp-WeinsRepresentation Learning Feature Interaction Modeling
Institute of Exercise Training and Sport Informatics, German Sport University Cologne, Cologne, Germany
Trajectory Geometry of Transformer Representations Across Layers
Jun 8, 2026Vishal Pandey, Gopal Singh, Yacine MahdidTransformer Interpretability Neural Representation Geometry
London, UK · Athens, GR · Montreal, CA
Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions
Jun 7, 2026Blanka Köver, Alexandra Butoi, Anej Svete +2Transformer Expressivity Boolean Function Learning
Department of Mathematics, ETH Zürich, Zürich, Switzerland · Department of Computer Science, ETH Zürich, Zürich, Switzerland · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany.
Chiaroscuro Attention: Spending Compute in the Dark
Jun 6, 2026Prateek Kumar SikdarTransformer Transformer Attention
Accenture, India
SegmentAnyTreeV2: Scaling Transformer-Based Tree Instance Segmentation Across Sensors, Platforms, and Forests
Jun 6, 2026Maciej Wielgosz, Stefano Puliti, Rasmus AstrupTransformer Instance Segmentation
Norwegian Institute of Bioeconomy Research (NIBIO), Høgskoleveien 7, 1433 Ås, Norway
TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding
Jun 6, 2026Mahbub E Sobhani, Anika Tasnim Rodela, Chowdhury Mofizur Rahman +2Transformer Autoencoders
United International University, Department of Computer Science and Engineering Dhaka, Bangladesh · BRAC University, Department of Computer Science and Engineering Dhaka, Bangladesh · Southeast University, Department of Computer Science and Engineering Dhaka, Bangladesh
RETROSPECT: RETROsynthesis via Sequential Prediction, and Chemically Transformed-ranking
Jun 5, 2026Raja Sekhar Pappala, Shreyas Vinaya Sathyanarayana, Ronit Kumar Choudhary +2Transformer Learning to Rank
Mstack AI
An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection
Jun 5, 2026Carl Lochstampfor, Ayan RoyTransformer Synthetic Data Generation
Department of Computer Science Old Dominion University Norfolk, VA, USA · Department of Computer Science Christopher Newport University Newport News, VA, USA
PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
Jun 4, 2026Senmiao Wang, Tiantian Fang, Haoran Zhang +4Language Model Pretraining Transformer
The Chinese University of Hong Kong, Shenzhen, China · Google LLC, Mountain View, CA, United States · Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen2026 Research Institute of Big Data +1
HAARES Half-Split Residual Basis Routing for Deep Transformers
Jun 4, 2026Kehan WangTransformer Residual Learning
Paper under double-blind review
ChessMimic: Per-Rating Transformer Models for Human Move, Clock, and Outcome Prediction in Online Blitz Chess
Jun 3, 2026Thomas JohnsonGame-Playing Agents Transformer
Nascent
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Generalizable Multi-Task Learning for Wireless Networks Using Prompt Decision Transformers
Jun 3, 2026Fatih Temiz, Shavbo Salehi, Melike Erol-KantarciReinforcement Learning Transformer
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Jun 2, 2026Zekun Qi, Xuchuan Chen, Dairu Liu +10Zero-Shot Learning Transformer
Tsinghua University · Galbot Inc. · Shanghai Jiao Tong University +2
Dynamic Short Convolutions Improve Transformers
Jun 2, 2026Oliver Sieberling, Bharat Runwal, Rameswar Panda +1Dynamic Convolution Transformer
Massachusetts Institute of Technology · MIT-IBM Watson AI Lab
Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
Jun 2, 2026Ekaterina Alimaskina, Gleb Molodtsov, Aleksandr BeznosikovTransformer Hyper-Connections
1MIRAI, Moscow, Russia · 2BRAIn Lab, Moscow, Russia · 3Yandex Research, Moscow, Russia +1
Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers
Jun 2, 2026Nishit SinghTransformer Interpretability Transformer
Birla Institute of Technology and Science, Pilani
Evaluating Transformer and LSTM Frameworks for Prediction in Ungauged Basins
Jun 1, 2026Taye Akinrele, James Halgren, Noorbakhsh Amiri Golilarz +2Transformer Time Series Forecasting
Dept. of Computer Science, The University of Alabama, Tuscaloosa, USA · Alabama Water Institute, USA
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
Jun 1, 2026Elia Cunegatti, Marcus Vukojevic, Erik Nielsen +1LLM Pruning LLM Compression
University of Trento
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengTransformer Expressivity Transformer
Shenzhen Research Institute of Big Data · University of Southern California
Structure and Scale in Simplicial Sequence Modelling
May 31, 2026Matthew Farrugia-RobertsRepresentation Geometry Transformer
Department of Computer Science, University of Oxford
Emergent Ordinal Geometry in Transformers Trained on Local Comparisons
May 31, 2026Nishit SinghLogical Reasoning Neural Representation Geometry
Birla Institute of Technology and Science, Pilani India
Prior-Guided Multi-Omic Transformers for Single-Cell Gene Regulatory Network Inference
May 30, 2026Tianyang Xu, Tianci Liu, Niraj Rayamajhi +4Transformer Gene Regulatory Network Inference
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA · Department of Horticulture and Landscape Architecture, Purdue University, West Lafayette, IN, USA · School of Biological Sciences, Illinois State University, Normal, IL, USA
ReFLEX: Length-Generalizable CSI Denoising for MIMO-OFDM via Relative-Frequency Bias
May 29, 2026Zhibin Zhang, Robert Potekhin, Ziwei Wan +2Transformer Length Generalization
Moscow Institute of Physics and Technology (State University), Moscow, Russia · Yangtze Delta Region Academy, Beijing Institute of Technology (Jiaxing), Jiaxing 314019, China · School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Assign and Add: A Mechanistic Study of Compositional Arithmetic
May 29, 2026Brady Exoo, Alberto Bietti, John SousTransformer Interpretability Transformer
Yale University · Flatiron Institute
Fixed Universal Transformers
May 29, 2026Jingwen Liu, Alexandr Andoni, Daniel HsuTransformer Expressivity Transformer
Columbia University New York, NY
HADT: A Heterogeneous Multi-Agent Differential Transformer for Autonomous Earth Observation Satellite Cluster
May 29, 2026Mohamad A. Hady, Muhammad Anwar Masum, Siyi Hu +3Transformer Remote Sensing
School of Computer Science and Information Technology, Adelaide University, Adelaide, 5095, SA, Australia. · School of Electrical Engineering, Computing and Mathematical Sciences (EECMS), Curtin University, Kent St, Bentley, 6102, WA, Australia. · Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland.

Jun 9, 2026, physics.ins-detJ/K move · Enter open · S save

SPADE: Split-and-Delay Embeddings for Autoregressive High-Granularity Calorimeter Simulation

Joschka Birk, Frank Gaede, Anna Hallin, Gregor Kasieczka+2

Institute for Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany · Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany

Transformer Point Cloud Learning Physics-Based Simulation

Open paper AskPDF arXiv

We introduce SPADE (SPlit And Delay Embeddings), an autoregressive transformer for sequences whose tokens carry multiple features. Rather than embedding these features jointly, SPADE embeds them independently. Delaying each feature stream relative to the previous one allows intra-token correlations to be learned by the standard self-attention mechanism. Applied to point-cloud calorimeter shower generation in the highly granular ILD detector, SPADE is competitive with the state of the art AllShowers model on photon showers, and substantially outperforms its VQ-VAE-based predecessor OmniJet-

α_C

. The mechanism is applicable to any generative task with multi-feature tokens, enabling LLM-style pretraining workflows for higher-dimensional data.

Momentum

Related topics

Latest papers 351