Transformer

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 44% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 351

All topics

Marformer: A Transformer for Predicting Missing Data Distributions
Oct 8, 2026Prabhav Singh, Xiheng Tom Wang, Haojun Shi +1Learning with Missing Data Conditional Distribution Estimation
School of Computing, College of Natural Sciences The University of Texas at Austin, Austin TX · Department of Computer Science Johns Hopkins University, Baltimore MD · Department of Computer Science Yale University, New Haven, CT
Causal-fate dynamics of unrealized influence
Oct 8, 2026Yiwei Liu, Luwei Yang, Shunbo LeiLatent Dynamics Modeling Causal Inference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, 518172, China. · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen, 518172, China.
How Do Transformers Learn to Represent Symmetries?
Oct 7, 2026Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1Invariant Representation Learning Transformer
Technical University of Munich Munich Center for Machine Learning · Massachusetts Institute of Technology
Leaner Transformers Can Easily Learn to Cluster
Oct 7, 2026Charlotte Park, Kenneth L. Clarkson, Lior Horesh +2Transformer Clustering
MIT · IBM Research
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Oct 6, 2026Shikhar Srivastava, Christopher KananTransformer Positional Encoding
University of Rochester
Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
Oct 6, 2026Yiping BaiLanguage Model Pretraining Language Modeling
Guangdong Haiqixing Marine Technology Co., Ltd.
No Transformer Beats Six Covariates: Long-Horizon Prediction of Depressive Symptoms from Childhood Essays
Oct 6, 2026Daniel Kua, Emrul Hasan, John-Jose Nunez +1Transformer Depression Detection
Department of Statistics, University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada · Department of Psychiatry, University of British Columbia, Vancouver, Canada +1
Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
Oct 6, 2026Soichiro KumanoAdversarial Training Transformer
LY Corporation
Stability of Measure-to-Measure Transformers on Sub-Gaussian Data
Oct 6, 2026Frank Cole, Nicholas H. Nelsen, Takashi FuruyaTransformer Statistical Learning Theory
UCLA · UT Austin · Doshisha University, RIKEN AIP
Exact-Solution Volume and Length Generalization in Transformers
Oct 6, 2026Yijia Jessica Zhu, David ChiangTransformer Expressivity Transformer
University of Notre Dame
Deep Learning for Sleep Heart Rate Estimation from Accelerometers: Toward Population-Scale Cardiac Insight Without Optical Sensors
Oct 5, 2026Tanbin Islam Rohan, Pranjol Sen Gupta, Tanusree Debi +1Transformer Wearable Sensing
Department of Information Technology Kennesaw State University Kennesaw, GA, USA · School of Computing University of Georgia Athens, GA, USA
MatrixFormer: A Foundation Model for Matrix Completion
Oct 5, 2026Dwaipayan Saha, Jacob Feitelberg, Kyuseong Choi +2Incomplete Data Imputation Transformer
Cornell Tech
MIRT: Transformers for Truthful Generative Auctions with Whole-feed Permutation Externalities
Oct 5, 2026Ali Elahi, Ermis Soumalias, Jason Cheuk Nam Liang +2Transformer Learning to Rank
University of Illinois Chicago · Meta
A Physics-Guided Transformer Framework for Electromigration Analysis in Multi-Segment Interconnects
Oct 5, 2026Pavlos Stoikos, Anuj Pathania, George FlorosNeural Surrogate Modeling Transformer
Dept. of Electrical and Computer Engineering University of Thessaly, Greece · University of Amsterdam Amsterdam, The Netherlands · Dept. of Electronic and Electrical Engineering, Trinity College Dublin, Ireland
SO(3)-RoPE for Spherical Transformers
Oct 5, 2026Christian Libner, Chase van de Geijn, Alexander S. Ecker +1Rotary Positional Embeddings Transformer
Institute of Computer Science & CIDAS, University of Göttingen · Max Planck Institute for Dynamics and Self-Organization, Göttingen · MIT CSAIL
Faynt: Scaling and Optimizing Policies for Competitive Melee
Oct 1, 2026Ali Janati, Nikita Kuzmin, Rohit Swamy +1Reinforcement Learning Self-Play RL
Frisson Labs
Universal interpolation for deep residual self-attention networks
Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax Attention Transformer
Department of Computer Science New York University
Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers
Oct 1, 2026Cigdem Kokenoz, Amir Salarpour, Alkim Domeke +5Open-Vocabulary 3D Segmentation Transformer
Department of Automotive Engineering, Clemson University, Greenville, SC, USA. · School of Computing, Clemson University, Clemson, SC, USA.
Association profile conditioning in a set-temporal transformer for cross-session intracortical motor decoding
Sep 30, 2026Xinyuan Zhang, Handong Mo, Pengfei Wen +5Transformer Neural Decoding
The University of Hong Kong, Hong Kong SAR, China · Southern University of Science and Technology, Shenzhen, China
Function-Space Transformer with Adaptive Anchors
Sep 29, 2026Guorui Sang, Pedram RooshenasRepresentation Learning Visual Representation Learning
University of Illinois Chicago
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Sep 29, 2026Dor Tirosh, Ido Amos, Mor GevaLanguage Model Pretraining Transformer
Blavatnik School of Computer Science and AI, Tel Aviv University · The Hebrew University of Jerusalem
The Geometry of Inference in Transformer Residual Streams
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Interpretability Transformer Inference
University of Luxembourg, Luxembourg · London Institute for Mathematical Sciences, London, UK
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Sep 29, 2026Zehao Jin, Ruixuan Deng, Junran WangTransformer LLM Fine-Tuning
Georgia Institute of Technology
Universal Approximation of Measure-to-Measure Operators by Pushforwards
Sep 28, 2026Takashi Furuya, Nicholas H. Nelsen, Frank ColeTransformer Universal Approximation
Doshisha University, RIKEN AIP · UT Austin · UCLA
Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation
Sep 28, 2026Yuanqing Ma, Zhenrui Zheng, Chenjun XiaoTransformer In-Context RL
The Chinese University of Hong Kong, Shenzhen
SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models
Sep 28, 2026Tianxiang Chen, Zhentao Tan, Zi Ye +9Efficient Multimodal Inference Transformer
End-User Intelligent Computing BU, Alibaba Cloud · Alibaba Group · Department of Computer Science, Maynooth University +2
Geometric Inductive Biases for Semi-Supervised Equalization: The Constellation-Aware Transformer
Sep 27, 2026Avi CaciularuTransformer Wireless Communications
Google Research
Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking
Sep 24, 2026Zhiyu Zhang, Yupeng LiTransformer Recurrent Neural Networks
Conflux Labs Ltd · Michigan State University
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
Sep 24, 2026Zhongjie Shi, Rongjie Lai, Alexander Cloninger +1Transformer Multi-Task Learning
School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, United States · Department of Mathematics, Purdue University, West Lafayette, IN 47907, United States · Department of Mathematics and Halıcıoğlu Data Science Institute, University of California, San Diego, La Jolla, CA 92093, United States
PBLH Estimation from Satellite Radiances via a Dual-Encoder Transformer
Sep 23, 2026Lorenzo Innocenti, Luca Catalano, Edoardo Arnaudo +4Transformer Remote Sensing
Politecnico di Torino, Torino, Italy · Fondazione LINKS, Torino, Italy · Institute of Integrated Methodologies for Earth Observation, National Research Council (IMIOT/CNR), Potenza, Italy +1

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Marformer: A Transformer for Predicting Missing Data Distributions

Prabhav Singh, Xiheng Tom Wang, Haojun Shi, Jason Eisner

School of Computing, College of Natural Sciences The University of Texas at Austin, Austin TX · Department of Computer Science Johns Hopkins University, Baltimore MD · Department of Computer Science Yale University, New Haven, CT

Learning with Missing Data Conditional Distribution Estimation Transformer

Open paper AskPDF arXiv

Real decisions are made under incomplete information. If we observe only some of the random variables we need, we can predict the others. The \textbf{conditional marginals} over the missing variables are the key ingredient for computing Bayes risk and Value of Information (VOI), the expected gain from acquiring one more observation before deciding. We present the Marformer, a Transformer trained to directly predict conditional marginals given any set of observed values. Like BERT, which is trained to predict missing words from context, the Marformer constructs a hidden-vector representation for each distribution

p(X_i)

and iteratively refines it through attention to other distributions

p(X_j)

. Unlike generative approaches, the Marformer does not model the full joint distribution, requires no domain knowledge of the data-generating process, and makes all predictions in a single forward pass. We evaluate across three synthetic domains with missing data---Bayesian networks, discretized multivariate Gaussians, and structured annotation data. The Marformer can match or outperform classical missing-data methods, even when those methods are given the true model family and prior that generated the synthetic data. We also evaluate on a real annotation dataset, where the Marformer outperforms the evaluated baselines at the largest training size. In both cases, the Marformer is substantially faster than the evaluated generative baselines.

Momentum

Related topics

Latest papers 351