Sequence Modeling

Follow this topic Ask about this topic

Momentum

40 papers in the last four weeks, up 135% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 265

All topics

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention
Mar 16, 2026Jeffrey D. VarnerSequence Modeling Feature Alignment
R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
Mar 11, 2026Shirin Amiraslani, Xin GaoSequence Modeling Higher-Order Structures
Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada
Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1Transformer Architectures Positional Encoding
University of Edinburgh · Heriot-Watt University
HiSAC: Hierarchical Sparse Activation Compression for Ultra-long Sequence Modeling in Recommenders
Feb 24, 2026Kun Yuan, Junyu Bi, Daixuan Cheng +5Real-World Content Recommendation Problem Recommendation
Alibaba Group · Beijing, China · Renmin University
Adaptive Time Series Reasoning via Segment Selection
Feb 20, 2026Shvat Messica, Jiawen Zhang, Kevin Li +2Time Series Large Reasoning Models
Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA · The Hong Kong University of Science and Technology (Guangzhou) · Massachusetts Institute of Technology +1
Learning State-Tracking from Code Using Linear RNNs
Feb 16, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3State-Tracking Sequence Modeling
University of Freiburg · Prior Labs · Microsoft Research +2
PRISM: Parallel Residual Iterative Sequence Model
Feb 11, 2026Jie Jiang, Ke Cheng, Xin Xu +8Sequence Modeling Transformer Residual Streams
Tencent AMS., Beijing, China · State Key Laboratory of software development environment, School of Computer Science and Engineering, Beihang University, Beijing, China · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, China.
Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences
Feb 6, 2026Alessio Pecilli, Matteo RosatiVariational Quantum Algorithms Transformer Attention
Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy
Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks
Feb 5, 2026Kevin Lu, Jannik Brinkmann, Stefan Huber +4Protein Structure Prediction Sequence Modeling
1Northeastern University · 2TU Clausthal · 3Harvard University +2
Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing
Feb 2, 2026Egor Serov, Ilya Kuleshov, Alexey ZaytsevNeural Ordinary Differential Equations Learning-Based Control
Applied AI Institute · Risk AI Research Lab
Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification
Jan 17, 2026Xiaomei Yang, Antai Liu, Xizhan Gao +3Re-Identification Cross-Modal Attention
Shandong Key Laboratory of Ubiquitous Intelligent Computing, School of Information Science and Engineering, University of Jinan, Jinan 250022, China · College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University, Nanjing 210037, China · Department of Informatics, Modeling, Electronics, and Systems, University of Calabria, Via P. Bucci, cubo 41C, 87036, Rende (CS), Italy
From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP
Jan 12, 2026Adithya V Ganesan, Vasudha Varadarajan, Oscar NE Kjell +6Longitudinal Natural Language Processing
Stony Brook University · Vanderbilt University · Carnegie Mellon University +3
DuaDeep-SeqAffinity: Dual-Branch Deep Learning for Tri-Stream Sequence-Based Antibody--Antigen Affinity Prediction
Dec 26, 2025Aicha Boutorh, Soumia Bouyahiaoui, Manel Kara Laouar +3Antibody Sequence Modeling
National School of Artificial Intelligence (ENSIA) Sidi Abdallah Campus, Algiers, Algeria · Lamine Debaghine University Hospital Center, CHU Bab El Oued, Algiers, Algeria
Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling
Dec 22, 2025Sutashu Tomonaga, Kenji Doya, Noboru MurataState Space Models State Space
Okinawa Institute of Science and Technology · Waseda University
Scriboora: Rethinking Human Pose Forecasting
Nov 19, 2025Daniel Bermuth, Alexander Poeppel, Wolfgang ReifWifi-Csi 3D Human Pose Estimation Human Motion Prediction
University of Augsburg, Germany
HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
Sep 28, 2025Md Mozaharul Mottalib, Thao-Ly T. Phan, Rahmatollah BeheshtiMedical World Model Hybrid Transformer
University of Delaware Newark, Delaware, USA · Thomas Jefferson University & Nemours Children’s Health Wilmington, Delaware, USA
Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
Sep 25, 2025Qiyu Chen, Guozhang ChenState Space Models Inductive Bias
School of Computer Science, National Key Laboratory for Multimedia Information Processing, Peking University, China · School of Physics, Peking University, China
Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Aug 22, 2025Yue Pei, Hongming Zhang, Chao Gao +7Goal-Conditioned Value Function Goal-Conditioned Reinforcement Learning
School of Artificial Intelligence, Beihang University, Beijing, China · Department of Computing Science and Amii, University of Alberta, Edmonton, Canada · Edmonton Research Center, Huawei Canada, Edmonton, Canada +2
Query Brand Entity Linking in E-Commerce Search
Feb 3, 2025Dong Liu, Sreyashi NagBrand-As-Memory Semantic Relationships
Amazon Luxembourg, Luxembourg · Amazon Palo Alto, California, United States
Unleash LLMs Potential for Sequential Recommendation by Coordinating Dual Dynamic Index Mechanism
Sep 14, 2024Jun Yin, Zhengxin Zeng, Mingzheng Li +11Multimodal Recommendation Model Sequential Recommendation
Central South University Changsha, China · Microsoft AI Beijing, China · Microsoft Research Asia Beijing, China +1
RotRNN: Modelling Long Sequences with Rotations
Jul 9, 2024Kai Biegun, Rares Dolga, Jake Cunningham +1Recurrent Model Sequence Modeling
UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK
ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks
Feb 26, 2024Laura Kriener, Ben von Hünerbein, Kristin Spicher +5Synaptic Plasticity Recurrent Neural Networks
Department of Physiology, University of Bern, 3012 Bern, Switzerland. · Institute of Neuroinformatics, University of Zurich and ETH Zurich, 8057 Zurich, Switzerland. · Catlab Engineering GmbH, 82284 Grafrath, Germany.
From system models to class models: An in-context learning paradigm
Aug 25, 2023Marco Forgione, Filippo Pura, Dario PigaSystem Identification In-Context Learning
IDSIA Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Via la Santa 1, CH-6962 Lugano-Viganello, Switzerland.
When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification
Date pendingSarwan Ali, Taslim Murad, Imdadullah Khan +1Contextual Embeddings Locality-Sensitive Hashing
Columbia University, NY, USA · Institute of Business Administration, Karachi, Pakistan · Lahore University of Management Sciences, Lahore, Pakistan +1
SIDE: Sensor Impersonation Detection at the Edge via Sequence Prediction
Date pendingNahom BirhanInternet Of Thing Sensors
Department of Electrical and Computer Engineering Old Dominion University Virginia, USA

Mar 16, 2026, cs.LGJ/K move · Enter open · S save

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

Jeffrey D. Varner

R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850

Sequence Modeling Feature Alignment Stochastic

Open paper AskPDF arXiv

Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples. Yet most protein families are small: the median Pfam seed alignment contains only 22 sequences, a regime where learned models overfit or collapse. We propose \emph{stochastic attention} (SA), a training-free sampler that treats the modern Hopfield energy over stored sequences as a Boltzmann distribution and draws samples via Langevin dynamics. The score function is the residual of a single softmax attention operation, eliminating the need for a trained score network, pretraining data, or graphics processing units (GPUs). Across eight Pfam families spanning 37 to 420 sequences and 23 to 262 residues, SA generates sequences with low composition divergence, novelty, and structural plausibility supported by ESMFold and AlphaFold2. Compared with profile hidden Markov models (HMMs), EvoDiff, and the multiple sequence alignment (MSA) Transformer, SA is the only tested method to simultaneously achieve low composition divergence, genuine novelty, and sequence identity within each family's nearest-neighbor identity range; the others drift outside this range or produce near-copies. The critical inverse temperature is predicted from principal component analysis (PCA) dimensionality alone, enabling fully automatic operation from a seed alignment. In two domains with deep mutational scanning data, SA-generated substitutions are enriched for experimentally tolerated mutations beyond a position-matched null, and an independent language model (ESM2-650M) scores them within the natural range. Stochastic attention thus opens training-free sequence generation to the long tail of protein families too small for deep learning.

Momentum

Related topics

Latest papers 265