Neural Sequence Model

Neural sequence models aim to process and generate sequential data, such as text or music, by learning patterns and relationships within the sequences. Current research focuses on understanding how these models, particularly transformers and recurrent neural networks, achieve in-context learning and compositional generalization, investigating their internal mechanisms and exploring algorithmic interpretations of their behavior. This research is significant for advancing our understanding of artificial intelligence and has implications for various applications, including natural language processing, machine translation, and symbolic music understanding, by improving model performance and robustness.

Papers

June 17, 2024

Delay Embedding Theory of Neural Sequence Models
Mitchell Ostrow, Adam Eisen, Ila Fiete
Language Model Theoretical Understanding Sequence Model Intermediate Layer Account Time Delay Neural Sequence Model Delay Embeddings

January 23, 2024

In-Context Language Learning: Architectures and Algorithms
Ekin Akyürek, Bailin Wang, Yoon Kim, Jacob Andreas
Language Model Practical Algorithm Context Learning Architecture Perspective Natural Language Model Neural Sequence Model

December 13, 2023

N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding
Jinhao Tian, Zuchao Li, Jiajia Li, Ping Wang
Symbolic Music Neural Sequence Model Music Sequence Feature Splatting

November 14, 2023

Transformers can optimally learn regression mixture models
Reese Pathak, Rajat Sen, Weihao Kong, Abhimanyu Das
Transformer Megatron Decepticons Model Based Mixture Model Neural Sequence Model Optimal Prediction Exponential Weight

August 24, 2023

A Brain-Inspired Sequence Learning Model based on a Logic
Bowen Xu
Sequence of Sequence Logic Reasoning Sequence Prediction Learning Mechanism Neural Sequence Model Sequence Learning

June 7, 2023

Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, Song Mei
Transformer Megatron Decepticons Context Learning Algorithm Selection Neural Sequence Model Statistical Task

June 5, 2023

Learning to Substitute Spans towards Improving Compositional Generalization
Zhaoyi Li, Ying Wei, Defu Lian
LeArning Abstract Inductive Bias Compositional Generalization Neural Sequence Model Compositional Data Augmentation Substitute Span

May 30, 2023

Grokking of Hierarchical Structure in Vanilla Transformers
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning
Transformer Language Model Hierarchical Structure Grokking Phenomenon Vanilla Transformer Neural Sequence Model Shallow Transformer

January 18, 2023

Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection
Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, Marine Carpuat
Neural Machine Translation Human Understanding Hallucination Detection Type II Hallucination Neural Sequence Model

November 28, 2022

What learning algorithm is in-context learning? Investigations with linear models
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou
Practical Algorithm Context Learning Comprehensive Investigation Implicit Model Linear Model Neural Sequence Model Context Learner Transformer Based in Context

October 18, 2022

Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models
Patrick Haller, Andreas Säuberli, Sarah Elisabeth Kiener, Jinger Pan, Ming Yan, Lena Jäger
Chinese Character Eye Tracking Eye Movement Neural Sequence Model Gaze Pattern Language Stimulus Chinese Learner Dyslexic Student

June 5, 2022

Rapid Learning of Spatial Representations for Goal-Directed Navigation Based on a Novel Model of Hippocampal Place Fields
Adedapo Alabi, Dieter Vanderelst, Ali Minai
Robot Navigation Navigation Task Self Organization Fast Learning Spatial Representation Spatial Cognition Neural Sequence Model Novel Model Place Cell

April 1, 2022

Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models
Felix Stahlberg, Ilia Kulikov, Shankar Kumar
High Uncertainty Anticipation Model Uncertainty Sequence to Sequence Model Many Natural Language Processing Moral Tractability Neural Sequence Model Dual Mode Intrinsic Uncertainty

March 14, 2022

Revisiting the Compositional Generalization Abilities of Neural Sequence Models
Arkil Patel, Satwik Bhattamishra, Phil Blunsom, Navin Goyal
Generalization Performance Compositional Generalization Seq2seq Model Neural Sequence Model Primitive Based Adversarial Training

January 30, 2022

Compositionality as Lexical Symmetry
Ekin Akyürek, Jacob Andreas
Inductive Bias Semantic Parsing Compositional Language Compositional Model Neural Sequence Model Compositional Data Augmentation

December 16, 2021

Learning and Analyzing Generation Order for Undirected Sequence Models
Yichen Jiang, Mohit Bansal
LeArning Abstract Sequence Model Generation Model Long Form Generation Neural Sequence Model

November 28, 2021

ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning
Bill Tuck Weng Pung, Alvin Chan
New Benchmark LSTM Network Long Sequence Relational Reasoning Systematic Generalization Neural Sequence Model Hierarchical Reasoning

November 4, 2021

How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction
Anthony Bau, Jacob Andreas
Neural Language Model Transformer Language Model Sequence Model \Sigma}{\Delta}$ Low Pas RNN Global Context Neural Sequence Model

Neural Sequence Model

Papers

Delay Embedding Theory of Neural Sequence Models

In-Context Language Learning: Architectures and Algorithms

N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding

Transformers can optimally learn regression mixture models

A Brain-Inspired Sequence Learning Model based on a Logic

Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection

Learning to Substitute Spans towards Improving Compositional Generalization

Grokking of Hierarchical Structure in Vanilla Transformers

Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection

What learning algorithm is in-context learning? Investigations with linear models

Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models

Rapid Learning of Spatial Representations for Goal-Directed Navigation Based on a Novel Model of Hippocampal Place Fields

Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models

Revisiting the Compositional Generalization Abilities of Neural Sequence Models

Compositionality as Lexical Symmetry

Learning and Analyzing Generation Order for Undirected Sequence Models

ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning

How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction