Transformer Language Model

Transformer language models are neural networks designed to process and generate human language, aiming to improve upon the capabilities of previous models by leveraging the attention mechanism for parallel processing of sequential data. Current research focuses on enhancing efficiency (e.g., through quantization and low-rank approximations), improving interpretability (e.g., by analyzing attention head behavior and internal representations), and addressing limitations in sample efficiency and compositional generalization. These advancements have significant implications for various NLP tasks, including question answering, text summarization, and machine translation, as well as for understanding the inner workings of these powerful models.

Papers

April 3, 2024

Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers
Sehyun Choi
Self Attention Transformer Language Model Linear Attention Transformer Inference Cross Architecture Generalization

March 30, 2024

Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning
Gustavo Bartz Guedes, Ana Estela Antunes da Silva
Contrastive Learning Classification Code Jina Embeddings Transformer Language Model Sentence Representation Long Document Scientific Paper

March 17, 2024

Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts
Daniel Enström, Viktor Kjellberg, Moa Johansson
Transformer Megatron Decepticons Complex Reasoning Average Approximation Spurious Correlation Transformer Language Model Reasoning Error Generative Transformer Generative Transformer Model BART Reasoning Shortcut

February 8, 2024

Limits of Transformer Language Models on Learning to Compose Algorithms
Jonathan Thomm, Aleksandar Terzic, Giacomo Camposampiero, Michael Hersche, Bernhard Schölkopf, Abbas Rahimi
LeArning Abstract Continuum Limit Transformer Language Model Compositional Learning Compositional Task

January 25, 2024

TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation
Gökçe Uludoğan, Zeynep Yirmibeşoğlu Balal, Furkan Akkurt, Melikşah Türker, Onur Güngör, Susan Üsküdarlı
Language Model Language Understanding Faithful Generation Multilingual Model Transformer Language Model Deep Understanding Monolingual Model Row Turning English Centric

December 26, 2023

Observable Propagation: Uncovering Feature Vectors in Transformers
Jacob Dunefsky, Arman Cohan
Large Language Model Transformer Megatron Decepticons Transformer Language Model Mechanistic Interpretability Feature Vector Language Detection

December 6, 2023

Compressed Context Memory For Online Language Model Interaction
Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song
Language Model Transformer Language Model Context Modeling Context Compression Language Model Inference

November 22, 2023

Efficient Transformer Knowledge Distillation: A Performance Review
Nathan Brown, Ashton Williamson, Tahj Anderson, Logan Lawrence
Knowledge Distillation Question Answering Transformer Language Model Performance Evaluation

November 15, 2023

Identifying Linear Relational Concepts in Large Language Models
David Chanin, Anthony Hunter, Oana-Maria Camburu
Latent Space Transformer Language Model Interpretable Concept Relational Concept

November 1, 2023

October 30, 2023

The Impact of Depth on Compositional Generalization in Transformer Language Models
Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, Tal Linzen
Language Model Global Impact Compositional Generalization Large Depth Transformer Language Model Deep Transformer Better Language Model

October 29, 2023

Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning
Language Model Transformer Language Model Self Attention Layer Recursive Algorithm Syntactic Generalization Pushdown Automaton

October 24, 2023

Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models
Lina Conti, Guillaume Wisniewski
Gender Bias Path Breaking Emergence Neural Model Neural Language Model Transformer Language Model Linguistic Property French Census

October 23, 2023

When Language Models Fall in Love: Animacy Processing in Transformer Language Models
Michael Hanna, Yonatan Belinkov, Sandro Pezzelle
Language Model Transformer Language Model Cognitive Process Sentient AI Parallel Personification Data

October 20, 2023

Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT
Soo Hyun Ryu
Integral Role GPT Neo Transformer Language Model Attention Head Semantic Knowledge Plausibility Processing

October 19, 2023

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization
Ningyu Xu, Qi Zhang, Jingting Ye, Menghan Zhang, Xuanjing Huang
Large Language Model Context Learning Transformer Language Model Cross Lingual Generalization

October 13, 2023

A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models
Takuma Udagawa, Aashka Trivedi, Michele Merler, Bishwaranjan Bhattacharjee
Large Language Model Knowledge Distillation Comparative Study Transformer Language Model Low Temperature Distillation Task Agnostic Distillation

October 12, 2023

Circuit Component Reuse Across Tasks in Transformer Language Models
Jack Merullo, Carsten Eickhoff, Ellie Pavlick
Language Model Transformer Language Model Mechanistic Interpretability Interpretable Algorithm Circuit Analysis Interpretability Scale

October 4, 2023

Scaling Laws for Associative Memories
Vivien Cabannes, Elvis Dohmatob, Alberto Bietti
High Dimensional Jina Embeddings Transformer Language Model Scaling Law Associative Memory Associative Memory Model

Transformer Language Model

Papers

Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers

Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning

Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts

Limits of Transformer Language Models on Learning to Compose Algorithms

TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation

Observable Propagation: Uncovering Feature Vectors in Transformers

Compressed Context Memory For Online Language Model Interaction

Efficient Transformer Knowledge Distillation: A Performance Review

Identifying Linear Relational Concepts in Large Language Models

Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation

Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?

The Impact of Depth on Compositional Generalization in Transformer Language Models

Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models

When Language Models Fall in Love: Animacy Processing in Transformer Language Models

Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models

Circuit Component Reuse Across Tasks in Transformer Language Models

Scaling Laws for Associative Memories