Encoder Decoder Transformer Model

Encoder-decoder transformer models are neural network architectures designed for sequence-to-sequence tasks, aiming to improve efficiency and performance in various applications like machine translation, image captioning, and trajectory prediction. Current research focuses on optimizing these models through architectural innovations, such as exploring decoder-only designs and incorporating memory augmentation to handle long sequences, as well as improving training efficiency via techniques like dynamic early exit and novel training strategies. These advancements are significant because they address limitations in computational cost and memory bandwidth, leading to more efficient and effective models for a wide range of tasks across diverse fields.

Papers

October 27, 2024

CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming
Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K. Ahmed, Amir Yazdanbakhsh, Ali Jannesari
Document Boundary CUDA Implementation Encoder Decoder Transformer Model Program Translation

October 22, 2024

Assessment of Transformer-Based Encoder-Decoder Model for Human-Like Summarization
Sindhu Nair, Y.S. Rao, Radha Shankarmani
Structured Summary Text Summarization Abstractive Summarization Human Like Direct Assessment Abstractive Summary Human Written Summary Encoder Decoder Transformer Model

October 7, 2024

Efficient transformer with reinforced position embedding for language models
Yen-Che Hsiao, Abhishek Dutta
Language Model Efficient Transformer Product Specific Position Information Token Embeddings Encoder Decoder Transformer Model

June 4, 2024

Short-term Inland Vessel Trajectory Prediction with Encoder-Decoder Models
Kathrin Donandt, Karim Böttger, Dirk Söffker
Encoder Decoder Encoder Decoder Model Vessel Trajectory Encoder Decoder Transformer Model Encoder Decoder Transformer

May 27, 2024

Vision-and-Language Navigation Generative Pretrained Transformer
Wen Hanlin
Temporal Dependency Vision and Language Navigation Language Navigation Encoder Decoder Transformer Model

March 21, 2024

AI and Memory Wall
Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer
Artificial Intelligence Major Challenge Bottleneck Neural Scaling Law Encoder Decoder Transformer Model Decoder Model

February 21, 2024

Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping
Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, Yuandong Tian
Transformer Megatron Decepticons Transformer Model Good Better Symbolic Planning Search Behavior Encoder Decoder Transformer Model Top Quality Planning

December 28, 2023

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi
Human Language Vision Paper Audio Driven Action Feature Multimodal Instruction Encoder Decoder Transformer Model Large Scale Multimodal

November 21, 2023

Visual Analytics for Generative Transformer Models
Raymond Li, Ruixin Yang, Wen Xiao, Ahmed AbuRaed, Gabriel Murray, Giuseppe Carenini
Transformer Based Model Visual Analytics Encoder Decoder Transformer Model Transformer Based Generative Encoder Based Generative Transformer Model Visual Analytics Framework

November 15, 2023

DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models
Peng Tang, Pengkai Zhu, Tian Li, Srikar Appalaraju, Vijay Mahadevan, R. Manmatha
Early Exit MEG Decoder Encoder Decoder Transformer Model Shallow Decoder

October 26, 2023

DecoderTracker: Decoder-Only Method for Multiple-Object Tracking
Liao Pan, Yang Feng, Wu Di, Liu Bo, Zhang Xingle
Practical Method Multi Object Tracking Multiple Object Tracking Encoder Decoder Transformer Model Decoder Only Model Decoder Architecture

May 31, 2023

Examining the Emergence of Deductive Reasoning in Generative Language Models
Peter Belcak, Luca A. Lanzendörfer, Roger Wattenhofer
Generative Language Model Path Breaking Emergence Deductive Reasoning Encoder Decoder Transformer Model Logical Reasoning Ability OpenAI GPT Generative Transformer Model

January 7, 2023

TunesFormer: Forming Irish Tunes with Control Codes by Bar Patching
Shangda Wu, Xiaobing Li, Feng Yu, Maosong Sun
Code Data Encoder Decoder Transformer Model Basic Melody

January 2, 2023

Transformer Based Geocoding
Yuval Solaz, Vitaly Shalumov
Encoder Decoder Transformer Model Surname Geocoding Geographical Location Representation

November 23, 2022

Rank-One Editing of Encoder-Decoder Models
Vikas Raunak, Arul Menezes
Neural Machine Translation Sequence Model Encoder Decoder Model Model Retraining Encoder Decoder Transformer Model Rank One Model Editing

November 16, 2022

Streaming Joint Speech Recognition and Disfluency Detection
Hayato Futami, Emiru Tsunoo, Kentaro Shibata, Yosuke Kashiwagi, Takao Okuda, Siddhant Arora, Shinji Watanabe
Speech Recognition Transformer Decoder Encoder Decoder Transformer Model Disfluency Detection Joint Audio BERT Like

October 10, 2022

Denoising Masked AutoEncoders Help Robust Classification
Quanlin Wu, Hang Ye, Yuntian Gu, Huishuai Zhang, Liwei Wang, Di He
Pre Trained Model Pre Trained Masked Autoencoders Robust Classification Encoder Decoder Transformer Model

September 15, 2022

Stateful Memory-Augmented Transformers for Efficient Dialogue Modeling
Qingyang Wu, Zhou Yu
Generation Task Encoder Decoder Transformer Model Memory Based Transformer Memory Augmented Transformer Transformer Baseline

May 23, 2022

Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer
Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, Marta R. Costa-jussà
Transformer Based Inherent Interpretability Neural Machine Translation Black Box Source Table Token Prediction Multilingual Transformer Encoder Decoder Transformer Model Target Identification

March 19, 2022

Sequence-to-Sequence Knowledge Graph Completion and Question Answering
Apoorv Saxena, Adrian Kochsiek, Rainer Gemulla
Knowledge Graph Question Answering Knowledge Graph Completion Sequence to Sequence Task Encoder Decoder Transformer Model Knowledge Graph Link Prediction