Parallel Transformer

Parallel Transformers represent a significant advancement in transformer architecture, aiming to overcome the computational limitations of autoregressive models by enabling parallel processing of input sequences. Current research focuses on improving efficiency through novel parallel architectures like Kraken, which optimizes multi-device inference, and on enhancing model fusion techniques using optimal transport to combine the strengths of multiple independently trained transformers. These advancements are impacting various fields, including natural language processing, computer vision, and speech recognition, by accelerating inference speeds and improving performance on tasks ranging from language modeling to audiovisual scene classification.

Papers

August 14, 2024

Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference
Rohan Baskar Prabhakar, Hengrui Zhang, David Wentzlaff
Transformer Based Model Device Inference NeMo Team Tensor Parallelism Parallel Transformer Layer Model Parallelism

March 13, 2024

Masked Generative Story Transformer with Character Guidance and Caption Augmentation
Christos Papadimitriou, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
Text Augmentation Story Visualization Masked Generative Parallel Transformer Guidance Method Generative AI Challenge

March 1, 2024

Merging Text Transformer Models from Different Initializations
Neha Verma, Maha Elbayad
Transformer Architecture Transformer Model Model Merging Masked Language Modeling Small Initialization Parallel Transformer

December 28, 2023

A New Perspective on Speaker Verification: Joint Modeling with DFSMN and Transformer
Hongyu Wang, Hui Li, Bo Li
Attention Mechanism Speaker Verification Memory Trace Voter Participation Speech Transformer Utterance Representation Parallel Transformer

October 9, 2023

Transformer Fusion with Optimal Transport
Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, Sidak Pal Singh
Optimal Transport Fusion Transformer Transformer Based Network Model Fusion Parallel Transformer

January 1, 2023

Relevance Classification of Flood-related Twitter Posts via Multiple Transformers
Wisal Mukhtiar, Waliiya Rizwan, Aneela Habib, Yasir Saleem Afridi, Laiq Hasan, Kashif Ahmad
Twitter Resource Noisy Data Natural Disaster Parallel Transformer

November 16, 2022

GAMMT: Generative Ambiguity Modeling Using Multiple Transformers
Xingcheng Xu
Sequential Data Multiple Representation Parallel Transformer Imprecise Probability

October 27, 2022

ViT-CAT: Parallel Vision Transformers with Cross Attention Fusion for Popularity Prediction in MEC Networks
Zohreh HajiAkhondi-Meybodi, Arash Mohammadi, Ming Hou, Jamshid Abouei, Konstantinos N. Plataniotis
Pre Trained ViT Popularity Prediction Edge Caching Cross Attention Fusion Parallel Transformer ViT LoS V2X ViT Architecture

October 18, 2022

Multi-Source Transformer Architectures for Audiovisual Scene Classification
Wim Boes, Hugo Van hamme
Visual Feature Audio Visual Scene Parallel Transformer Binary Cross Entropy

September 28, 2022

MUTE-Reco: MUTual Information Assisted Ensemble Feature RECOmmender System for Healthcare Prognosis
Abhishek Dey, Debayan Goswami, Rahul Roy, Susmita Ghosh, Yu Shrike Zhang, Jonathan H. Chan
Recommender System Feature Selection Recommendation Module Parallel Transformer Personalized Treatment Recommendation

September 15, 2022

Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision
Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
Computer Vision Transformer Architecture Numerical Data Attention Head Transformer Encoders Generalization Capability Parallel Transformer

June 16, 2022

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
Zhifu Gao, Shiliang Zhang, Ian McLoughlin, Zhijie Yan
Non Autoregressive Autoregressive Transformer Parallel Transformer Non Autoregressive End to End

June 8, 2022

Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction
Mateusz Jurewicz, Leon Derczynski
Neural Network Sequence to Sequence Sequence to Sequence Model Sequence Length Structure Prediction Parallel Transformer Permutation Training

Parallel Transformer

Papers

Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference

Masked Generative Story Transformer with Character Guidance and Caption Augmentation

Merging Text Transformer Models from Different Initializations

A New Perspective on Speaker Verification: Joint Modeling with DFSMN and Transformer

Transformer Fusion with Optimal Transport

Relevance Classification of Flood-related Twitter Posts via Multiple Transformers

GAMMT: Generative Ambiguity Modeling Using Multiple Transformers

ViT-CAT: Parallel Vision Transformers with Cross Attention Fusion for Popularity Prediction in MEC Networks

Multi-Source Transformer Architectures for Audiovisual Scene Classification

MUTE-Reco: MUTual Information Assisted Ensemble Feature RECOmmender System for Healthcare Prognosis

Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction