Attention Based Encoder

Attention-based encoders are neural network components designed to process sequential data by weighting the importance of different input elements, enabling the model to focus on relevant information. Current research emphasizes improving their efficiency and applicability across diverse domains, including natural language processing, speech recognition, and image processing, with explorations into hybrid architectures combining attention mechanisms with other techniques like recurrent neural networks or Fourier transforms to enhance performance and reduce computational costs. These advancements are driving progress in areas such as code generation from natural language, efficient edge computing for speech understanding, and improved performance in various machine learning tasks, particularly those involving long sequences or complex data.

Papers

January 25, 2024

Improving Natural Language Capability of Code Large Language Model
Wei Li, Daoguang Zan, Bei Guan, Ailun Yu, Xiaolin Chen, Yongji Wang
Large Language Model Natural Language Code Generation Code LLM Attention Based Encoder

November 22, 2023

Efficient Deep Speech Understanding at the Edge
Rongxiang Wang, Felix Xiaozhu Lin
Automatic Speech Recognition Extreme Edge Spoken Language Understanding Speech Model Connectionist Temporal Classification Inference Framework Attention Based Encoder

November 9, 2023

Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform
Daniele Giofré, Sneha Ghantasala
Language Model Transformer Architecture Abstractive Text Summarization Attention Based Encoder Context Token

October 5, 2023

Logical Languages Accepted by Transformer Encoders with Hard Attention
Pablo Barcelo, Alexander Kozachinskiy, Anthony Widjaja Lin, Vladimir Podolskii
Transformer Encoders Pay Attention Circuit Complexity Attention Based Encoder

August 9, 2023

Scalability of Message Encoding Techniques for Continuous Communication Learned with Multi-Agent Reinforcement Learning
Astrid Vanneste, Thomas Somers, Simon Vanneste, Kevin Mets, Tom De Schepper, Siegfried Mercelis, Peter Hellinckx
Multi Agent Reinforcement Learning Potential Scalability Multiagent System Agent Communication Encoding Scheme Attention Based Encoder

May 25, 2023

UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation
Zhiming Mao, Huimin Wang, Yiming Du, Kam-fai Wong
Language Model Text to Text Joint Contrastive Learning Attention Based Encoder History Representation Text Aware

October 10, 2022

SCAM! Transferring humans between images with Semantic Cross Attention Modulation
Nicolas Dufour, David Picard, Vicky Kalogeiton
Image Generation Real Human Digital Image Cross Attention Mechanism Semantic Region Pose Transfer Attention Based Encoder

June 5, 2022

MPANet: Multi-Patch Attention For Infrared Small Target object Detection
Ao Wang, Wei Li, Xin Wu, Zhanchao Huang, Ran Tao
Convolutional Neural Network Data Detection Small Target Detection Infrared Small Patch Attention Attention Based Encoder

May 13, 2022

ImageSig: A signature transform for ultra-lightweight image recognition
Mohamed R. Ibrahim, Terry Lyons
Image Recognition RGB Image Convolutional Structure Signature Transform Attention Based Encoder

January 26, 2022

Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR
Yufei Liu, Rao Ma, Haihua Xu, Yi He, Zejun Ma, Weibin Zhang
Speech Recognition ASR Model Context Vector Attention Based Encoder Internal Language Model

December 22, 2021

A Deep Reinforcement Learning Approach for Solving the Traveling Salesman Problem with Drone
Aigerim Bogyrbayeva, Taehyun Yoon, Hanbum Ko, Sungbin Lim, Hyokun Yun, Changhyun Kwon
Reinforcement Learning Single Drone Attention Based Attention Based Model Deep Reinforcement Learning Approach Traveling Salesman Problem Attention Based Encoder Decoder Attention Based Encoder