Multimodal Attention

Multimodal attention focuses on intelligently combining information from different data sources (e.g., text, images, audio) to improve the performance of machine learning models. Current research emphasizes developing sophisticated attention mechanisms, often within transformer-based architectures, to dynamically weigh the contribution of each modality and learn complex cross-modal relationships. This approach is proving highly effective across diverse applications, including improved accuracy in sentiment analysis, image fusion, and medical diagnosis, leading to more robust and informative models.

Papers

February 17, 2023

Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection
Yang Yang, Kaixiong Xu, Kaizheng Wang
Feature Enhancement Pedestrian Detection Multimodal Attention MultiSpectral Pedestrian Detection

February 8, 2023

Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
Mozhgan Pourkeshavarz, Shahabedin Nabavi, Mohsen Ebrahimi Moghaddam, Mehrnoush Shamsfard
Image Captioning Multimodal Attention Modal Feature Captioning Benchmark Image Caption Generation

October 28, 2022

DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
Fenglin Liu, Xian Wu, Shen Ge, Xuancheng Ren, Wei Fan, Xu Sun, Yuexian Zou
Vision Language Meaningful Representation Multi Modality Visual Concept Multimodal Attention Sentence Image Pair Multimodal BERT

September 20, 2022

An Efficient End-to-End Transformer with Progressive Tri-modal Attention for Multi-modal Emotion Recognition
Yang Wu, Pai Peng, Zhenyu Zhang, Yanyan Zhao, Bing Qin
End to End Multimodal Attention End to End Model Cross Modal Fusion

September 12, 2022

SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning
Wei Han, Hui Chen, Zhen Hai, Soujanya Poria, Lidong Bing
Contrastive Learning Multimodal Attention Helpfulness Score

August 25, 2022

Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis
Ming Jiang, Shaoxiong Ji
Multimodal Data Multimodal Sentiment Analysis Multimodal Attention Modality Data Modality Pair

June 17, 2022

Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis
Michal Golovanevsky, Carsten Eickhoff, Ritambhara Singh
Alzheimer'S Disease Cross Modal Attention Multimodal Attention Diagnostic Model Input Modality

March 29, 2022

MAP-Gen: An Automated 3D-Box Annotation Flow with Multimodal Attention Point Generator
Chang Liu, Xiaoyan Qian, Xiaojuan Qi, Edmund Y. Lam, Siew-Chong Tan, Ngai Wong
Multimodal Attention 3D Point 3D Label

March 9, 2022

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion
Ziqi Huang, Li Lin, Pujin Cheng, Linkai Peng, Xiaoying Tang
Tumor Segmentation Brain Tumor Segmentation Multimodal Attention Multi Modal Magnetic Resonance Modality Data Modal Attention

February 25, 2022

Prediction of Depression Severity Based on the Prosodic and Semantic Features with Bidirectional LSTM and Time Distributed CNN
Kaining Mao, Wei Zhang, Deborah Baofeng Wang, Ang Li, Rongqi Jiao, Yanhui Zhu, Bin Wu, Tiansheng Zheng, Lei Qian, Wei Lyu, Minjie Ye, Jie Chen
Human Prediction Prosodic Feature Depression Detection Semantic Feature Multimodal Attention Bidirectional LSTM Depression Severity

January 5, 2022

Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation
Yang Zhang, Yang Yang, Chenyun Xiong, Guodong Sun, Yanwen Guo
Semantic Segmentation Attention Based Encoder Decoder Model Multimodal Attention

December 14, 2021

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking
Yidi Li, Hong Liu, Hao Tang
Self Supervised Learning Multi Modal Fusion Multimodal Attention Spatial Coherence Multi Modal Tracking Speaker Tracking

December 6, 2021

November 17, 2021

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition
Hengshun Zhou, Jun Du, Yuanyuan Zhang, Qing Wang, Qing-Feng Liu, Chin-Hui Lee
Emotion Recognition Adaptive Importance Speech Emotion Recognition Attention Network Multimodal Emotion Recognition Information Fusion Multimodal Attention

Multimodal Attention

Papers

Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection

Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning

DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention

An Efficient End-to-End Transformer with Progressive Tri-modal Attention for Multi-modal Emotion Recognition

SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning

Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis

Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

MAP-Gen: An Automated 3D-Box Annotation Flow with Multimodal Attention Point Generator

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Prediction of Depression Severity Based on the Prosodic and Semantic Features with Bidirectional LSTM and Time Distributed CNN

Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition