Multimodal Graph Learning

Follow this topic Ask about this topic

Multimodal Graph Learning studies how to use relationships between entities alongside their text, images, sensor readings, or other features. Common approaches use graph neural networks, attention, contrastive alignment, and learned graph structures, while hypergraphs model relationships among several entities or modalities at once. Vision-language models and large language models are also being adapted to graph tasks, including recommendation, biomedical prediction, and social-media analysis. A central challenge is balancing shared cross-modal information against useful modality-specific signals, especially when graph connections are noisy, modalities are missing, or message passing spreads errors.

Latest papers

110 papers12 papers in the last four weeks, up 20% on the four weeks before. 0.1% of all new papers.

EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval
Mar 26, 2026Yuhan Chen, Pengwen Dai, Chuan Wang +2Vision-Language Retrieval Vision-Language Alignment
Sun Yat-sen University, Guangzhou, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · Shenzhen Key Laboratory of Adversarial Artificial Intelligence, Shenzhen, China +2
Probabilistic Concept Graph Reasoning for Multimodal Misinformation Detection
Mar 26, 2026Ruichao Yang, Wei Gao, Xiaobin Zhu +5Graph Reasoning Multimodal Robustness
University of Science and Technology Beijing · Singapore Management University · Hong Kong Baptist University
Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition
Mar 22, 2026Ying Liu, Yuntao Shou, Wei Ai +2Differential Attention Emotion Recognition in Conversations
College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA
GraPHFormer: A Multimodal Graph Persistent Homology Transformer for the Analysis of Neuroscience Morphologies
Mar 21, 2026Uzair Shah, Marco Agus, Mahmoud Gamal +5Topological Representation Learning Multimodal Graph Learning
Hamad Bin Khalifa University, Qatar · University of Turin, Italy · Neuroscience Institute Cavalieri Ottolenghi, Italy +3
Hierarchical Multiscale Structure-Function Coupling for Brain Connectome Integration
Mar 21, 2026Jianwei Chen, Zhengyang Miao, Wenjie Cai +10Brain Network Analysis Neuroimaging
School of Medicine, University of Dundee, UK · College of Medicine and Biological Information Engineering, Northeastern University, China · School of Airspace Science and Engineering, Shandong University, China +5
Multimodal Connectome Fusion via Cross-Attention for Autism Spectrum Disorder Classification Using Graph Learning
Mar 16, 2026Ansar Rahman, Hassan Shojaee-Mend, Sepideh HatamikiaMultimodal Graph Learning Multimodal Fusion
Department of Medicine, Danube Private University (DPU), Krems, Austria · Department of Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria · Department of Medical Informatics, Faculty of Medicine, Gonabad University of Medical Sciences, Gonabad, Iran +1
BrainSCL: Subtype-Guided Contrastive Learning for Brain Disorder Diagnosis
Mar 11, 2026Xiaolong Li, Guiliang Guo, Guangqi Wen +6Brain Disorder Classification Contrastive Learning
Computer Science and Engineering, Northeastern University, Shenyang, China · School of Computer Science and Artificial Intelligence, Shandong Normal University, Jinan, China · AiShiWeiLai AI Research, Beijing, China +2
TopoOR: A Unified Topological Scene Representation for the Operating Room
Mar 10, 2026Tony Danjun Wang, Ka Young Kim, Tolga Birdal +2Higher-Order GNNs Topological Representation Learning
Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Germany · Kyung Hee University, South Korea +1
Multimodal Graph Representation Learning with Dynamic Information Pathways
Mar 10, 2026Xiaobin Hong, Mingkai Lin, Xiaoli Wang +2Multimodal Graph Learning Multimodal Embedding
State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd
GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
Mar 9, 2026Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2Multimodal Graph Learning Multimodal Model Evaluation
NYU Shanghai · New York University · East China Normal University +1
AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Mar 7, 2026Yunsheng Wang, Yuntao Shou, Yilong Tan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
Mario: Multimodal Graph Reasoning with Large Language Models
Mar 5, 2026Yuanfu Sun, Kang Li, Pengkang Guo +2LLM Reasoning with Graphs Multimodal Reasoning
New York University Shanghai · New York University · Tsinghua University +1
PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest
Mar 3, 2026Josh Beal, Eric Kim, Jinfeng Rao +3Multimodal Embedding Multimodal IR
Pinterest Inc., San Francisco, CA, USA
Graph Attention Based Prioritization of Disease Responsible Genes from Multimodal Alzheimer's Network
Mar 1, 2026Binon Teji, Subhajit Bandyopadhyay, Swarup RoyGenomics Multimodal Graph Learning
Network Reconstruction & Analysis Lab and Department of Computer Applications, Sikkim University, Sikkim, India, 737102 · Department of Computer Science & Engineering, Tezpur University, Assam, India, 784028
E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications
Feb 24, 2026Jiwoo Kang, Yeon-Chang LeeMultimodal KGs Multimodal Recommendation
UNIST Ulsan, Korea
Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendation
Feb 24, 2026Ji Dai, Quan Fang, DeSheng CaiSparse Mixture-of-Experts Recommender Systems
Beijing University of Posts and Telecommunications, China · Beijing University of Posts and Telecommunications, Beijing, China · Tianjin University of Technology, China
Distilling and Adapting: A Topology-Aware Framework for Zero-Shot Interaction Prediction in Multiplex Biological Networks
Feb 20, 2026Alana Deng, Sugitha Janarthanan, Yan Sun +2Link Prediction Graph Representation Learning
Department of Computer Science, Western University, London, ON, Canada · Department of Biochemistry, Western University, London, ON, Canada
Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection
Feb 18, 2026Rong Fu, Ziming Wang, Shuo Yin +3Multimodal Emotion Recognition Multimodal Sentiment Analysis
Independent Researcher
MRC-GAT: A Meta-Relational Copula-Based Graph Attention Network for Interpretable Multimodal Alzheimer's Disease Diagnosis
Feb 17, 2026Fatemeh Khalvandi, Saadat Izadi, Abdolah ChalechaleAlzheimer's Disease Classification Graph Attention Networks
Computer Engineering Department, Razi University, Kermanshah, Iran
EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation
Aug 22, 2025Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng +2Cross-Modal Alignment Multimodal Embedding
College of Computing and Data Science, Nanyang Technological University, Singapore

Mar 26, 2026J/K move · Enter open · S save

EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval

Yuhan Chen, Pengwen Dai, Chuan Wang, Dayan Wu+1

Sun Yat-sen University, Guangzhou, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · Shenzhen Key Laboratory of Adversarial Artificial Intelligence, Shenzhen, China+2

Vision-Language Retrieval Vision-Language Alignment Multimodal Contrastive Learning Multimodal Graph Learning

Open paper AskPDF arXiv

Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at https://github.com/draym28/EagleNet.