NaviGATE: Noise-Aware Dual-View Generation for Multimodal Knowledge Graph Reasoning
Organizations: College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China
Abstract
Multimodal knowledge graph reasoning (MKGR) aims to infer missing facts by jointly exploiting relational structures, textual descriptions, and visual evidence. Existing embedding-based methods effectively capture graph topology, but their fixed multimodal representations are less effective for query-specific fine-grained semantic reasoning, while generation-based methods are vulnerable to noisy retrieval and may produce invalid entities. To address these challenges, we propose NaviGATE, a noise-aware generative framework for MKGR. NaviGATE first aligns graph-derived structural representations with textual semantics, then performs structure-aware internal knowledge activation by fine-tuning the multimodal language model with multimodal entity summaries and injected structural tokens. In the noise-aware dual-view instruction tuning stage, NaviGATE jointly optimizes retrieval-free and retrieval-aware views: reliable retrieved candidates are reinforced through contrastive logit sharpening, while misleading candidates are suppressed through parametric consistency distillation. In inference, structure-constrained decoding restricts generation to valid entities in the target knowledge graph. Experiments on standard benchmarks show that NaviGATE consistently outperforms strong unimodal, multimodal, and generation-based baselines, and remains robust under noisy retrieval scenarios.
Figures & tables
| Models | MKG-W | DB15K | ||||||
|---|---|---|---|---|---|---|---|---|
| MRR | Hits@1 | Hits@3 | Hits@10 | MRR | Hits@1 | Hits@3 | Hits@10 | |
| Unimodal KGR Methods | ||||||||
| TransE ( Bordes et al. 2013a ) | 0.292 | 0.211 | 0.332 | 0.442 | 0.249 | 0.128 | 0.315 | 0.471 |
| DistMult ( Yang et al. 2015 ) | 0.210 | 0.159 | 0.223 | 0.309 | 0.230 | 0.148 | 0.263 | 0.396 |
| ComplEx ( Trouillon et al. 2016 ) | 0.249 | 0.191 | 0.267 | 0.367 | 0.275 | 0.184 | 0.316 | 0.454 |
| RotatE ( Sun et al. 2019 ) | 0.334 | 0.270 | 0.362 | 0.451 | 0.291 | 0.184 | 0.355 | 0.479 |
| Variants | MRR | H@1 | H@3 | H@10 |
|---|---|---|---|---|
| Stage-1 Structural Design | ||||
| w/o stage1 pretrain | 0.401 | 0.357 | 0.421 | 0.478 |
| w/o structural contrastive loss | 0.405 | 0.357 | 0.430 | 0.486 |
| Stage-2 Reranking Design | ||||
| w/o stage2 finetune | 0.338 | 0.256 | 0.390 | 0.468 |
| w/o multimodal text summary | 0.387 | 0.334 | 0.411 | 0.478 |
Multimodal KGs
Multimodal knowledge graphs connect entities and relations with text, images, video, time, and other evidence for structured reasoning.
See all 46 papers