Mario: Multimodal Graph Reasoning with Large Language Models

Yuanfu Sun; Kang Li; Pengkang Guo; Jiajin Liu; Qiaoyu Tan

cs.CVMar 5, 2026

Mario: Multimodal Graph Reasoning with Large Language Models

Authors: Yuanfu Sun, Kang Li, Pengkang Guo, Jiajin Liu, Qiaoyu Tan

Organizations: New York University Shanghai · New York University · Tsinghua University · EPFL

Abstract

Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.

Figures & tables

Figure 1: (a) Cosine similarity between text and image embeddings across three models on four datasets. (b) Venn diagram over three prompt templates with different modality inputs: Text-Only, Image-Only, and Text+Image. Each colored circle corresponds to one template. Among nodes correctly classified by at least one template, each region shows the proportion exclusively covered by the corresponding template combination (where overlapping regions blend the colors). Results are averaged over four datasets.

Figure 2: Overview of the proposed Mario framework. Given a MMG, Stage 1 uses a graph-conditioned vision–language model to perform structure-aware image–text alignment: images and texts are initially encoded, symmetrically refined by a transformer-embedded mixer that injects graph structure into token embeddings, and then aligned via contrastive learning. Stage 2 builds on these aligned features with modality-adaptive graph instruction tuning, where a lightweight router, trained under LLM supervision (a), infers each node’s modality preference and selects the most suitable modality-specific template for effective multimodal graph reasoning (b).

Methods	Node Classification Accuracy (%)				Link Prediction Accuracy (%)
Datasets	Movies	Reddit	CDs	Arts	Movies	Reddit	CDs	Arts
Text-only
GCN	$43.77$ ↑9.86	$84.29$ ↑11.01	$51.44$ ↑11.99	$76.92$ ↑15.21	$70.20$ ↑23.70	$74.23$ ↑17.07	$71.63$ ↑21.07	$66.27$ ↑23.69
GATv2	$48.71$ ↑4.92	$85.57$ ↑9.73	$54.67$ ↑8.76	$80.39$ ↑11.74	$72.63$ ↑21.27	$70.63$ ↑20.67	$73.27$ ↑19.43	$69.47$ ↑20.49
SAGE	$43.17$ ↑10.46	$85.64$ ↑9.66	$52.16$ ↑11.27	$85.26$ ↑6.87	$65.47$ ↑28.43	$71.53$ ↑19.77	$68.73$ ↑23.97	$65.50$ ↑24.46
LLaMA3-8B	$15.50$ ↑38.13	$77.23$ ↑18.07	$33.52$ ↑29.91	$72.42$ ↑19.71	$63.10$ ↑30.80	$79.30$ ↑12.00	$74.30$ ↑18.40	$66.40$ ↑23.56

Table 1: Single-Focus performance comparison on four MMG datasets. Red

\uparrow

number denotes the absolute accuracy gain over each baseline. Since LLaVA 1.5 does not support multiple image inputs, we concatenate images of node pairs and their neighbors into a single canvas before feeding the model to ensure a fair comparison. We fine-tune Qwen2.5-VL by feeding it the anchor node and its neighbors’ texts+images. The best baseline in each modality setting is highlighted in bold, and Mario (ours) is shown in the last row with underline.

Methods	Node Classification Accuracy (%)				Link Prediction Accuracy (%)
Datasets	Movies	Reddit	CDs	Arts	Movies	Reddit	CDs	Arts
Text-only
GCN	$47.15$ ↑3.83	$86.17$ ↑7.03	$50.76$ ↑9.17	$79.03$ ↑12.17	$70.30$ ↑22.15	$74.50$ ↑16.23	$72.93$ ↑20.57	$62.33$ ↑30.27
SAGE	$46.85$ ↑4.13	$89.96$ ↑3.24	$\mathbf{53.24}$ ↑6.69	$\mathbf{87.46}$ ↑3.74	$63.30$ ↑29.15	$72.03$ ↑18.70	$62.03$ ↑31.47	$63.70$ ↑28.90
LLaGA	$\mathbf{47.80}$ ↑3.18	$\mathbf{91.14}$ ↑2.06	$51.33$ ↑8.60	$74.02$ ↑17.18	$\mathbf{87.28}$ ↑5.17	$\mathbf{88.95}$ ↑1.78	$\mathbf{90.32}$ ↑3.18	$87.06$ ↑5.54
GraphPrompter	$45.21$ ↑5.77	$90.36$ ↑2.84	$44.70$ ↑15.23	$83.91$ ↑7.29	$70.27$ ↑22.18	$86.70$ ↑4.03	$83.20$ ↑10.30	$\mathbf{88.30}$ ↑4.30

Table 2: Mix-Training performance comparison on four multimodal graph datasets. The table design is consistent with Table 1 .

	Toys $\rightarrow$ Movies		Toys+Movies $\rightarrow$ CDs		Toys $\rightarrow$ Books
Model Tasks	NC	LP	NC	LP	NC	LP
GCN	$5.29$	$62.23$	$5.64$	$55.70$	$11.47$	$45.43$
GATv2	$6.58$	$62.03$	$6.58$	$63.00$	$14.50$	$49.00$
SAGE	$3.61$	$61.80$	$5.87$	$66.13$	$7.73$	$48.40$
GraphPrompter	$11.15$	$72.70$	$36.66$	$51.00$	$23.00$	$62.50$
LLaGA	$8.63$	$62.00$	$9.69$	$72.85$	$8.85$	$52.50$

Table 3: Zero-Shot Results (Accuracy).

Panel of Figure 3 — Figure 3: Training curves of Mario vs. the text-only template (Fixed) on two datasets, with early-stopping epochs in the end.

Model	Arts		Reddit		Movies
	Acc(%)	s/epoch	Acc(%)	s/epoch	Acc(%)	s/epoch
GCN	$90.32$ ↑2.00	$114$	$93.30$ ↑2.14	$85$	$50.90$ ↑5.36	$78$
SAGE	$90.75$ ↑1.52	$117$	$92.10$ ↑3.47	$89$	$51.10$ ↑4.95	$81$
GATv2	$90.03$ ↑2.33	$129$	$92.90$ ↑2.58	$93$	$51.30$ ↑4.54	$85$
MLP	$89.95$ ↑2.42	$109$	$92.70$ ↑2.80	$79$	$50.70$ ↑5.78	$75$
GVLM	$92.13$	$174$	$95.30$	$135$	$53.63$	$122$

Table 4: Ablation Study of Stage 1’s Model.

Panel of Figure 4 — Figure 4: Comparison of Mario with three fixed prompt templates containing different modality information across the four datasets.

Panel of Figure 5 — Figure 5: Visualization of Router Selections across two MMGs.

Dataset	Domain	# Nodes	# Edges	# Classes
Movies	E-commerce	16,672	109,195	20
Toys	E-commerce	20,694	63,443	18
CDs	E-commerce	36,272	844,878	15
Arts	E-commerce	28,195	197,428	7
Reddit(S)	Social Media	15,894	566,160	20
Goodreads	Literature	685,294	7,235,084	11

Table 5: Dataset statistics across multiple MMG datasets.

Panel of Figure 6 — Figure 6: t-SNE visualizations of aligned multimodal features on Movies (top) and Reddit (bottom) for the three models in Fig. 1. For each dataset, we project a randomly sampled subset of nodes from the full graph, using their aligned text and image representations as input to t-SNE. Comparing the six panels reveals how different alignment strategies affect the relative organization of text and image features in the shared latent space.

Model	Movies	Arts
SAGE	44.07	85.35
GATv2	49.29	81.19
GCN	46.96	76.76
MMGCN	46.79	86.63
MGAT	40.17	87.25

Table 6: Node classification accuracy (%) on Movies and Arts for additional multimodal baselines (MMGCN, MGAT) and representative unimodal GNNs (text+image settings).

Model	Trainable Params	Movies	Reddit	CDs	Arts
Node Classification (Trainable parameters are from Stage 2)
Frozen Mario	18,886,656 (0.2346%)	50.85	93.60	60.45	89.69
Mario + LoRA	22,294,528 (0.2768%)	53.63	95.30	63.43	92.13
Link Prediction (Trainable parameters are from Stage 2)
Frozen Mario	18,886,656 (0.2346%)	90.90	89.00	88.60	86.30
Mario + LoRA	22,294,528 (0.2768%)	93.90	91.30	92.70	89.96

Table 7: Frozen Mario versus LoRA-Tuned Mario (Accuracy %).

Model	Toys $\rightarrow$ Movies		Toys+Movies $\rightarrow$ CDs
Model	NC	LP	NC	LP
MLP	6.12	52.60	7.04	50.20
GCN	3.29	62.13	10.01	64.17
GATv2	4.32	64.47	8.13	67.97
SAGE	3.11	55.83	6.14	59.63
MLP(Vision Only)	4.61	52.13	9.06	46.09

Table 8: Zero-Shot Results (Accuracy %).

Figure 7: Sensitivity analysis of the projection layer in Arts and Movies

Figure 8: Sensitivity analysis of the number of neighbors per hop in Arts and Toys

Different Size	Arts	Toys
Mario-7B (LLaMA2)	91.06	81.20
Mario-13B (LLaMA2)	91.23	80.93
Mario-7B (Vicuna-v1.5)	91.09	81.07
Mario (FLAN-T5-XXL)	92.08	81.63
Mario-8B (LLaMA3)	92.13	82.58

Table 9: Ablation Study of Different LLMs. (Accuracy %)

Dataset	Movies	Toys	Arts	CDs	Goodreads	Reddit
Heterophily Ratio	0.53	0.26	0.34	0.69	0.33	0.04

Table 10: Heterophily Ratios of Benchmark Datasets

Method	Movies (NC)	Arts (LP)
Mario (Single Focus)	$53.63\pm 0.07$	$89.96\pm 0.14$
Mario (Mix Training)	$50.98\pm 0.08$	$92.60\pm 0.12$

Table 11: Partial variance results of Mario across datasets and tasks over 3 runs.

Method	Arts	Reddit	Movies	#Epoch	Avg Acc(%)
Tuned CLIP/Other Baselines vs. GVLM (stage1) (Columns 2–4: GPU-hours)
GCN	0.95	0.94	0.88	36.9	78.17 ↑2.79
SAGE	0.91	0.90	0.88	34.3	77.98 ↑3.04
GATv2	0.93	0.91	0.89	33.0	78.08 ↑2.92
MLP	0.90	0.87	0.85	37.0	77.78 ↑3.30
Tuned CLIP	0.99	0.98	0.99	22.3	78.01 ↑3.00

Table 12: Detailed Cost Breakdown (Stage 1 & 2).

Task	Prompt Template
Node Classification	I’m starting a node classification task in the <dataset> . Each node represents a <product> with text and image features, and edges indicate <relationship> . Given a target node, the raw text is …, the text feature is <text feature> and the image feature is <image feature> . The neighbors are described in the following template: <text feature> , <image feature> , and <label> . It has the following neighbors at hop 1: N1: <1-hop neighbor 1 text feature> , <1-hop neighbor 1 image feature> , <1-hop neighbor 1 label> N2: <1-hop neighbor 2 text feature> , <1-hop neighbor 2 image feature> , <1-hop neighbor 2 label> N3: ……….. It has the following neighbors at hop 2: N1: <2-hop neighbor 1 text feature> , <2-hop neighbor 1 image feature> , <2-hop neighbor 1 label> N2: <2-hop neighbor 2 text feature> , <2-hop neighbor 2 image feature> , <2-hop neighbor 2 label> …… Based on the information provided, please classify the target node into one of the following categories: {all_categories} .
Link Prediction	I’m starting a link prediction task in the <dataset> . Each node represents a <product> with text and image features, and edges indicate <relationship> . Given the two nodes: Node 1: The raw text is … the text feature is <text feature> , and the image feature is <image feature> . Node 2: The raw text is … the text feature is <text feature> , and the image feature is <image feature> . The neighbors of node 1 (common neighbors with node 2) are described in the following template: <text feature> , <image feature> . It has the following neighbors at hop 1 (Directly connected): N1: <1-hop neighbor 1 text feature> , <1-hop neighbor 1 image feature> N2: <1-hop neighbor 2 text feature> , <1-hop neighbor 2 image feature> N3: ……….. It has the following neighbors at hop 2 (Indirectly connected by shared neighbors): N1: <2-hop neighbor 1 text feature> , <2-hop neighbor 1 image feature> N2: <2-hop neighbor 2 text feature> , <2-hop neighbor 2 image feature> …… Based on the information provided, please determine whether a link exists between the two nodes. Answer ”yes” if a link exists or ”no” if it does not.

Table 13: Prompt Templates for Node Classification and Link Prediction Tasks. Note that this template is designed to include both text and image features of the node. If the input is text-only or image-only, simply retain the corresponding single modality feature.

Panel of Figure 9 — Figure 9: A case from the Movies dataset in the NC task that Mario identifies as preferring Text+Image modality information.

Figure 10: Anchor node’s image in Figure 9 .

Panel of Figure 11 — Figure 11: A case from the Movies dataset in the NC task that Mario identifies as preferring Text-only modality information.

Figure 12: Anchor node’s image in Figure 11

Panel of Figure 13 — Figure 13: A case from the Movies dataset in the NC task that Mario identifies as preferring Image-only modality information.

Figure 14: Anchor node’s image in Figure 13

Panel of Figure 15 — Figure 15: A case from the Toys dataset in the LP task that Mario identifies as preferring Text+Image modality information.

Panel of Figure 16 — Figure 16: Node Pair’s images in Figure 15 .

Panel of Figure 17 — Figure 17: A case from the CDs dataset in the LP task that Mario identifies as preferring Text-only modality information.

Panel of Figure 18 — Figure 18: Node Pair’s images in Figure 17 .

Panel of Figure 19 — Figure 19: A case from the CDs dataset in the LP task that Mario identifies as preferring Image-only modality information.

Panel of Figure 20 — Figure 20: Node Pair’s images in Figure 19

Explore similar work

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
Mar 9, 2026Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2Multimodal Graph Learning Multimodal Model Evaluation
NYU Shanghai · New York University · East China Normal University +1
Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
Jul 7, 2026Zhiwei Yang, Yuanchen Wu, Nan Zhang +3Multimodal Large Language Models Multimodal CoT Reasoning
Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University

Mar 9, 2026J/K move · Enter open · S save

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

Jiajin Liu, Dongzhe Fan, Chuanhao Ji, Daochen Zha+1

NYU Shanghai · New York University · East China Normal University+1

Multimodal Graph Learning Multimodal Model Evaluation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.

Abstract

Explore similar work

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

MG2^22-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation