GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

Jiajin Liu; Dongzhe Fan; Chuanhao Ji; Daochen Zha; Qiaoyu Tan

cs.CVMar 9, 2026

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

Authors: Jiajin Liu, Dongzhe Fan, Chuanhao Ji, Daochen Zha, Qiaoyu Tan

Organizations: NYU Shanghai · New York University · East China Normal University · Rice University

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.

Figures & tables

Figure 1: Overview of the GraphVLM benchmark with a timeline of graph learning research. Existing graph learning methods are categorized into three groups based on the prediction backbone. The bottom-left corner illustrates the functional roles VLMs play in each category.

Benchmarks	Backbones			VLM Roles
	GNN	LLM	VLM	Encoder		Aligner		Predictor
	GNN	LLM	VLM	Pre-trained	Fine-tuned	Prompt	Latent	Zero-shot	SFT
MM-Bench [ 42 ]	✓	✗	✗	✓	✗	✗	✗	✗	✗
MAGB [ 37 ]	✓	✗	✓	✓	✗	✗	✗	✓	✗
GraphVLM	✓	✓	✓	✓	✓	✓	✓	✓	✓

Table 1: Comparison of existing multimodal graph benchmarks and their coverage of MMGL method backbones and VLM roles.

Model	Encoder	Movies	Toys	Grocery	Arts	CDs	Reddit
MLP	Imagebind	40.14 $\pm$ 0.14	52.30 $\pm$ 0.45	65.17 $\pm$ 0.17	74.79 $\pm$ 0.37	30.76 $\pm$ 0.90	42.62 $\pm$ 0.27
	CLIP	45.12 $\pm$ 0.12	74.59 $\pm$ 0.09	84.67 $\pm$ 0.04	84.28 $\pm$ 0.17	52.22 $\pm$ 0.05	76.70 $\pm$ 0.25
	CLIP-F	47.46 $\pm$ 0.08	73.22 $\pm$ 0.09	82.79 $\pm$ 0.05	82.66 $\pm$ 0.07	48.74 $\pm$ 0.07	79.77 $\pm$ 0.10
	CLIP-F-S	39.73 $\pm$ 0.12	59.88 $\pm$ 0.25	76.94 $\pm$ 0.03	84.89 $\pm$ 0.06	49.89 $\pm$ 0.07	79.71 $\pm$ 0.18
GCN	Imagebind	45.64 $\pm$ 0.56	71.02 $\pm$ 0.13	79.01 $\pm$ 0.07	75.01 $\pm$ 0.16	47.42 $\pm$ 0.43	70.89 $\pm$ 0.20
	CLIP	46.97 $\pm$ 0.24	74.36 $\pm$ 0.07	79.59 $\pm$ 0.29	76.76 $\pm$ 0.21	52.68 $\pm$ 0.03	66.06 $\pm$ 0.11

Table 2: Node classification accuracy (%) comparison under different enhanced encoder settings. Bold values denote the highest performance within the same model and dataset setup. Blue cells indicate the highest result within each dataset.

Figure 2: Average accuracy (%) across six datasets under different modalities for GNN-based methods, using CLIP as the encoder.

Model	Input	Movies	Toys	Grocery	Arts	CDs
GraphPrompter	[regular]file	46.36	74.08	85.00	84.41	46.27
	[regular]file ✧	45.55	76.56	85.47	64.87	52.06
	[regular]file ✧ s	46.12	75.07	80.57	56.61	52.58
	[regular]file ✦	45.88	70.96	84.91	85.85	45.73
LLaGA	[regular]file	49.57	77.51	86.25	89.32	53.12
	[regular]file ✧	48.73	74.97	86.16	76.34	54.45

Table 3: Node classification accuracy (%) of GraphLLM-based methods under different input and alignment strategies. [regular]file indicates the text modality, while represents the vision modality. ✧ indicates prompt-level alignment, ✧ s represents structure-aware prompt-level alignment, and ✦ denotes latent-space alignment. Bold denotes the highest performance within the same model and dataset setup. Blue cells indicate the highest result within each dataset.

VLM	SFT	Fusion	Movies	Toys	Grocery	Arts	CDs	Reddit
LLaVA-1.5-7B	✗		13.19	47.52	57.88	72.96	41.30	34.82
	✓		45.58	74.32	85.89	84.47	53.55	82.45
	✓	[regular]file ✧	44.65	74.82	85.91	82.94	53.73	82.76
Qwen2.5-VL-7B	✗		14.96	58.27	68.79	69.34	34.94	34.33
	✓		54.21	82.56	88.32	93.37	54.86	67.48
	✓	[regular]file ✧	53.10	80.38	88.53	92.43	54.13	67.04

Table 4: Node classification accuracy (%) comparison under different VLM models, structure data modality, and structural fusion settings. Models are evaluated with and without SFT. ✧ indicates prompt-level structural fusion, and ✦ denotes latent-space structural fusion. Bold values mark the best performance within each model setup, while Blue cells highlight the overall best results across all baseline settings for each dataset. Underline indicates the higher performance of fusion approaches under the same neighbor modality for each dataset in Qwen-VL-7B.

Figure 3: The impact of different VLM-based methods with multimodal structure information at the prompt level for node classification tasks, using Qwen-VL-7B as the backbone.

Figure 4: Performance comparison of three VLM roles.

Figure 5: Extra performance gain of structure-awareness.

Dataset	#Nodes	#Edges	Avg.#Degree	#Classes	Domain
Movies	16,672	218,390	26	19	E-commerce
Toys	20,695	126,886	12	18	E-commerce
Grocery	84,379	693,154	16	20	E-commerce
Arts	28,195	197,428	14	7	E-commerce
CDs	36,272	844,878	47	15	E-commerce
Reddit	99,638	1,167,188	23	50	Social Network

Table 5: Dataset statistics.

Learning Rate	#Neighbors	Epoch	Batch Size	Temperature	Optimizer
$1\times 10^{-5}$	5	1	16	0.5	Adam

Table 6: Training hyperparameters for the CLIP-F-S model.

Model	Settings	Movies	CDs	Grocery	Arts
MLP	Text+Image (CLIP)	4.90	6.85	9.17	14.14
GCN	Text+Image (CLIP)	7.69	10.05	22.02	12.55
GraphSAGE	Text+Image (CLIP)	3.73	6.20	5.56	20.69
MLP	Text+Image (CLIP-F-S)	5.50	8.79	14.12	18.14
GCN	Text+Image (CLIP-F-S)	3.16	7.61	8.66	18.12
GraphSAGE	Text+Image (CLIP-F-S)	1.20	9.85	11.40	18.14

Table 7: Transfer ability comparison: Node classification accuracy (%) when trained on Toys and evaluated on other datasets.

Stage	GNN-based				LLM-based		VLM-based
Stage	GCN	SAGE	MMGCN	MGAT	LLaGA	GraphGPT	Qwen-VL	Qwen2.5-VL
Training	~2min	~2min	~3min	~3min	~17min	~60min	~5h	~5h
Inference	~10s	~10s	~15s	~20s	~10min	~30min	~20min	~20min

Table 8: Efficiency comparison on the Movies dataset.

Movies:	<image input> Given an image of a movie from the Amazon movies dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the product as described in user reviews, without losing critical details or introducing unnecessary information.
Toys:	<image input> Given an image of a toy from the Amazon toys dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the product as described in user reviews, without losing critical details or introducing unnecessary information.
Grocery:	<image input> Given an image of a grocery from the Amazon grocery dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the product as described in user reviews, without losing critical details or introducing unnecessary information.
CDs:	<image input> Given an image of a CD from the Amazon CD dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the product as described in user reviews, without losing critical details or introducing unnecessary information.
Arts:	<image input> Given an image of an artwork from the Amazon Art dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the product as described in user reviews, without losing critical details or introducing unnecessary information.
Reddit:	<image input> Given an image of a post from the Reddit dataset , generate a concise and detailed summary. Focus on describing key visual concepts. Ensure the summary is informative and useful for understanding the post as described in the caption, without losing critical details or introducing unnecessary information.

Table 9: Prompts used to generate a text description of the image by VLM

Movies:	Given the text information of a product from the Amazon Movies dataset: <text information> . Image summary: <image summary> Questions: Using the title, description, and image summary of the product provided above, create an informative and concise description that effectively highlights the product’s key features.
Toys:	Given the text information of a product from the Amazon toys dataset: <text information> . Image summary: <image summary> Questions: Using the title, description, and image summary of the product provided above, create an informative and concise description that effectively highlights the product’s key features.
Grocery:	Given the text information of a product from the Amazon grocery dataset: <text information> . Image summary: <image summary> Questions: Using the title, description, and image summary of the product provided above, create an informative and concise description that effectively highlights the product’s key features.
CDs:	Given the text information of a product from the Amazon CD dataset: <text information> . Image summary: <image summary> Questions: Using the title, description, and image summary of the product provided above, create an informative and concise description that effectively highlights the product’s key features.
Arts:	Given the text information of a product from the Amazon Art dataset: <text information> . Image summary: <image summary> Questions: Using the title, description, and image summary of the product provided above, create an informative and concise description that effectively highlights the product’s key features.
Reddit:	Given the text information of a post from the Reddit dataset: <text information> . Image summary: <image summary> Questions: Using the caption and image summary of the post provided above, create an informative and concise description that effectively highlights the post’s key features.

Table 10: Prompts for the non-structure-aware aligner cases.

Movies:	Given the text information of a product from the Amazon movies dataset: <text information> . Image summary: <image summary> . Also given the information of co-purchased or co-reviewed products: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-purchased or co-reviewed product information is available.’) Questions: Using the product’s title, description, and image summary provided above, along with any co-purchase or co-review data, generate a concise yet informative description of the product.
Toys:	Given the text information of a product from the Amazon toys dataset: <text information> . Image summary: <image summary> . Also given the information of co-purchased or co-reviewed products: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-purchased or co-reviewed product information is available.’) Questions: Using the product’s title, description, and image summary provided above, along with any co-purchase or co-review data, generate a concise yet informative description of the product.
Grocery:	Given the text information of a product from the Amazon grocery dataset: <text information> . Image summary: <image summary> . Also given the information of co-purchased or co-reviewed products: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-purchased or co-reviewed product information is available.’) Questions: Using the product’s title, description, and image summary provided above, along with any co-purchase or co-review data, generate a concise yet informative description of the product.
CDs:	Given the text information of a product from the Amazon CD dataset: <text information> . Image summary: <image summary> . Also given the information of co-purchased or co-reviewed products: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-purchased or co-reviewed product information is available.’) Questions: Using the product’s title, description, and image summary provided above, along with any co-purchase or co-review data, generate a concise yet informative description of the product.
Arts:	Given the text information of a product from the Amazon Art dataset: <text information> . Image summary: <image summary> . Also given the information of co-purchased or co-reviewed products: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-purchased or co-reviewed product information is available.’) Questions: Using the product’s title, description, and image summary provided above, along with any co-purchase or co-review data, generate a concise yet informative description of the product.
Reddit:	Given the text information of a post from the Reddit dataset: <text information> . Image summary: <image summary> . Also given the information of co-commented posts: text information: <neighbor text information> , image summary: <neighbor image summary> (or, if unavailable: ’No co-commented post information is available.’) Questions: Using the post’s caption and image summary provided above, along with any co-commented data, generate a concise yet informative description of the post.

Table 11: Prompts for the structure-aware aligner cases.

Movies / Toys / Grocery / CDs / Arts:	Given the target product information on Amazon : Picture: <image input> Title and description: <text information> . Question: Based on the target product’s picture, title, and description, which category does the target product belong to? Choose from the following options: <candidates set> . Assistant: <truth label>
Reddit:	Given the target post information on Reddit : Picture: <image input> Caption: <text information> . Question: Based on the target post’s picture and caption, which category does the target post belong to? Choose from the following options: <candidates set> . Assistant: <truth label>

Table 12: Prompts for the non-structure-aware predictor case.

Movies / Toys / Grocery / CDs / Arts:

Given the target product information on Amazon : Picture: <image input> Title and description: <text information> . Co-purchased or co-reviewed products: Picture1: <image input> ; Title1: <text information> ; Picture2: <image input> ; Title2: <text information> ; Picture3: <image input> ; Title3: <text information> . Question: Based on the target product’s picture, title, description, and related products, which category does the target product belong to? Choose from the following options: <candidates set> . Assistant: <truth label>

Reddit:

Given the target post information on Reddit : Picture: <image input> Caption: <text information> . Co-commented posts: Picture1: <image input> ; Caption1: <text information> ; Picture2: <image input> ; Caption2: <text information> ; Picture3: <image input> ; Caption3: <text information> . Question: Based on the target post’s picture, caption, and related posts, which category does the target post belong to? Choose from the following options: <candidates set> . Assistant: <truth label>

Table 13: Prompts for the structure-aware predictor case.

Explore similar work

Mario: Multimodal Graph Reasoning with Large Language Models
Mar 5, 2026Yuanfu Sun, Kang Li, Pengkang Guo +2LLM Reasoning with Graphs Multimodal Reasoning
New York University Shanghai · New York University · Tsinghua University +1
VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
Mar 2, 2026Athanasios Efthymiou, Stevan Rudinac, Monika Kackovic +2KG Embedding Multimodal KGs
University of Amsterdam, Amsterdam, The Netherlands
GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models
Aug 7, 2026Yuanfu Sun, Yuanhang Ren, Kang Li +5VLM Evaluation Visual Reasoning
New York University · Sensetime Research · Tsinghua University +3

Mar 5, 2026J/K move · Enter open · S save

Mario: Multimodal Graph Reasoning with Large Language Models

Yuanfu Sun, Kang Li, Pengkang Guo, Jiajin Liu+1

New York University Shanghai · New York University · Tsinghua University+1

LLM Reasoning with Graphs Multimodal Reasoning Multimodal Graph Learning Multimodal Large Language Models

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.

Abstract

Explore similar work

Mario: Multimodal Graph Reasoning with Large Language Models

VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings

GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models