Hierarchical Response Preservation for Continual Adaptation of Zero-Shot Graph-Text Models

Haopeng Zhang; Yuhan Wang; Yubing Su; Yingxin Chen; Xiao Wang; Ruijie Wang; Jianxin Li

cs.LGSep 27, 2026

Hierarchical Response Preservation for Continual Adaptation of Zero-Shot Graph-Text Models

Authors: Haopeng Zhang, Yuhan Wang, Yubing Su, Yingxin Chen, Xiao Wang, Ruijie Wang, Jianxin Li

Organizations: School of Computer Science and Engineering, Beihang University Beijing, China · School of Electronic and Information Engineering Beihang University Beijing, China · School of Software, Beihang University Beijing, China

Abstract

Pretrained graph-text models align graph representations with textual semantics, enabling recognition of unseen classes and transfer across graph domains. However, as graph data and classes continually arrive, models should learn from new supervision while retaining their zero-shot transfer capabilities and historical task knowledge. Two challenges arise: (i) new classes can overturn historical predictions despite preserved distinctions among historical classes, and (ii) overly strict response preservation can stall learning of new classes. To address these challenges, we propose Hierarchical Response Preservation (HiRP). HiRP represents this competition through a hierarchical response that keeps each historical-class probability and sums new-class probabilities, preserving historical distinctions and aggregate competition while allowing distinctions within the new class group to adapt. It further uses the geometry induced by this response to guide constrained updates, retaining useful adaptation directions while controlling response drift. Across three class-incremental settings, HiRP achieves absolute gains of 1.84-7.95 percentage points in average accuracy over the strongest compared baseline in each setting, while mitigating zero-shot transfer degradation.

Figures & tables

Figure 1: Zero-shot accuracy of GraphCLIP ( Zhu et al., 2025 ) on History.

Figure 2: Evolving class competition: new classes can overturn historical predictions despite preserved discrimination among historical classes.

Figure 3: Historical accuracy with expanded candidates.

Figure 4: Uniform step shrinking near the KL limit reduces update norms while loss stalls.

Figure 5: Overview of HiRP. (1) Shared representation: Frozen encoders provide node and class features; only the readout is updated. (2) Hierarchical competition: Historical classes remain separate, while new classes share an aggregate response. (3) Constrained adaptation: The response defines drift budgets and guides feasible readout updates.

Method	4 datasets / 4 tasks			6 datasets / 6 tasks			4 datasets / 10 tasks
	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$
GCN	$50.30_{\pm 0.06}$	$21.71_{\pm 0.03}$	$\mathord{\scriptstyle-}81.26_{\pm 0.45}$	$1.00_{\pm 0.82}$	$11.99_{\pm 0.24}$	$\mathord{\scriptstyle-}84.37_{\pm 0.30}$	$14.08_{\pm 0.02}$	$9.69_{\pm 0.01}$	$\mathord{\scriptstyle-}90.58_{\pm 0.32}$
SGD	$75.79_{\pm 0.12}$	$58.36_{\pm 0.20}$	$\mathord{\scriptstyle-}25.73_{\pm 0.17}$	$83.13_{\pm 0.19}$	$58.89_{\pm 0.27}$	$\mathord{\scriptstyle-}22.64_{\pm 0.20}$	$50.24_{\pm 0.16}$	$50.05_{\pm 0.17}$	$\mathord{\scriptstyle-}41.57_{\pm 0.20}$
LwF	$\underline{76.60}_{\pm 0.14}$	$58.46_{\pm 0.56}$	$\mathord{\scriptstyle-}25.21_{\pm 0.73}$	$\underline{84.23}_{\pm 0.14}$	$61.63_{\pm 0.22}$	$\mathord{\scriptstyle-}19.03_{\pm 0.22}$	$56.70_{\pm 0.63}$	$49.87_{\pm 0.39}$	$\mathord{\scriptstyle-}40.99_{\pm 0.43}$
EWC	$75.84_{\pm 0.00}$	$58.13_{\pm 0.00}$	$\mathord{\scriptstyle-}25.95_{\pm 0.00}$	$83.29_{\pm 0.00}$	$59.08_{\pm 0.00}$	$\mathord{\scriptstyle-}22.44_{\pm 0.00}$	$50.30_{\pm 0.00}$	$50.12_{\pm 0.00}$	$\mathord{\scriptstyle-}41.36_{\pm 0.00}$
MAS	$66.09_{\pm 0.84}$	$61.11_{\pm 0.56}$	$\mathord{\scriptstyle-}12.26_{\pm 1.77}$	$66.85_{\pm 0.00}$	$62.79_{\pm 0.00}$	$\mathord{\scriptstyle-}7.79_{\pm 0.00}$	$57.74_{\pm 1.10}$	$54.27_{\pm 1.12}$	$\mathord{\scriptstyle-}31.22_{\pm 2.40}$

Table 1: Class-incremental classification over three dataset/task schedules. Means are shown with sample standard deviations in subscript. FA and AA are accuracy (%); signed AF is forgetting (percentage points), with higher values better. Bold and underline mark the best and second-best means, respectively. GraphCLIP denotes sequential joint fine-tuning; backbones and † adaptations are specified in Appendix B.2 .

Figure 6: Accuracy throughout learning, averaged over arrived tasks. Each point uses all classes introduced up to that task.

Figure 7: Zero-shot accuracy of GraphCLIP and HiRP on History during class-incremental fine-tuning.

Method	4 datasets / 4 tasks			6 datasets / 6 tasks			4 datasets / 10 tasks
	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$
Mass + radial	$77.10_{\pm 0.40}$	$67.23_{\pm 0.45}$	$\mathord{\scriptstyle-}13.91_{\pm 0.83}$	$84.12_{\pm 1.09}$	$69.60_{\pm 0.41}$	$\mathord{\scriptstyle-}10.38_{\pm 0.26}$	$64.49_{\pm 1.37}$	$63.07_{\pm 1.20}$	$\mathord{\scriptstyle-}25.38_{\pm 1.13}$
Mass + tangent	$78.50_{\pm 0.11}$	$68.07_{\pm 1.20}$	$\mathord{\scriptstyle-}14.07_{\pm 2.12}$	$85.82_{\pm 0.54}$	$68.18_{\pm 0.63}$	$\mathord{\scriptstyle-}13.24_{\pm 0.63}$	$62.31_{\pm 1.58}$	$58.06_{\pm 1.84}$	$\mathord{\scriptstyle-}31.27_{\pm 2.04}$
Hier. + radial	$75.52_{\pm 0.88}$	$69.03_{\pm 0.14}$	$\mathbf{\mathord{\scriptstyle-}7.79}_{\pm 0.41}$	$82.83_{\pm 0.47}$	$71.59_{\pm 0.14}$	$\mathbf{\mathord{\scriptstyle-}5.17}_{\pm 0.17}$	$74.36_{\pm 0.11}$	$69.71_{\pm 0.15}$	$\mathbf{\mathord{\scriptstyle-}12.76}_{\pm 0.27}$
HiRP (ours)	$\mathbf{80.09}_{\pm 0.19}$	$\mathbf{71.12}_{\pm 0.26}$	$\mathord{\scriptstyle-}8.43_{\pm 0.32}$	$\mathbf{87.04}_{\pm 0.42}$	$\mathbf{73.22}_{\pm 0.23}$	$\mathord{\scriptstyle-}6.21_{\pm 0.32}$	$\mathbf{75.43}_{\pm 0.05}$	$\mathbf{69.86}_{\pm 0.13}$	$\mathord{\scriptstyle-}14.74_{\pm 0.15}$

Table 2: Response and update-direction ablation under the same data, support rules, and numerical KL budgets. The mass response keeps only historical/current group probabilities. HiRP combines the hierarchical response with tangent correction.

Figure 8: Preserving historical decisions and maintaining adaptation. (a,b) Ten-task setting, Task 5: changes after 32 common-start updates; SGD* matches HiRP’s step norms. Error bars show sample standard deviations. (c,d) CiteSeer, Task 2: accepted update norms and training loss under the same decreasing step-radius budget. Gray denotes the shared initial trajectory.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Protected quantity	Inputs and enforcement
LwF	Historical-task outputs	Current-task inputs; distillation loss.
FRCL / FROMP	Function posteriors / past predictions	Inducing inputs / memorable examples; functional regularization.
MiB	Historical-class outputs and aggregated background/current-class probability	Segmentation inputs; background-aware distillation.
DKD	Target-versus-rest mass and non-target conditional distribution	Labeled distillation inputs; separately weighted losses.
HiRP	Historical discrimination and aggregate group competition, represented by individual historical probabilities and current-group mass	Bounded unlabeled features; local and task-level KL budgets on the same expanded candidates, with checked updates.

Appendix

Table 3: Response protection in related formulations. The comparison concerns the protected object and its inputs, not a claim that probability aggregation or functional regularization is unique to HiRP.

Method	AA $\uparrow$	Zero-shot mean $\uparrow$
SGD	50.71	56.82
Fisher proposal without KL constraints	56.84	55.20
HiRP (ours)	69.74	58.27

Appendix

Table 4: Full-sequence comparisons in the ten-task setting from a shared task-1 checkpoint. Zero-shot mean is the unweighted accuracy over History, Computer, and Instagram after task 10. Values are percentages; higher is better.

Dataset	Nodes	Classes	Train	Valid.	Test	Single schedule
Cora	2,708	7	1,624	542	542	$2,2,3$
CiteSeer	3,186	6	1,911	637	638	$2,2,2$
WikiCS	11,701	10	580	1,769	5,847	$2,2,2,2,2$
Ele-Photo	48,362	12	29,017	9,672	9,673	$2,2,2,2,2,2$
Sports-Fitness	173,055	13	34,611	17,305	121,139	–
Actor (HetGB)	4,416	5	2,116	1,411	889	–

Appendix

Table 5: Exact dataset instances and prescribed splits. The single-graph schedule lists the classes introduced at each task; the two external graphs are used in the six-task setting. WikiCS retains its official partially assigned split.

Method family	Updated representation	Historical information used in adaptation
HiRP / four-cell ablation	Shared readout on frozen GraphCLIP features	Bounded historical and external unlabeled features; response constraints.
SGD / LwF	Same readout and frozen feature cache	SGD: current CE only. LwF: previous-model responses on current inputs; no historical-node CE.
EWC / MAS / OGD	Shared readout	Stored importance/projection information under each saved recipe; not the HiRP response constraint.
LoRA / G2LoRA †	Graph adapters/readout; selected graph and text adapters	Published-mechanism adaptations with their own retained state and optimization budgets.
InfLoRA †	Last graph-attention K/V adapters and task-wise linear heads	Activation subspaces and frozen prior factors; no historical supervised replay.
SD-LoRA	Last graph-attention Q/V adapters and task-wise linear heads	Frozen prior low-rank directions with learned magnitudes; no historical supervised replay.

Appendix

Table 6: What is and is not shared across the compared implementations. All displayed methods use the prescribed current training roots and cumulative-candidate test protocol; history access and adaptation remain method-specific.

Method	4 datasets / 4 tasks	6 datasets / 6 tasks	4 datasets / 10 tasks
GraphGPT †	250.9 $\pm$ 0.8	742.2 $\pm$ 128.2	217.5 $\pm$ 26.4
LLaGA †	174.5 $\pm$ 59.7	523.9 $\pm$ 67.5	142.2 $\pm$ 19.6
InfLoRA †	19.4 $\pm$ 2.0	31.8 $\pm$ 3.6	27.2 $\pm$ 1.7
SD-LoRA	21.2 $\pm$ 0.3	37.2 $\pm$ 5.0	34.2 $\pm$ 2.6
HiRP (ours)	6.7 $\pm$ 1.1	16.2 $\pm$ 5.5	7.2 $\pm$ 2.5

Appendix

Table 7: Recorded training time per complete sequence in minutes (mean

\pm

sample standard deviation). Feature precomputation, validation, and testing are excluded.

Method	Cora		CiteSeer		WikiCS		Photo
	FA $\uparrow$	AA $\uparrow$	FA $\uparrow$	AA $\uparrow$	FA $\uparrow$	AA $\uparrow$	FA $\uparrow$	AA $\uparrow$
GCN	54.61 $\pm$ 0.18	31.42 $\pm$ 0.11	32.45 $\pm$ 0.31	31.36 $\pm$ 0.30	43.76 $\pm$ 3.69	36.63 $\pm$ 4.14	14.58 $\pm$ 0.04	16.39 $\pm$ 0.05
Cosine	65.87 $\pm$ 1.39	62.54 $\pm$ 0.98	42.79 $\pm$ 4.27	38.87 $\pm$ 4.34	58.07 $\pm$ 1.38	51.58 $\pm$ 1.85	56.34 $\pm$ 1.86	60.00 $\pm$ 3.01
TEEN	59.04 $\pm$ 4.36	54.27 $\pm$ 3.70	54.49 $\pm$ 2.91	48.21 $\pm$ 2.67	59.76 $\pm$ 1.51	55.31 $\pm$ 0.60	49.75 $\pm$ 3.19	52.89 $\pm$ 0.99
TPP-style	68.20 $\pm$ 0.59	63.53 $\pm$ 0.47	70.64 $\pm$ 0.59	69.10 $\pm$ 0.55	58.10 $\pm$ 1.10	58.61 $\pm$ 1.23	50.61 $\pm$ 0.04	66.57 $\pm$ 0.09
SGD	67.65 $\pm$ 0.28	56.49 $\pm$ 0.70	63.85 $\pm$ 0.50	60.82 $\pm$ 0.49	62.22 $\pm$ 0.06	55.13 $\pm$ 0.06	28.68 $\pm$ 0.05	28.85 $\pm$ 0.07

Appendix

Table 8: Single-graph class-incremental classification. All entries use the same class splits and cumulative-candidate evaluation. FA and AA are percentages; bold denotes the largest mean among the displayed methods. The displayed methods retain their respective backbones.

Method	CiteSeer		Photo
	FA $\uparrow$	AA $\uparrow$	FA $\uparrow$	AA $\uparrow$
BERT	31.24 $\pm$ 0.36	30.20 $\pm$ 0.35	14.29 $\pm$ 0.01	16.06 $\pm$ 0.01
SimpleCIL (BERT)	57.58 $\pm$ 4.65	57.75 $\pm$ 4.23	31.98 $\pm$ 6.98	43.63 $\pm$ 5.02
HiRP (ours)	73.88 $\pm$ 1.27	70.71 $\pm$ 1.79	71.23 $\pm$ 0.66	75.45 $\pm$ 1.70

Appendix

Table 9: Additional language-model comparisons on CiteSeer and Photo. Entries are means and sample standard deviations, using the same evaluation splits as Table 8 .

Dataset	Trans.	$N_{\mathrm{ref}}$	$C_{+}$ (%)	$C_{\mathrm{cert}}$ (%)	Forgotten / certified
Cora	6	724	97.63 $\pm$ 0.79	88.89 $\pm$ 2.54	0 / 642
CiteSeer	6	1,107	98.28 $\pm$ 1.80	91.82 $\pm$ 1.91	0 / 1,032
WikiCS	12	21,565	98.21 $\pm$ 0.16	96.42 $\pm$ 0.24	0 / 20,717
Photo	15	44,367	96.41 $\pm$ 0.18	80.86 $\pm$ 0.48	0 / 35,906

Appendix

Table 10: Single-graph certificate coverage.

N_{\mathrm{ref}}

counts reference-correct historical sample–transition pairs on expanded candidates; Trans. counts transitions across runs.

C_{+}

and

C_{\mathrm{cert}}

are percentages (transition mean within each run, then mean

\pm

sample SD). The last column reports pooled correct-to-incorrect changes among certified pairs.

Figure 9: Single-graph coverage of Theorem 1 . Each point is a reference-correct historical sample–transition pair with $\delta_{R}>0$ . Circles indicate retained predictions and crosses indicate correct-to-incorrect changes. The dashed line is $D_{H}=\delta_{R}^{2}/2$ ; the shaded region below it satisfies the strict certificate and contains no forgotten pairs. Panel annotations give the number of nonpositive-margin pairs excluded from the plot. Both axes use logarithmic scales.

Method	4 datasets / 4 tasks			6 datasets / 6 tasks			4 datasets / 10 tasks
	$R$ $\uparrow$	$Q$ $\uparrow$	AF $\uparrow$	$R$ $\uparrow$	$Q$ $\uparrow$	AF $\uparrow$	$R$ $\uparrow$	$Q$ $\uparrow$	AF $\uparrow$
TPP-style	68.28 $\pm$ 0.34	70.57 $\pm$ 0.34	$-$ 2.28 $\pm$ 0.04	70.79 $\pm$ 0.06	72.19 $\pm$ 0.05	$-$ 1.40 $\pm$ 0.02	59.60 $\pm$ 0.19	69.58 $\pm$ 0.24	$-$ 9.98 $\pm$ 0.09
SimpleCIL	59.83 $\pm$ 1.15	61.59 $\pm$ 1.19	$-$ 1.76 $\pm$ 0.05	48.57 $\pm$ 0.73	49.90 $\pm$ 0.79	$-$ 1.33 $\pm$ 0.07	50.75 $\pm$ 0.95	58.51 $\pm$ 0.87	$-$ 7.77 $\pm$ 0.09
Hier. + radial	66.38 $\pm$ 0.69	74.17 $\pm$ 0.63	$-$ 7.79 $\pm$ 0.41	71.53 $\pm$ 0.33	76.70 $\pm$ 0.26	$-$ 5.17 $\pm$ 0.17	67.65 $\pm$ 0.16	80.42 $\pm$ 0.12	$-$ 12.76 $\pm$ 0.27
HiRP (ours)	66.48 $\pm$ 0.37	74.91 $\pm$ 0.57	$-$ 8.43 $\pm$ 0.32	73.45 $\pm$ 0.13	79.66 $\pm$ 0.25	$-$ 6.21 $\pm$ 0.32	67.26 $\pm$ 0.09	82.00 $\pm$ 0.11	$-$ 14.74 $\pm$ 0.15

Appendix

Table 11: Historical-task retention for baseline variants and the hierarchical-response ablation.

R

is final historical-task accuracy and

Q

is accuracy when each historical task was first learned; the final task is excluded. Entries report means

\pm

sample standard deviations in percentages or percentage points, and

\mathrm{AF}=R-Q

before rounding. Bold marks the largest displayed mean.

Method	4 datasets / 4 tasks			6 datasets / 6 tasks			4 datasets / 10 tasks
	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$	FA $\uparrow$	AA $\uparrow$	AF $\uparrow$
SGD	75.79 $\pm$ 0.18	58.35 $\pm$ 0.22	$-$ 25.67 $\pm$ 0.08	83.02 $\pm$ 0.14	58.77 $\pm$ 0.28	$-$ 22.70 $\pm$ 0.10	50.16 $\pm$ 0.15	49.96 $\pm$ 0.15	$-$ 41.67 $\pm$ 0.18
LwF	76.63 $\pm$ 0.18	58.43 $\pm$ 0.61	$-$ 25.24 $\pm$ 0.69	84.14 $\pm$ 0.07	61.55 $\pm$ 0.26	$-$ 19.06 $\pm$ 0.18	56.31 $\pm$ 0.33	49.64 $\pm$ 0.30	$-$ 41.23 $\pm$ 0.27
Memory-LwF	77.13 $\pm$ 0.17	59.37 $\pm$ 0.63	$-$ 24.14 $\pm$ 0.65	84.99 $\pm$ 0.08	63.63 $\pm$ 0.48	$-$ 16.75 $\pm$ 0.36	54.47 $\pm$ 0.11	47.39 $\pm$ 0.39	$-$ 44.13 $\pm$ 0.38
Memory-KD	78.18 $\pm$ 0.13	68.14 $\pm$ 0.22	$-$ 11.19 $\pm$ 0.27	85.46 $\pm$ 0.12	70.58 $\pm$ 0.17	$-$ 7.63 $\pm$ 0.19	60.71 $\pm$ 0.12	58.65 $\pm$ 0.42	$-$ 29.97 $\pm$ 0.45
HiRP (ours)	80.09 $\pm$ 0.19	71.12 $\pm$ 0.26	$-$ 8.43 $\pm$ 0.32	87.04 $\pm$ 0.42	73.22 $\pm$ 0.23	$-$ 6.21 $\pm$ 0.32	75.43 $\pm$ 0.05	69.86 $\pm$ 0.13	$-$ 14.74 $\pm$ 0.15

Appendix

Table 12: Additional distillation controls with identical feature support. Means

\pm

sample standard deviations; FA/AA are accuracy (%) and AF is signed forgetting (points). Bold marks the highest mean.

Figure 10: Sensitivity to four core parameters on three continual-learning streams. Curves show final AA; stars mark default values. The first three horizontal axes use logarithmic scales, and support capacity uses a linear scale.

Explore similar work

Sep 27, 2026cs.LG

ZeroGAR: Benchmarking the Adversarial Robustness of Zero-Shot Graph Models

Zero-shot graph models (ZGMs), which learn transferable knowledge from source graphs and directly apply to unseen target graphs without any adaptation, have achieved promising performance and attracted considerable attention. Despite their proliferation, existing ZGMs are predominantly evaluated on clean graphs, while existing graph robustness benchmarks mainly focus on supervised settings, leaving a fundamental question largely unexplored: How robust are ZGMs when their unseen target graphs are exposed to adversarial manipulation? In this paper, we answer this question by proposing ZeroGAR, the first systematic benchmark for evaluating the adversarial robustness of ZGMs. ZeroGAR evaluates 13 representative ZGMs from 3 different paradigms on 8 graph datasets across 4 domains, covering both in-domain and cross-domain transfer under structural, textual, and node injection attacks with multiple perturbation budgets. It further investigates whether existing graph defenses remain effective in the zero-shot setting. Extensive experiments reveal that strong clean zero-shot performance does not guarantee adversarial robustness, with three key findings: (1) Vulnerability patterns are related to model prediction mechanisms: GNN-based methods are particularly vulnerable to structural and node injection attacks, whereas LLM-based methods are more vulnerable to textual attacks; (2) Stronger LLM backbones introduce a structure-text robustness trade-off; (3) Existing graph defense methods do not consistently improve zero-shot robustness and may compromise clean performance. We hope that ZeroGAR will facilitate rapid, equitable evaluation and inspire further innovative research in ZGM security.

Zhongjian Zhang, Xiao Wang, Busheng Zhang +5

Beijing University of Posts and Telecommunications · Beihang University · Tsinghua University +2

Zero-Shot Robustness LLM Defense Mechanisms Large Language Model Backbones

Ask PDF

Jul 11, 2026cs.AI

UNIT: Unleash Large Language Models Potential for Graph Continual Learning

In real-world multimodal web scenarios, graph-structured data often arrives in a streaming manner, making graph continual learning a crucial paradigm for continuously modeling such evolving structures. However, existing graph continual learning methods still face two fundamental challenges. 1) semantic-structural separation, where the graph-based methods excel at modeling topological relationships but neglect deep semantics. 2) imbalanced knowledge transfer, where existing models fail to effectively leverage general knowledge gained from early tasks to benefit subsequent new tasks. To address above issues, we propose a novel framework, \textbf{UN}leash Large Language Models PotentIal for Graph ConTinual Learning (UNIT). By fine-tuning large language model only on the first task, we bridge the distributional gap between the pre-trained LLM corpus and the target task dataset to enhance the adaptability of LLMs for graph-structured tasks. Meanwhile, we propose an uncertain-aware anchor generation mechanism to effectively preserve representative knowledge across tasks, avoiding the neglect of universal knowledge learned from previous tasks. Additionally, we introduce structural confluence modeling to explicitly integrates graph topology information into semantic information, enhancing the collaborative capabilities between semantic understanding and structural modeling. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in the graph continual learning task.

Tairan Huang, Yili Wang, Beibei Hu +4

Central South University Changsha, China · Hongkong University of Science and Technology (Guangzhou) Guangzhou, China · Hunan Institute of Engineering Changsha, Hunan

Replay-Based Continual Learning Sparse Spatial Anchors Separation

Ask PDF

Jul 28, 2026cs.AI

CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain. This setting presents two key challenges. First, models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. Second, without target-domain fine-tuning, node representations remain entangled with domain-specific structures and modality-specific characteristics, obscuring shared concepts in unseen domains. To address these challenges, we propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer. CHARM replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relations. These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for a large language model . Experiments show consistent improvements on zero-shot multimodal graph tasks.

Ankang Yang, Jitao Zhao, Di Jin +2

School of Computer Science and Technology, College of Intelligence and Computing, Tianjin University · Data Science Program, Columbian College of Arts and Sciences, The George Washington University

Graph Foundation Model Multimodal Foundation Model Cross-Modal

Ask PDF