Training Graph Foundation Models on The Web Graph

Ryoma Sato

cs.AISep 25, 2026

Training Graph Foundation Models on The Web Graph

Authors: Ryoma Sato

Organizations: National Institute of Informatics

Abstract

We introduce Acacia, a graph foundation model, trained on the web graph. Acacia (i) supports arbitrary feature dimensionalities and semantics without additional training, (ii) supports a wide range of tasks, including node classification, link prediction, node clustering, and graph generation, without additional training, (iii) has in-context learning capabilities, and (iv) does not rely on pretrained LLMs. In particular, existing graph foundation models often require training additional classification heads or feature projectors to accommodate new graphs or new labels, whereas Acacia does not. Moreover, existing graph foundation models often gain their capabilities by being stitched together with pretrained LLMs, whereas Acacia is trained from scratch using only the Common Crawl web graph. This is also an important result because it provides evidence that graph models can acquire emergent capabilities from scratch like LLMs.

Figures & tables

Figure 1: Token types. Acacia has four types of tokens: headers, nodes, edges, and node labels.

Figure 2: Examples of tasks supported by Acacia. Different tasks can be handled by changing the tokens included in the prompt and the generation targets. These are only examples; tasks such as jointly predicting edges and node labels are also supported.

Figure 3: An example of in-context learning. In-context learning is naturally enabled by taking entire labeled graphs or extracting ego graphs around labeled nodes and adding them to the test graph as new connected components. The example graphs can be fixed, or graphs with structures similar to the test graph can be retrieved from a database and used as examples, as in retrieval-augmented generation (RAG).

Conditioning elements	Autoregressive targets	Sampling probability
nodes + edges	labels	50%
nodes	edges + labels	10%
nodes + labels	edges	10%
labels	nodes + edges	7.5%
edges	nodes + labels	7.5%
edges + labels	nodes	7.5%

Table 1: Sampling probabilities for the elements provided as conditions and those predicted autoregressively.

Dataset	Total nodes	Classes	Original feature dim.	Official train / val / test
Cora	2,708	7	1,433	140 / 500 / 1,000
CiteSeer	3,327	6	3,703	120 / 500 / 1,000
PubMed	19,717	3	500	60 / 500 / 1,000
ogbn-products	2,449,029	47	100	196,615 / 39,323 / 2,213,091

Table 2: Dataset statistics and official data splits.

	Cora	CiteSeer	PubMed	ogbn-products
Chance level	$41.94\pm 0.29$	$32.38\pm 0.13$	$19.68\pm 0.72$	$44.15\pm 0.66$
Untrained, frozen	$41.07\pm 1.37$	$33.07\pm 0.67$	$19.80\pm 0.56$	$45.00\pm 0.17$
Acacia, frozen	$58.17\pm 1.35$	$39.50\pm 1.22$	$19.87\pm 0.74$	$66.00\pm 0.72$

Table 3: Node classification performance. Each value is the mean accuracy

\pm

sample standard deviation over 3 seeds.

	Cora	CiteSeer	PubMed	ogbn-products
Chance level	$14.29$	$16.67$	$33.33$	$2.35$
Untrained, frozen	$15.27\pm 0.92$	$17.30\pm 1.57$	$33.67\pm 1.03$	$2.37\pm 0.40$
Acacia, frozen	$54.33\pm 1.61$	$39.37\pm 2.06$	$37.43\pm 2.05$	$68.38\pm 0.32$

Table 4: Classification performance with In-Context Learning. Each value is the mean accuracy

\pm

sample standard deviation over 3 seeds.

	Cora	CiteSeer	PubMed	ogbn-products
Untrained (4 nodes, ARI)	$0.0013\pm 0.0039$	$0.0008\pm 0.0036$	$0.0009\pm 0.0030$	$0.0016\pm 0.0015$
Acacia (4 nodes, ARI)	$0.2643\pm 0.0120$	$0.2833\pm 0.0311$	$0.2034\pm 0.0043$	$0.0117\pm 0.0069$
Untrained (4 nodes, AMI)	$0.0006\pm 0.0024$	$0.0025\pm 0.0046$	$0.0003\pm 0.0018$	$0.0006\pm 0.0006$
Acacia (4 nodes, AMI)	$0.3295\pm 0.0053$	$0.3241\pm 0.0178$	$0.2627\pm 0.0121$	$0.0222\pm 0.0106$
Untrained (1 node, ARI)	$-0.0013\pm 0.0029$	$0.0050\pm 0.0167$	$0.0006\pm 0.0047$	$0.0009\pm 0.0023$
Acacia (1 node, ARI)	$0.1191\pm 0.0084$	$0.0667\pm 0.0304$	$0.0387\pm 0.0068$	$0.0047\pm 0.0116$

Table 5: Clustering performance of the untrained model and Acacia. Each value is the mean

\pm

sample standard deviation over 3 seeds.

Graph Foundation Models

Graph Foundation Models are pretrained graph-learning systems designed to transfer across datasets, tasks, and domains with different structures and features.

See all 97 papers

Mochi: Aligning Pre-training and Inference for Efficient Graph Foundation Models via Meta-Learning
Apr 23, 2026João Mattos, Arlei SilvaGraph Representation Learning Meta-Learning
Computer Science Department · Ken Kennedy Institute
To Learn is to Wander: Learning Across Graphs and Tasks with Random Walks
Oct 5, 2026Louis Tichelman, Xingyue Huang, Jinwoo Kim +1Multi-Task Learning Graph Transfer Learning
TU Wien · AITHYRA · University of Oxford +1
Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning
Sep 8, 2026Jonatan Frank, David Richerby, Ansgar ScherpTransfer Learning Text-Attributed Graph Learning
University of Ulm, Ulm, Germany · University of Essex, Colchester, UK

Apr 23, 2026J/K move · Enter open · S save

Mochi: Aligning Pre-training and Inference for Efficient Graph Foundation Models via Meta-Learning

João Mattos, Arlei Silva

Computer Science Department · Ken Kennedy Institute

Graph Representation Learning Meta-Learning Graph Foundation Models

Open paper AskPDF arXiv

We propose Mochi, a Graph Foundation Model that addresses task unification and training efficiency by adopting a meta-learning based training framework. Prior models pre-train with reconstruction-based objectives such as link prediction, and assume that the resulting representations can be aligned with downstream tasks through a separate unification step such as class prototypes. We demonstrate through synthetic and real-world experiments that this procedure, while simple and intuitive, has limitations that directly affect downstream task performance. To address these limitations, Mochi pre-trains on few-shot episodes that mirror the downstream evaluation protocol, aligning the training objective with inference rather than relying on a post-hoc unification step. We show that Mochi, along with its more powerful variant Mochi++, achieves competitive or superior performance compared to existing Graph Foundation Models across 25 real-world graph datasets spanning node classification, link prediction, and graph classification, while requiring 8

\sim

27 times less training time than the strongest baseline.

Abstract

Graph Foundation Models

Explore similar work

Mochi: Aligning Pre-training and Inference for Efficient Graph Foundation Models via Meta-Learning

To Learn is to Wander: Learning Across Graphs and Tasks with Random Walks

Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning