GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

Murad Hossen; Tasneem Selim; Gurur Gamgam; Tuga Yousif; Abderrahmane Kasmi; Ikram Aissiou; Mubaraq Onipede; Faran Taimoor Butt

cs.LGOct 4, 2026

GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

Authors: Murad Hossen, Tasneem Selim, Gurur Gamgam, Tuga Yousif, Abderrahmane Kasmi, Ikram Aissiou, Mubaraq Onipede, Faran Taimoor Butt, +17 more

Organizations: University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt · Bogazici University, Turkey · Ankara Yıldırım Beyazıt University, Turkey · ESI, Algiers, Algeria · University of Algiers 1, Algeria · York St John University London, UK · Air University, Islamabad, Pakistan · LSISI, ENSA, Mohammed Premier University, Oujda, Morocco · Universidad Católica San Pablo, Arequipa, Peru · Busitema University, Uganda · University of Laghouat, Algeria · ISI, University of Tunis El Manar, Tunisia · Tribhuvan University, Nepal · IIT Roorkee, India · Shobhit Institute of Engineering and Technology, India · MIT World Peace University, Pune, India · University of Kinshasa, DR Congo · University of Bertoua, Cameroon · University of the Witwatersrand, Johannesburg, South Africa · African Institute for Mathematical Sciences, Research and Innovation Centre (AIMS RIC), Kigali, Rwanda · KNUST, Ghana · NSUT Delhi, India · ISIMM Monastir, Tunisia · Addis Ababa University, Ethiopia

Abstract

Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network (GNN) coding tasks under realistic competition settings. To address this gap, this paper introduces GNN-CB, the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. GNN-CB consists of 18 curated competitions spanning node-, edge-, and graph-level prediction across diverse graph categories, domains, and difficulty tiers. All submissions are evaluated through a unified automated pipeline with hidden test sets and standardized scoring. Human participants solve tasks under controlled competition constraints, while LLMs are evaluated using a frozen zero-shot prompting protocol based on a plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation within the same protocol. Under our evaluated protocol, LLMs rarely match Human Top performance and show less stable performance across competitions. No single model dominates: a few competitions are won by LLMs, yet humans still hold the top score on most tasks. We release GNN-CB as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible execution pipelines. Beyond benchmarking, GNN-CB provides a practice-oriented resource for studying GNN implementation across progressively diverse graph-learning tasks. The benchmark and evaluation framework are publicly available at https://basiralab.github.io/GNN-CB/.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of GNN-CB .

#	Competition	Task Description	Cat.	Diff.	N	E	Feat.	Domain	Metric	Dataset
C01	PROVEN-GNN	Graph classification on source code representations	Hetero	Med	260K	1.7M	527	Cybersecurity	Macro-F1	DiverseVul
C02	CGCC	Graph classification of city street-network layouts	Homo	Hard	9.396K	13.533K	3	Urban	Macro-F1	OpenStreetMap
C03	Real Or Fake!	Binary graph classification for fake news detection	Hier	Easy	314K	308K	1078	Social	Binary F1	GossipCop
C04	GLIMPS-GNN	Inductive cfRNA $\to$ preeclampsia classification	Gen	Hard	320	3200	6650	Biomedical	F1	GEO
C05	GNN_BACE	Molecular graph classification for BACE-1 inhibition	Mol	Med	32	34	8	Medical	Macro-F1	BACE (MoleculeNet)
C06	DiaGraph	Binary node classification for diabetes prediction	Homo	Hard	96K	1.2M	12	Medical	Macro-F1	Kaggle Diabetes

Table 1: Overview of all 18 GNN Coding Benchmark competitions. Cat. = Graph Category; Diff. = Difficulty; N = typical number of nodes; E = typical number of edges; Feat. = node feature dimensionality. For classification tasks, we report Macro-F1, Accuracy, and Area Under the Curve (AUC), depending on the competition specification. For regression tasks, we use Mean Absolute Error (MAE) for single-target regression and Normalized Mean Absolute Error (NMAE) for multi-target regression.

Panel of Figure 2 — Figure 2: LLM evaluation pipeline.

Panel of Figure 3 — Figure 3: Per-task human-performance statistics across all 18 GNN-CB competitions: minimum, average, maximum, and standard deviation of participant scores. Panel (a) covers the 16 classification competitions on a common 0–1 axis (F1 / Accuracy / AUC); panel (b) covers the two regression competitions (C14 BrainAge, MAE; C17 Mix2Print, NMAE) on independent axes due to different metric scales. Higher is better in (a); lower is better in (b).

#	Competition	Diff.	P	AVG	STD	High	Low	Metric
C01	PROVEN-GNN	Med	17	0.792	0.101	0.900	0.435	F1
C02	CGCC	Hard	17	0.383	0.086	0.537	0.205	F1
C03	Real Or Fake!	Easy	15	0.945	0.053	0.980	0.767	F1
C04	GLIMPS-GNN	Hard	16	0.389	0.094	0.600	0.28	F1
C05	GNN_BACE	Med	14	0.496	0.088	0.674	0.391	F1
C06	DiaGraph	Hard	15	0.570	0.125	0.822	0.477	F1

Table 2: Descriptive statistics of human performance across competitions. Difficulty level (Diff), number of participants (P), average score (AVG), and standard deviation (STD) summarize overall human performance.High and Low columns report the maximum and minimum human scores observed, respectively, Metric indicates the evaluation criterion used for tasks.

Figure 4: Best, Mean, and Worst Gap performance for leading LLMs on the classification tasks of the GNN-CB benchmark, grouped by model family. Best Gap denotes the best performance margin across the 16 classification competitions compared to the human top performance. Mean Gap is the average performance difference relative to the human top performance across all classification competitions. Worst Gap represents the most negative performance difference relative to the human top performance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Composition of the 18 GNN-CB competitions attempted by human participants, summarized along four axes: graph category (top-left), difficulty tier (top-right), task type (bottom-left), and application domain (bottom-right). Per-slice counts and competition codes are listed in the legend below each donut.

Figure 6: Global distribution of the 18 participating universities that co-designed and evaluated the GNN-CB benchmark. Each highlighted country is annotated with the competition(s) contributed by its team(s); the colour key appears in the legend. The cohort spans 12 countries across Africa, the Americas, Asia, and Europe.

Panel of Figure 7 — Figure 7: Human evaluation pipeline of GNN-CB . Participants prepare graph data (nodes, edges, adjacency, and feature matrices) from the released data files, then design and train a GNN-only solution under fixed reproducibility constraints ( seed = 25 , three CPU hours, at most five hyperparameter-tuning runs, early stopping permitted). They generate a predictions.csv file, encrypt it locally into predictions.csv.enc together with a metadata.json , push the encrypted submission to their GitHub fork, and open a pull request against the main repository. An automated GitHub Actions workflow decrypts the submission, scores it against the hidden test labels, updates the live leaderboard, and closes the pull request. A documentation log—maintained throughout development—records the hyperparameter-search strategy, runtime, hardware specifications, loss-curve verification, and any LLM-assisted utility usage.

#	Model Name	Model Engine Name	Source
Proprietary Chat LLMs
1	Claude Opus 4.6 Anthropic [2026a]	claude-opus-4-6	Link
2	Gemini-3 Flash Google DeepMind [2026]	gemini-3.0-flash	Link
Open-source Chat LLMs
3	Llama-3.3 70B Meta AI [2024]	Llama-3.3-70B-Instruct	Link
4	Qwen2.5-Coder 32B Alibaba Cloud [2024]	Qwen2.5-Coder-32B-Instruct	Link

Appendix

Table 4: List of LLMs evaluated in GNN-CB , grouped by access type (proprietary vs. open-source). Default decoding parameters are temperature = 0 and max_output_tokens = 10K , applied where the provider’s API exposes these settings. API-only (non-agent) runs follow this per-call output budget, while agent-mode runs may consume substantially more tokens across multiple internal tool calls and extended-thinking traces, governed by each agent’s native session budget rather than the per-call cap. The Model Engine Name column lists the exact API identifier used in our orchestration scripts; the Source column links to the provider’s official model documentation.

Panel of Figure 8 — Figure 8: A frozen prompt construction is instantiated from the repo, and then it is passed to the LLM, which outputs a plan and code.

Panel of Figure 9 — Figure 9: Plan-then-code paradigm illustrated for C03 (Real or Fake!): a 5-step structured plan (left) and the LLM-generated, executable implementation skeleton (right).

Figure 10: Best, Mean, and Worst Gap performance for leading LLMs on the Easy classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top

Figure 11: Best, Mean, and Worst Gap performance for leading LLMs on the Medium classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top

Figure 12: Best, Mean, and Worst Gap performance for leading LLMs on the Hard classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top

Figure 13: LLM vs Human Average Performance across difficulty levels

Figure 14: LLM performance grouped by model family on Easy classification tasks. Performance metrics capture the best, mean, and worst scores achieved by each model.

Explore similar work

Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models
Aug 2, 2026Fali Wang, Ali Al-Lawati, Iliyas Bektas +5Reasoning Benchmark Multi-Turn Benchmark
The Pennsylvania State University, University Park, PA, USA · Rensselaer Polytechnic Institute, Troy, NY, US
GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs
Sep 14, 2026Zixiang Xu, Yanbo Wang, Chenxi Wang +6Graph Representations Large Language Model Benchmarks
University of Southern California · University of California, Los Angeles · Mohamed bin Zayed University of Artificial Intelligence +2
GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks
Aug 3, 2026Jiarui Tan, Zhongjian Zhang, YaBo Guo +5Large Language Model Agents
Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong · Peking University

Aug 2, 2026, cs.CLJ/K move · Enter open · S save

Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models

Fali Wang, Ali Al-Lawati, Iliyas Bektas, Jinxuan Fang+4

The Pennsylvania State University, University Park, PA, USA · Rensselaer Polytechnic Institute, Troy, NY, US

Reasoning Benchmark Multi-Turn Benchmark

Open paper AskPDF arXiv

Graph reasoning provides a promising testbed for evaluating the reasoning ability of large language models (LLMs), as graph instances can be programmatically generated, structurally controlled, and naturally scaled to long-input settings. However, existing graph reasoning benchmarks have limited coverage of data complexity, rely heavily on manual construction, and lack unified evaluation across text-based and code-based reasoning modes. To address these limitations, we propose {\dataset}, a five-stage \textit{semi-automatic} framework for constructing complex graph reasoning benchmarks. It expands benchmark coverage along five dimensions: \textit{Graph Size}, \textit{Task Complexity}, \textit{Task Description}, \textit{Graph Loading}, and \textit{Task Source}. The framework uses an LLM-based data generator to automatically produce task descriptions, graph data, reference solutions, graph-loading scripts, question forms, and evaluation scripts, while retaining human validation at key quality-control stages. Based on it, we construct a benchmark with

202

tasks and evaluate LLMs under text-based, code-based, and augmented reasoning settings. Experiments show that the complexity dimensions reveal model limitations that are less visible in existing benchmarks; existing fine-tuned models struggle to generalize to GraphGym, whereas retrieval-augmented methods show scenario-dependent adaptability, improving textual reasoning but not consistently improving coding reasoning. These findings suggest that ours serves as a challenging and diagnostic benchmark for graph reasoning and provides empirical guidance for future enhancement methods. Code and dataset will be published soon.

Abstract

Explore similar work

Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models

GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs

GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks