cs.LGOct 10, 2026
SavePATTERN Can Be Solved from Memory: Generator-Aware Baselines for Synthetic Node Classification
Organizations: University of Zanjan
Abstract
PATTERN, a synthetic node-classification benchmark still used to rank graph networks, plants a random subgraph in a random host graph and asks which nodes belong to it. Its generator draws 100 patterns once and plants the same ones in every split, so each test graph carries a pattern that also appears in 100 training graphs. A matcher that reads these patterns off the training labels and looks them up in each test graph reaches 99.2 class-weighted accuracy with no trained parameters, against 89.8 for the best published model. Inference under the generator without the patterns reaches 86.8. Two published models, CKGConv and GRIT, score lower on patterns they have not seen: on new graphs with fresh patterns both fall below this pattern-blind predictor, and withholding patterns from CKGConv's training lowers its accuracy on the test graphs that carry them. Training CKGConv with a new pattern in every graph raises its accuracy on fresh patterns by about 2 points, to the level of the pattern-blind predictor, at some cost on the reused patterns. The same kind of inference under the generator matches the best published result on CLUSTER and, without any training, scores above every published result we found on Minesweeper's released board. We recommend that synthetic benchmarks report scores on fresh templates beside reused ones, publish baselines built from their generators, and state how their metric is weighted and pooled.
Figures & tables
Figure 1: The identical upper-left pattern appears in a training graph and a test graph of PATTERN’s released files. Test graph 60 is the first test graph whose pattern has 20 nodes, the size the benchmark’s paper gives; training graph 300 is the first training graph that carries it. Each adjacency matrix places pattern nodes first in the same canonical order. The host’s 100 and 96 nodes follow in spectral-clustering groups, larger groups first and stored order within each group. Edges among pattern nodes are black, edges from pattern to host dark gray, and host edges light gray; the grayscale strip gives node features. The pattern’s edges and features are identical, while the host and the edges joining it to the pattern are redrawn. This pattern appears in 100 training graphs.
| Benchmark | Without the graph | Best in the benchmark’s paper | Best published |
|---|---|---|---|
| PATTERN | 50.52 (MLP) | 86.51 (GatedGCN-PE) | 89.75 0.03, PPGT ( Ma et al., 2026 ) |
| CLUSTER | 20.97 (MLP) | 76.08 (GatedGCN-PE) | 81.59 0.19, GRPE-Small ( Park et al., 2022 ) |
| Minesweeper | 50.89 (ResNet) | 93.91 (GAT-sep) | 99.55 0.15, SIES ( Chen et al., 2026 ) |
Table 1: Published scores for the three benchmarks. PATTERN and CLUSTER use class-weighted accuracy on 2,000 and 1,000 test graphs; Minesweeper uses ROC AUC over ten masks of one board of 10,000 cells. Standard deviations are over the authors’ seeds, or over masks for Minesweeper. The predictor without the graph (a multilayer perceptron for PATTERN and CLUSTER, a residual network for Minesweeper) and the next column come from each benchmark’s paper. The last column is the best comparable published score we found, preprints included; the best peer-reviewed Minesweeper score we found is 99.07 0.25 ( Ribeiro et al., 2026 ) .
| Predictor | Reference | First public | Class-weighted accuracy |
| MLP | Dwivedi et al. (2023) | 2020 | 50.5 |
| GatedGCN-PE | Dwivedi et al. (2023) | 2020 | 86.5 |
| DGN | Beaini et al. (2021) | 2020 | 86.7 |
| SAN | Kreuzer et al. (2021) | 2021 | 86.6 |
| GCN-AK+ | Zhao et al. (2022) | 2021 | 86.9 |
| k-subtree SAT | Chen et al. (2022) | 2022 | 86.9 |
Table 2: PATTERN’s official test split: selected published models and our predictors. First public is the year of the first arXiv version that contains the cited score. Published values are the authors’ means over seeds; our intervals resample graphs. The original benchmark baselines use batch means, and the code of GraphGPS, GRIT, CKGConv, and GRASS pools the split (Appendix C ). The matcher and the largest-match rule use the training patterns directly and have no trained parameters. We leave out QPE’s 90.2: its own run of GRIT scores 90.4, above GRIT’s published 87.2, it states no rule for choosing the reported epoch, and we could not open the code it links to check its metric ( Thabet et al., 2024 ) .
Figure 2: Published class-weighted accuracy on PATTERN, by the first arXiv version that reports each score. The gray band is the pattern-blind predictor’s graph-bootstrap interval, with its score marked inside; the band describes this predictor, not the best possible pattern-blind score. The matcher at 99.2 and the multilayer perceptron at 50.5 lie outside the plotted range; Table 2 gives the values.
| Official test split | Training patterns, | Fresh patterns | Fresh patterns, minus the | |
|---|---|---|---|---|
| new graphs | pattern-blind predictor | |||
| Likelihood matcher | 99.2 (99.0 to 99.3) | 99.4 (99.0 to 99.7) | 60.1 (59.8 to 60.5) | 26.3 ( 26.8 to 25.8) |
| Largest-match rule | 97.3 (96.9 to 97.6) | 97.6 (97.0 to 98.2) | 53.7 (53.4 to 54.1) | 32.7 ( 33.2 to 32.2) |
| Pattern-blind predictor | 86.8 (86.5 to 87.0) | 86.5 (86.1 to 87.0) | 86.4 (85.9 to 86.8) | |
| CKGConv, seed 0 | 88.8 (88.6 to 89.1) | 88.6 (88.1 to 89.1) | 84.5 (84.0 to 85.0) | 1.9 ( 2.2 to 1.6) |
| CKGConv, seed 1 | 88.3 (88.0 to 88.5) | 87.8 (87.3 to 88.3) | 84.3 (83.8 to 84.8) | 2.1 ( 2.4 to 1.7) |
Table 3: CKGConv trained with its released recipe, and three inference methods, on official, familiar-pattern, and fresh-pattern graphs. Class-weighted accuracy is pooled over each set. Intervals resample whole graphs; the last column is a paired difference on the same fresh graphs, computed before rounding. Appendix D gives CKGConv’s results by seed.
Figure 3: Class-weighted accuracy on the official test split, on new graphs with training patterns, and on fresh patterns. Right of the divider are official graphs whose patterns were kept in or held out of CKGConv’s training. Circles: the pattern-blind predictor; triangles: CKGConv trained on the released patterns; diamonds: GRIT; squares: CKGConv trained without the held-out patterns. Filled and open markers are seeds 0 and 1. Bars are 95% graph-bootstrap intervals for each fitted model, not intervals across seeds. The matcher lies outside this range (Table 3 ); Tables 11 and 12 give the other models’ scores.
Figure 4: CKGConv trained on the released patterns (triangles) or with a new pattern per training and validation graph (squares), and the pattern-blind predictor (circles), on the same graph sets. Filled and open markers are seeds 0 and 1. Bars are 95% graph-bootstrap intervals for each fitted model. Training on fresh patterns raises accuracy on fresh patterns and lowers it on the reused ones. Table 13 gives the values.
| Benchmark | Information used | Method | Estimate | 95% interval | SD over splits | Diagnostics |
| CLUSTER | edges, labeled nodes | Gibbs sampler, block swaps, 8 chains | 82.3 | 81.5 to 83.0 | chain agreement 0.96 (median graph), 0.50 (worst graph) | |
| best published | 81.6 | GRPE-Small ( Park et al., 2022 ) , SD 0.19 over its seeds; later GRIT 80.0, GRASS 79.5, PPGT 80.0 | ||||
| Minesweeper | features | belief propagation | 98.77 | 0.12 | stopped at its limit of 400 sweeps, last change 0.021 | |
| features | strip sampler | 99.86 | 0.05 | not converged; 4 chains ran, and all 4 reached a valid board; each alone 99.11 to 99.58 | ||
| features, training labels | belief propagation | 99.92 | 0.06 | stopped at its limit of 400 sweeps on every split | ||
| best published | 99.55 | 0.15 | SIES ( Chen et al., 2026 ) |
Table 4: Baselines built from the generators of CLUSTER and Minesweeper. CLUSTER uses pooled class-weighted accuracy with a graph-bootstrap interval; its chain agreement is the lowest, over successive chains, agreement with the labels of the earlier chains pooled together. Minesweeper uses mean ROC AUC and standard deviation across ten masks of one board. Inference from the features alone and inference conditioned on training labels use different information. Belief propagation stopped at its sweep limit, and neither it nor the samplers are known to have converged. The best published Minesweeper score is from a preprint; the best peer-reviewed one we found is 99.07 ( Ribeiro et al., 2026 ) .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | What the generator draws |
|---|---|
| PATTERN | Host: a stochastic block model with 5 blocks of sizes drawn uniformly from 5 to 34, edge probability 0.5 within a block and 0.35 across blocks; features uniform on three values. Pattern: a random graph of a size drawn uniformly from 5 to 34 nodes, with edge probability 0.5 and uniform features, drawn once for each of the 100 patterns; each pattern node joined to each host node with probability 0.5. Released pattern sizes: 6 to 34 nodes. |
| CLUSTER | A stochastic block model with 6 blocks of sizes drawn uniformly from 5 to 34, edge probability 0.55 within a block and 0.25 across; one node per block, chosen uniformly, carries the block’s index as its feature, every other node a zero. |
| Minesweeper | One square grid of side 100 with each cell joined to its eight neighbors; mines on a fixed share 0.2 of the cells, placed uniformly at random; a random 50% of the cells show the number of mined neighbors, capped at 5, as a one-hot feature, and the rest show a flag for an unknown count. |
Table 5: What each generator draws, with the parameters as its code sets them.
| Quantity | Value |
| Training graphs | 10,000 |
| Distinct planted patterns in training | 100 |
| Training graphs per pattern | 100 |
| Validation graphs whose pattern occurs in training (%) | 100.0 |
| Test graphs whose pattern occurs in training (%) | 100.0 |
| Distinct planted patterns in the test split | 100 |
Table 6: PATTERN’s patterns in the released files, counted by canonical form with node features.
| Exact recovery (%) | Likelihood matcher, inexact graphs | |||||||
| Graphs | Pattern size | Count | Likelihood | Largest | Near miss | Other size | Low overlap | None found |
| Test split | 6 to 13 | 480 | 74.8 | 45.0 | 30 | 73 | 18 | 0 |
| Test split | 14 to 34 | 1,520 | 99.7 | 99.7 | 5 | 0 | 0 | 0 |
| Test split | all | 2,000 | 93.7 | 86.6 | 35 | 73 | 18 | 0 |
| Fresh patterns | 6 to 13 | 135 | 0.0 | 0.0 | 2 | 121 | 12 | 0 |
| Fresh patterns | 14 to 34 | 365 | 0.0 | 0.0 | 0 | 365 | 0 | 0 |
Table 7: Exact recovery of the planted set by pattern size, for the likelihood matcher and the largest-match rule, and the likelihood matcher’s kinds of error, on the official test split and on the fresh-pattern graphs. The kinds depend only on the labeled set’s size and its overlap with the planted set: a near miss has the planted set’s size and shares all but one or two of its nodes; other size has a different size; low overlap has the planted set’s size and shares fewer nodes; none found labels no node.
| Class-weighted accuracy | Pooled minus | Plain node | Plain minus | ||
|---|---|---|---|---|---|
| Predictor | pooled | batch mean | batch mean | accuracy | pooled |
| Likelihood matcher | 99.2 | 99.0 | 0.1 | 99.6 | 0.4 |
| Largest-match rule | 97.3 | 96.9 | 0.3 | 98.4 | 1.1 |
| Pattern-blind predictor | 86.8 | 86.7 | 0.1 | 86.8 | 0.0 |
Table 8: One set of predictions per predictor on PATTERN’s official test split, scored under the three conventions in use. Differences are computed before rounding.
| Seed 0 | Seed 1 | |
|---|---|---|
| Best-validation epoch, of 200 | 159 | 184 |
| Training time (hours on one T4) | 19.7 | 19.6 |
| Official test split | 88.8 (88.6 to 89.1) | 88.3 (88.0 to 88.5) |
| Training patterns, new graphs | 88.6 (88.1 to 89.1) | 87.8 (87.3 to 88.3) |
| Fresh patterns | 84.5 (84.0 to 85.0) | 84.3 (83.8 to 84.8) |
| Training patterns minus official | 0.2 ( 0.7 to 0.4) | 0.5 ( 1.0 to 0.1) |
Table 9: CKGConv trained with its released recipe, by seed. Class-weighted accuracy pooled over each set, with 95% bootstrap intervals over graphs. Differences are computed before rounding.
| Test set | Pattern size | Seed 0 | Seed 1 | Pattern-blind predictor |
|---|---|---|---|---|
| Official test split | under 14 nodes | 86.4 | 85.8 | 85.7 |
| Official test split | 14 nodes or more | 88.9 | 88.4 | 86.5 |
| Training patterns, new graphs | under 14 nodes | 86.1 | 86.1 | 86.0 |
| Training patterns, new graphs | 14 nodes or more | 88.8 | 87.8 | 86.2 |
| Fresh patterns | under 14 nodes | 84.2 | 84.5 | 86.1 |
| Fresh patterns | 14 nodes or more | 84.1 | 83.8 | 85.8 |
Table 10: CKGConv’s scores by pattern size, and the pattern-blind predictor’s on the same graphs. Pooled over a whole set, class-weighted accuracy can lie above its value on both halves: pattern nodes come mostly from large patterns, host nodes from every graph.
| Seed 0 | Seed 1 | |
| Test graphs whose pattern was held out | 84.2 (83.6 to 84.7) | 84.5 (83.9 to 85.0) |
| Test graphs whose pattern was kept | 89.2 (88.9 to 89.5) | 89.3 (89.0 to 89.6) |
| Held out minus kept | 5.0 ( 5.7 to 4.4) | 4.8 ( 5.5 to 4.2) |
| The pattern-blind predictor on the held-out graphs | 86.9 (86.4 to 87.4) | |
| The pattern-blind predictor on the kept graphs | 86.7 (86.4 to 87.0) | |
| Held out minus the pattern-blind predictor, paired | 2.8 ( 3.1 to 2.4) | 2.5 ( 2.8 to 2.1) |
Table 11: CKGConv trained without 20 of the 100 patterns, on the official test graphs split by whether their pattern was held out (400 graphs, 100 of them with patterns under 14 nodes) or kept (1,600). Class-weighted accuracy pooled over each set, with 95% bootstrap intervals over graphs where given. CKGConv trained on all 100 patterns is the model of Table 9 with the same seed, and the pattern-blind predictor is the same for both seeds. Differences are computed before rounding.
| Quantity | Seed 0 | Seed 1 |
| Official accuracy (%) | 87.20 | 87.20 |
| Fresh accuracy (%) | 85.80 | 85.67 |
| Familiar accuracy (%) | 87.00 | 87.04 |
| Official minus fresh (points) | 1.40 | 1.53 |
| Fresh minus pattern-blind (points) | 0.60 | 0.73 |
| Graph-paired 95% interval | [ 0.80, 0.39] | [ 0.94, 0.52] |
Table 12: GRIT with the released model and optimization configuration, and the batching and continuation changes described in Appendix D . Accuracy is class-weighted and pooled over each set. Differences from the pattern-blind predictor use the same fresh graphs, with paired graph-bootstrap intervals. The AUC expectation holds for one seed only. Checkpoint epochs count from zero.
| Quantity | Seed 0 | Seed 1 |
| Fresh-trained official accuracy (%) | 86.77 | 86.77 |
| Fresh-trained fresh accuracy (%) | 86.45 | 86.46 |
| Fresh-trained familiar accuracy (%) | 86.55 | 86.62 |
| Full-recipe official accuracy (%) | 88.81 | 88.28 |
| Full-recipe fresh accuracy (%) | 84.48 | 84.34 |
| Full-recipe familiar accuracy (%) | 88.64 | 87.82 |
Table 13: CKGConv trained and validated with a new pattern per graph, compared with the same-seed models trained on the released patterns. Accuracy is pooled class-weighted accuracy. Gains and differences from the pattern-blind predictor have paired graph-bootstrap intervals. Both the training and the validation distributions change; checkpoint epochs count from zero.
| Model | Seed | Pattern recall difference | Host recall difference |
|---|---|---|---|
| GRIT | 0 | 0.57 [ 0.97, 0.14] | 0.64 [ 0.86, 0.41] |
| GRIT | 1 | 0.85 [ 1.25, 0.42] | 0.62 [ 0.85, 0.38] |
| CKGConv | 0 | 1.13 [0.87, 1.39] | 1.05 [ 1.19, 0.90] |
| CKGConv | 1 | 1.87 [1.60, 2.17] | 1.76 [ 1.91, 1.62] |
Table 14: Exploratory differences in class recall from the pattern-blind predictor on the same fresh graphs, in percentage points, with paired graph-bootstrap intervals. The CKGConv rows use the new training and validation patterns. The intervals compare fitted models and do not cover variation across training seeds.
| Criterion | Check | Required | Observed | Met |
|---|---|---|---|---|
| The code | full-width strip against enumeration | within | yes | |
| single strips against their conditionals | within | yes | ||
| Valid chains | chains that reach a valid board | at least 6 of 8 | 4 ran; all 4 reached one | no |
| Split | mine count | at most 1.01 | 1.013 | no |
| AUC trace | at most 1.01 | 1.40 | no | |
| Effective sample size | mine count | at least 400 | 24,432 | yes |
Table 15: The six convergence criteria we set before the run of the strip sampler, one check per row, as set and as observed; a criterion is met when all of its checks are. Differences are computed before rounding.
| Model | arXiv | Venue | Metric as named | PATTERN |
|---|---|---|---|---|
| GRIT 2-QiRW (QPE) | 2406.06547 | ICML | Weighted Accuracy | 90.17 † |
| PPGT | 2504.12588 | TMLR | Weighted Accuracy | 89.75 |
| GRASS | 2407.05649 | ICLR | Accuracy | 89.18 |
| CKGConv | 2404.13604 | ICML | Weighted Accuracy | 88.66 |
| GSM++ (MoT) | 2411.15671 | ICML | Accuracy | 87.93 |
| GraphMinNet | 2502.00282 | Neural Networks | Accuracy | 87.55 |
Table 16: Papers from 2024 to 2026 that report PATTERN results, in decreasing order of the selected number, with an arXiv identifier where available. The source is the main-text PATTERN classification comparison, or the appendix comparison, marked (appendix), when the main text has none. For method papers we select the best result of a proposed method; for papers marked (benchmark), which compare existing models or encodings, the best result of a model the authors evaluated. Isolated ablation results are excluded. Numbers are in percent, rounded half up to two decimals from the printed value. The metric is the name the paper gives PATTERN’s metric in its results or dataset table, the one that states a weighting where the two differ; where no table names it, or only the text states a weighting, it is the text’s name, marked (text). The paper marked (other) reports active learning under a labeling budget: its number is AULC, not accuracy after full training, and it is not counted among the method papers. † QPE prints this number for its best model and a higher one for its own run of GRIT; we could not open the code it links to check its metric, so its numbers are left out of the comparison of Table 2 (Appendix C ).
| Experiment or check | What was predicted | What would have refuted it | What happened |
|---|---|---|---|
| Section 3 : PATTERN’s test split | |||
| Recurrence in the released files | about 100 distinct training patterns (90 to 110), each in about 100 training graphs; at least 95% of validation and test graphs carry one | fewer than 90% of test graphs carrying a training pattern | held: exactly 100 patterns, each in 100 training graphs; 100.0% of validation and test graphs carry one |
| The largest-match rule on 200 random test graphs, then on all 2,000 | 97 to 100 (batch mean), written before the first sample; for the whole split, 96.0 to 97.5, written before the full run | a score below 90 | 96.7 on the sample (pooled 96.5) and 96.9 on the whole split (pooled 97.3), both under the first range; the whole split’s score lies inside the second; not refuted |
| The likelihood matcher on all test graphs | 98.8 (98.0 to 99.6), batch mean; exact recovery at least 93% (largest match about 84%), and at least 75% for patterns under 14 nodes | the matcher below 97.5 | held on the scores: 99.0 (pooled 99.2); exact recovery 93.7% (largest match 86.6%); missed for small patterns, at 74.8% |
| Section 4 : the pattern-blind level and patterns no model has seen | |||
| The pattern-blind predictor on the test split | 86.5 (85.0 to 88.2), batch mean. Separately, that the best model published up to 2023, GRIT at 87.2, lies within one point of this score, so that the score lies from 86.2 to 88.2 | for the second statement only: a score below 86.2 | held: 86.4 (pooled 86.4), inside both ranges; the sampler with the size prior, which the paper reports, scores 86.7 (pooled 86.8) |
Table 17: What we expected from each experiment and check, in the order of the paper, and what happened. Batch mean and pooled are the two ways of computing class-weighted accuracy (Appendix C ); where a prediction named the batch mean, its row says so. AUC is in percent.
More in this topic
Benchmark Contamination
Benchmark contamination is the presence of evaluation data or cues in training or development, which can inflate scores and weaken claims of generalization.
See all 49 papers