Persistence Positional Encoding: A Minimal Transformer Recipe for Persistence Diagrams

Eunwoo Heo; Ganghyun Kim; Jaejun Yoo

cs.LGOct 13, 2026

Persistence Positional Encoding: A Minimal Transformer Recipe for Persistence Diagrams

Authors: Eunwoo Heo, Ganghyun Kim, Jaejun Yoo

Organizations: Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology

Abstract

Persistence diagrams (PDs) represent multiscale topological information as multisets of birth-death pairs $(b,d)$ . How best to use them with Transformers, however, remains unclear. Connecting PDs to Transformers requires choosing how topological information is tokenized and exposed to self-attention. Prior methods use specialized designs for tokenization, aggregation, Transformer blocks, or attention, but are hard to compare since reported results rarely share PD inputs, splits, or model-selection rules. We instead confine the diagram-specific design to point representation, mapping each birth-death pair through fixed random Fourier features while leaving the Transformer backbone unchanged, and call this recipe Persistence Positional Encoding (PPE). In matched controls on eight TU graph benchmarks (TU8), changing only point representation shifts macro accuracy by 5.49 to 6.47 percentage points (pp), against 0.47 to 1.45 pp from changing the downstream module, identifying point representation as the dominant design variable tested. We further reevaluate vectorizations, kernels, learned representations, and Transformer-based models under one protocol with shared PD inputs, outer folds, and validation-only selection. Among all evaluated PD methods and downstream configurations, PPE achieves the highest mean on most benchmarks, including TU8, ORBIT5K, and MCB-C. Finally, we prove that grid averaging leaves a reconstruction error at any resolution, whereas a Transformer with fixed random Fourier frequencies is almost surely universal. PPE thus offers a minimal, Transformer-native baseline for persistence diagrams, and a common protocol for future comparison.

Figures & tables

Figure 1: A schematic overview of PPE . (a) Filtration converts topological data, e.g., graphs, point clouds, or other structured domains, into persistence diagrams. (b) Each PD point $x_{i}=(b_{i},d_{i})$ is encoded by fixed random Fourier features, linearly projected, and augmented with a learnable homology-dimension embedding ( $H_{0}$ or $H_{1}$ ). The resulting unordered token sequence is processed by a Transformer encoder without positional encoding. (c) The CLS representation drives downstream tasks, such as molecular/graph classification and shape/object classification.

(a) Transformer-based comparison
Method	ORBIT5K	MUTAG	PROTEINS	COX2	DHFR	IMDB-B	IMDB-M	NCI1	NCI109	TU8
MST	81.56 $\pm$ 5.53	78.99 $\pm$ 7.90	68.10 $\pm$ 2.89	77.53 $\pm$ 1.56	60.34 $\pm$ 3.53	67.78 $\pm$ 4.34	39.28 $\pm$ 3.09	65.44 $\pm$ 1.28	59.97 $\pm$ 1.29	64.68
Persformer	76.90 $\pm$ 13.58 ∗	82.48 $\pm$ 6.23	71.23 $\pm$ 3.18	78.16 $\pm$ 0.58	60.83 $\pm$ 0.65	67.44 $\pm$ 2.75	39.63 $\pm$ 3.73	64.10 $\pm$ 3.21	64.64 $\pm$ 1.66	66.06
xPerT	83.86 $\pm$ 1.24	83.38 $\pm$ 5.75	72.29 $\pm$ 3.59	77.51 $\pm$ 3.05	75.80 $\pm$ 3.27	66.48 $\pm$ 4.13	46.15 $\pm$ 3.24	66.36 $\pm$ 1.14	66.56 $\pm$ 1.02	69.32
PPE (Ours)	90.68 $\pm$ 0.72	87.44 $\pm$ 3.40	72.96 $\pm$ 4.61	78.70 $\pm$ 3.47	80.03 $\pm$ 3.18	70.24 $\pm$ 3.33	46.60 $\pm$ 3.45	75.66 $\pm$ 1.47	74.09 $\pm$ 1.66	73.22

Table 1: Benchmark and controlled-interface results. (a) reports the unified Transformer-based comparison, (b) additional larger-scale evaluations, and (c) a matched

2\times 2

control, whose

\Delta

column reports the signed difference (Transformer aggregation minus pooled MLP aggregation) in percentage points. Gray shading and bold mark the highest mean, while underline marks the second-highest mean within each comparison.

Method	Learning model	TU8
Transformer-based PD methods
PPE	Transformer	73.22
xPerT	Transformer	69.32
Persformer	Transformer	66.06
MST	Transformer	64.68
Learned PD representations

Table 2: TU8 comparison across PD method families. Each row is a method–downstream configuration. TU8 is the unweighted mean over the eight graph datasets. Gray shading and bold mark the highest mean, and underline marks the second-highest mean.

Varied factor	Point representation	Downstream module	MUTAG	PROTEINS	COX2	DHFR	IMDB-B	IMDB-M	NCI1	NCI109	TU8
Both	Point MLP	Pooled MLP (mean+sum)	81.38	72.58	78.08	60.32	64.28	43.56	62.46	62.84	65.69
	Point MLP	Transformer	82.69	71.48	78.16	60.64	68.64	45.68	64.56	65.29	67.14
	Fixed RFF	Pooled MLP (mean+sum)	85.74	71.61	77.46	79.85	69.00	46.43	73.64	73.58	72.16
	Fixed RFF	Transformer	85.38	71.72	78.30	79.56	69.88	46.47	75.67	74.07	72.63
Point representation	Raw coordinates	DeepSets	80.43	71.82	78.16	60.85	63.84	39.45	62.41	62.42	64.92
Point representation	Fixed RFF	DeepSets	86.08	72.09	77.55	79.37	68.56	46.67	74.08	73.22	72.20

Table 3: Point-representation and downstream-module controls. Accuracy (%) under the matched TU8 protocol. Shared configurations are repeated across controls so that each comparison is self-contained. TU8 is the unweighted mean over the eight graph datasets. Gray shading and bold mark the highest mean, while underline marks the second-highest mean within each control.

Panel of Figure 2 — Figure 2: Coordinate-encoder ablation and validation convergence. ( 2(a) ) Accuracy (%) for fixed Fourier versus learned point-MLP encoders in the same PPE backbone on TU graph benchmarks, where bold marks the better encoder per row. ( 2(b) ) Validation accuracy versus epoch on ORBIT5K under the unified protocol. Each line is the mean over the ten cross-validation folds and each band spans those folds. Each label gives a method’s peak mean accuracy and the epoch at which it occurs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	PI + Transformer	RFF + Transformer	PI $-$ RFF
MUTAG	81.70	85.38	$-$ 3.68
PROTEINS	73.22	71.72	+1.50
COX2	78.16	78.30	$-$ 0.14
DHFR	60.50	79.56	$-$ 19.06
IMDB-B	50.30	69.88	$-$ 19.58
IMDB-M	43.11	46.47	$-$ 3.36

Appendix

Table A1: Per-dataset PI versus RFF comparison with a matched Transformer backbone. Accuracy (%) over ten folds and five seeds per fold. Both representations use the same Transformer architecture, folds, and validation-only selection rule. Bold marks the higher mean for each dataset.

PersLay variant	ORBIT5K	TU8
Image	81.36	71.10
Line/PEQ	36.49	66.31
Landscape	32.97	64.05
Exponential	27.65	65.94

Appendix

Table A2: PersLay variant comparison. All variants use the same PyTorch PersLay implementation, prepared persistence diagrams, outer folds, training protocol, and validation-only selection. TU8 is the unweighted mean over the eight graph datasets. Gray shading and bold mark the highest mean, while underline marks the second-highest mean.

Representation	MUTAG	PROTEINS	COX2	DHFR	IMDB-B	IMDB-M	NCI1	NCI109	TU8
Common MLP head from the persistence-image configuration
Persistence image	82.76 $\pm$ 6.24	73.47 $\pm$ 3.66	77.39 $\pm$ 1.44	61.16 $\pm$ 0.90	65.72 $\pm$ 3.44	43.27 $\pm$ 5.17	66.01 $\pm$ 2.55	65.57 $\pm$ 1.77	66.92
ATOL	86.54 $\pm$ 4.28	71.66 $\pm$ 3.54	77.65 $\pm$ 0.79	72.20 $\pm$ 5.78	67.32 $\pm$ 6.46	44.03 $\pm$ 3.98	64.32 $\pm$ 1.30	66.00 $\pm$ 1.57	68.71
Persistence landscape	80.95 $\pm$ 10.87	66.45 $\pm$ 2.08	78.08 $\pm$ 0.55	71.14 $\pm$ 5.06	64.96 $\pm$ 3.06	40.99 $\pm$ 2.85	61.14 $\pm$ 1.41	60.73 $\pm$ 2.22	65.56
Persistence silhouette	66.30 $\pm$ 1.34	67.53 $\pm$ 2.46	78.08 $\pm$ 0.76	60.90 $\pm$ 0.40	65.44 $\pm$ 3.51	41.29 $\pm$ 2.85	56.76 $\pm$ 2.19	56.98 $\pm$ 2.56	61.66
Betti curve	85.19 $\pm$ 5.70	68.71 $\pm$ 2.51	77.35 $\pm$ 0.89	78.18 $\pm$ 2.93	65.12 $\pm$ 2.97	40.92 $\pm$ 3.37	69.49 $\pm$ 2.01	70.10 $\pm$ 1.35	69.38

Appendix

Table A3: Controlled comparison across TU8 datasets. Rows are grouped by downstream model or model family, with matched representations sharing the same downstream model and evaluation protocol. Per-dataset entries report mean test accuracy and population standard deviation across ten outer folds; TU8 is the unweighted mean over the eight datasets. Kernel construction uses training diagrams only, and SVM selection uses validation data only. Gray shading and bold mark the highest mean in each column across all configurations, and underlining marks the second highest.

ogbg-molhiv ROC-AUC
Method	Score
PersLay	70.92 $\pm$ 0.30
Persistence image + MLP	70.38 $\pm$ 0.42
PPE	69.32 $\pm$ 1.51
xPerT	68.27 $\pm$ 0.71
ATOL + random forest	67.71 $\pm$ 0.74

Appendix

Table A4: Full larger-scale comparison. The ogbg-molhiv comparison reports ROC-AUC and the MCB-C comparison reports balanced accuracy. Values are pooled seed-run mean and population standard deviation. Models marked † use the original graph or point cloud rather than a persistence diagram and are included only as task-level references, and are excluded from the ranking. The graph references are GIN ( Xu et al., 2019 ) and GCN ( Kipf & Welling, 2017 ) , and the point-cloud reference follows PointNet ( Qi et al., 2017 ) . Gray shading and bold mark the highest mean in each column, while underline marks the second-highest.

Figure A1: ORBIT5K coordinate-noise sweep. Mean test accuracy as a function of point-replacement noise. The fixed Fourier encoder wins every noise level. The gap is largest at zero noise (+25.8 pp) and persists across the sweep. Averaged over the seven noisy settings, Fourier reaches 53.1% versus the learned MLP’s 46.6%.

Panel of Figure 11 — Figure A2: Representative SDXL-Turbo outputs at stage (i) of the donut-dataset construction pipeline. Left shows a plain donut sample. Right shows a sprinkle donut sample. These 2D renderings are subsequently lifted to 3D meshes by TripoSR and uniformly subsampled to 2048-point clouds.

Figure A3: xPerT tokenization of plain (left) versus sprinkle (right) donut persistence diagrams. Each row pairs a plain-donut sample with a sprinkle-donut sample whose xPerT tokens activate the same grid patches. The hard $R\times R$ grid binning maps every PD point in a cell to the cell centre, so sub-cell coordinate shifts—which carry the plain-vs-sprinkle distinction—are averaged away. The two classes therefore become indistinguishable at the tokenization stage, the empirical analogue of the grid-collapse mechanism formalized in Theorem 1 .

Method	Val Acc (%)	Test Acc (%)
MST	64.12 $\pm$ 7.83	50.91 $\pm$ 10.34
Persformer	55.18 $\pm$ 9.18	52.32 $\pm$ 11.19
xPerT	62.82 $\pm$ 9.80	54.11 $\pm$ 12.62
PPE (Ours)	77.29 $\pm$ 11.71	71.97 $\pm$ 11.36

Appendix

Table A5: Plain-vs-Sprinkle Donut classification. Accuracy (%) over ten folds and five seeds per fold. For each method, we report the configuration selected by validation accuracy during random hyperparameter search. Gray shading and bold mark the highest mean in each column, while underline marks the second-highest.

Component	Configuration
Fourier Feature Encoder
Number of frequencies ( $m$ )	128
Frequency scale ( $\sigma$ )	10.0
Output dimension	256 ( $m$ cosine and $m$ sine components)
Homology-Dimension Embedding
Number of dimensions	2 ( $H_{0}$ , $H_{1}$ )

Appendix

Table A6: PPE Transformer hyperparameters. Architecture and training defaults for the unified ORBIT5K and TU experiments. Per-dataset learning rate, weight decay, and dropout are tuned via random grid search on the validation split; batch size, warmup, total epochs, and early-stopping patience are fixed per dataset family (full protocol in Appendix G ).

Dataset	Wall-clock time
MUTAG	3m 28s
DHFR	7m 30s
COX2	7m 45s
IMDB-B	9m 46s
PROTEINS	22m 4s
IMDB-M	24m 50s

Appendix

Table A7: Representative logged wall-clock times for the final selected PPE configuration. Each entry covers 5 random seeds

\times

10-fold cross-validation and was measured on a single RTX 3090 or RTX 4090 GPU (24 GB). These measurements demonstrate reproduction on 24 GB consumer hardware.

Cap per homology component	Balanced accuracy
512	65.82 $\pm$ 3.59
1,024 (primary)	65.79 $\pm$ 4.27
2,048	65.10 $\pm$ 4.07

Appendix

Table A8: MCB-C token-cap sensitivity. The cap applies to the computed persistence diagram, bounding the number of points taken from each homology dimension and keeping those of largest lifetime

d-b

. MCB-C uses

H_{0}

H_{1}

, and

H_{2}

, so at cap

C

a diagram contributes at most

3C

points. Values are balanced accuracy over 50 seed-runs.

Hyperparameter	Range	Distribution
learning rate	$[10^{-5},\,3\times 10^{-3}]$	log-uniform
weight decay	$[10^{-6},\,10^{-2}]$	log-uniform
dropout	$[0,\,0.5]$	uniform

Appendix

Table A9: Shared optimization-hyperparameter grid. Identical across all six methods.

Method	Hyperparameter	Values
PPE a	d_model	$\{64,\,128,\,192,\,256\}$
	nhead	$\{2,\,4,\,8\}$
	num_layers	$\{2,\,3,\,4,\,5\}$
	num_frequencies	$\{64,\,128,\,256,\,384\}$
	ff_scale	$[1,\,30]$ , log-uniform
	use_betti_embedding	$\{\text{true},\,\text{false}\}$

Appendix

Table A10: Per-method architecture-hyperparameter grids. Method-specific search space drawn under the shared random-search protocol.

Dataset family	Batch	Warmup epochs	Total epochs	ES patience
TU graphs (MUTAG, PROTEINS, COX2, DHFR, IMDB-B/M, NCI1/109)	64	50	300	20
ORBIT5K	16	50	300	20
Synthetic donut	32	5	400	60

Appendix

Table A11: Fixed per-dataset training defaults. Selected once per dataset family and not part of the search; same defaults apply to all methods on a given dataset family.

Method	ORBIT5K	MUTAG	PROTEINS	COX2	DHFR	IMDB-B	IMDB-M	NCI1	NCI109
PersLay	87.7 $\pm$ 1.0	89.8	74.8	80.9	80.3	71.2	48.8	73.5	69.5
Persformer	91.2 $\pm$ 0.8	89.9 $\pm$ 2.1	—	—	—	—	—	—	—
xPerT	87.0 $\pm$ 0.7	91.0 $\pm$ 5.2	75.7 $\pm$ 3.8	84.4 $\pm$ 3.5	81.9 $\pm$ 2.9	72.6 $\pm$ 3.4	50.0 $\pm$ 2.0	—	—
MST (Ord., no clustering)	—	89.25 $\pm$ 0.77	72.92 $\pm$ 0.47	81.20 $\pm$ 1.04	74.12 $\pm$ 0.37	70.76 $\pm$ 0.47	45.44 $\pm$ 0.35	69.12 $\pm$ 0.17	66.38 $\pm$ 0.40

Appendix

Table A12: Prior-paper accuracy (%). As reported in each method’s original publication, using each paper’s original pipeline. The MST row uses ordinary persistence diagrams without clustering ( Wang et al., 2024 ) ; dashes denote unreported results.

Explore similar work

Jul 29, 2026

Voronoi Histograms for Adaptive Vectorization of Expected Persistence Diagrams

Persistence Diagram (PD) is known to capture point cloud topology effectively, but its computation has high time complexity. Expected Persistence Diagram (EPD) has been developed to reduce the time cost by studying the topology of multiple subsets of a point cloud and it serves as a distribution of topological features. Existing EPD vectorizations often rely on predefined point transformations, such as Gaussian or landscape functions. We study an alternative discretization based on Voronoi histograms, which trades smooth functional approximation for adaptive partition-based counting. We propose to use Voronoi Diagram-based histogram as the vectorization of EPD, without imposing an explicit smooth point transformation model. Under stated separation and normalization conditions, we establish stability bounds and characterize when the histogram representation preserves Wasserstein-scale variation. We demonstrate the effectiveness of our proposed representation on real-world datasets which have significant topological features for classification and dimensionality reduction tasks.

Kaifeng Zhang, Kai Ming Ting

School of Artificial Intelligence Nanjing University, China

Persistent Homology

Ask PDF

Apr 24, 2026

FILTR: Extracting Topological Features from Pretrained 3D Models

Recent advances in pretraining 3D point cloud encoders (e.g., Point-BERT, Point-MAE) have produced powerful models, whose abilities are typically evaluated on geometric or semantic tasks. At the same time, topological descriptors have been shown to provide informative summaries of a shape's multiscale structure. In this paper we pose the question whether topological information can be derived from features produced by 3D encoders. To address this question, we first introduce DONUT, a synthetic benchmark with controlled topological complexity, and propose FILTR (Filtration Transformer), a learnable framework to predict persistence diagrams directly from frozen encoders. FILTR adapts a transformer decoder to treat diagram generation as a set prediction task. Our analysis on DONUT reveals that existing encoders retain only limited global topological signals, yet FILTR successfully leverages information produced by these encoders to approximate persistence diagrams. Our approach enables, for the first time, data-driven extraction of persistence diagrams from raw point clouds through an efficient learnable feed-forward mechanism.

Louis Martinez, Maks Ovsjanikov

LIX, ´Ecole Polytechnique, IP Paris

Point Cloud Learning 3D Representation Learning Persistent Homology Topological Representation Learning Topological Data Analysis

Ask PDF

Aug 7, 2026

Transformer Geometry Observatory TGO-IV: Developmental Topology Observatory

Transformers have had a profound impact on the world of language processing and computer vision. As efforts to answer the million-dollar question of ``How does a Transformer learn?" have been increasing, existing interpretability studies primarily analyze representations at isolated layers or the network as a whole, while the developmental evolution of individual representations and its manifolds across transformer layers remains underexplored. With this work, we aim at providing a comprehensive analysis of the evolution of representations as the representation point cloud transforms across the layers; thereby attempting to isolate layers or establish a trend which comes closer to justifying how and when raw input representations evolve into task-relevant feature representations. Thus, Transformer Geometry Observatory-TGO-IV introduces a topological framework for analysing the evolution of Transformer representations through the lens of Persistent Homology. Rather than studying local geometric properties alone, TGO-IV constructs Vietoris--Rips simplicial complexes from token-level representation point clouds and investigates the evolution of their persistent topological signatures across Transformer layers. The proposed framework comprises complementary topological observatories including Persistence Diagrams, Barcode Diagrams, Betti Curves, Persistence Landscapes, Bottleneck Distance, and Wasserstein Distance, enabling a comprehensive analysis of how the global topology of representation point clouds develops throughout the forward pass.

Kaustubh Kapil, Kishor P. Upla

Neuromorphic Intelligence Research Collective · Department of Electronics Engineering · Sardar Vallabhai National Institute of Technology (SVNIT), Surat, India

Transformer Interpretability Neural Representation Geometry

Ask PDF