Universal Sample-Based Embeddings of Higher-Order Objects

Radovan Haluška; Christopher Solinas; Nathan R. Sturtevant; Martin Schmid; Michael Bowling

cs.LGOct 14, 2026

Universal Sample-Based Embeddings of Higher-Order Objects

Authors: Radovan Haluška, Christopher Solinas, Nathan R. Sturtevant, Martin Schmid, Michael Bowling

Organizations: Charles University, Prague · University of Alberta · Alberta Machine Intelligence Institute (Amii) · EquiLibre Technologies, Inc.

Abstract

Neural networks operate naturally on finite-dimensional vectors, yet many problems require reasoning about higher-order objects such as distributions, functions, and graphs. Such objects capture relationships within a ground set and rarely admit an explicit compact representation. Moreover, we typically want a compact representation not of a single higher-order object but of an entire family of such objects, including members unseen during training. While a few approaches exist for particular object types, none is designed to work across object types. We propose a unified framework that represents any higher-order object as a fixed-size embedding vector learned over a family of such objects. An encoder maps an object, seen only through a finite set of samples, to an embedding, and a domain-specific decoder supplies the training signal by modeling the object conditioned on that embedding. Since the encoder works only from samples, one architecture serves different object types. Because the decoder is a model already suited to single instances of the object type, the framework inherits that model's native operations and inductive biases. We instantiate the framework on distributions, functions, and graphs. In each case we show that a single trained model represents an entire family and generalizes to members unseen during training. Furthermore, we demonstrate the downstream utility of these embeddings: interpolating distributions in embedding space, solving differential equations in embedding space, and predicting unseen edges from a graph's embedding.

Figures & tables

Figure 1: General training procedure. Each sample set

\mathcal{D}

is split into two halves,

\mathcal{D}_{\text{embed}}

and

\mathcal{D}_{\text{train}}

. The embedder

\mathcal{E}_{\phi}

maps

\mathcal{D}_{\text{embed}}

to an embedding vector

\theta

that conditions the object-specific decoder

g_{\psi}

. The held-out half

\mathcal{D}_{\text{train}}

is then used to score

g_{\psi}

and compute the training loss

\mathcal{L}

. Finally,

\mathcal{L}

is differentiated with respect to the trainable parameters of both networks and backpropagated.

Figure 2: Distribution embeddings on the donut family. (a) Three donuts (color-coded), each embedded from a small conditioning set of samples drawn from it (crosses); the decoder then generates new samples (dots) from that embedding, recovering the donut the conditioning points were drawn from. (b) The density the same embedding induces over the plane, evaluated on a grid. From only a handful of conditioning samples, the embedding captures each ring’s center, radius, and width.

Model	NLL $\downarrow$	SWD $\downarrow$	Center $\downarrow$	Radius $\downarrow$	Width $\downarrow$
FM	$1.218\pm 0.017$	$0.070\pm 0.001$	$0.059\pm 0.003$	$0.012\pm 0.000$	$0.066\pm 0.011$
NF	$1.212\pm 0.012$	$0.063\pm 0.000$	$0.048\pm 0.001$	$0.013\pm 0.001$	$0.061\pm 0.007$
KDE	$2.011$	$0.152$	$0.082$	$0.048$	$0.627$
Parametric fit	$1.332$	$0.077$	$0.081$	$0.010$	$0.049$

Table 1: Donut reconstruction at

n{=}256

embedding samples (mean

\pm

SEM over 3 seeds, lower is better). FM and NF are our flow matching and normalizing flow decoders on a shared Set Transformer embedder; KDE and a method-of-moments parametric fit are per-sample-set baselines.

Figure 3: Function embeddings on the Lotka–Volterra family. (a) Reconstruction of three solutions (color-coded) from observed samples: the embedder maps the samples (dots) to an embedding from which the decoder reconstructs the prey trajectory

x(t)

(solid line, ground truth shaded). (b) Prediction of the same solutions from their parameters

(x_{0},y_{0},\alpha,\beta,\gamma,\delta)

alone, with no samples, via the operator. (c) The predicted trajectories drawn as a predator–prey phase portrait. The same decoder recovers the trajectory whether the embedding is formed from samples or predicted from parameters.

Family	Embedding	Operator
Polynomials	$0.0143\pm 0.0011$	$0.0185\pm 0.0022$
Degree 0	$0.0016\pm 0.0001$	$0.0133\pm 0.0013$
Degree 1	$0.0103\pm 0.0004$	$0.0151\pm 0.0014$
Degree 2	$0.0201\pm 0.0019$	$0.0271\pm 0.0039$
Degree 3	$0.0251\pm 0.0024$	—
Damped Oscillator	$0.0173\pm 0.0029$	$0.0056\pm 0.0002$

Table 2: Relative

L^{2}

reconstruction error by family (mean

\pm

SEM over 3 seeds). Embedding : the function is reconstructed from an embedding inferred from a sample set; Operator : from an embedding produced by an operator. For polynomials, operator error is reported on the output derivative and indexed by its degree. A degree-three derivative is omitted, as it would require a degree-four input.

Dim.	Embedding	Operator
$2$	$0.0173\pm 0.0029$	$0.0056\pm 0.0002$
$4$	$0.0178\pm 0.0010$	$0.0070\pm 0.0006$
$8$	$0.0279\pm 0.0003$	$0.0122\pm 0.0006$
$16$	$0.0404\pm 0.0042$	$0.0199\pm 0.0013$
$32$	$0.0645\pm 0.0015$	$0.0348\pm 0.0015$

Table 3: Damped-oscillator dimension scaling: relative

L^{2}

reconstruction error vs. state dimension (mean

\pm

SEM over 3 seeds; columns as in Table 2 ).

Figure 4: Graph embeddings on the random-trees family. For three sampled trees (color-coded), we show (a) the ground-truth graphs, (b) the predicted geometry (each pair of vertices joined by a line that grows thicker and darker as their representation distance shrinks), and (c) the reconstruction from thresholding those distances with the medians network. The tree structure is already visible in the geometry alone.

Family	ROC-AUC	Vtx-degree F1	Medians F1
Random trees ( $n=10$ )	$0.993\pm 0.001$	$0.983\pm 0.001$	$0.922\pm 0.005$
Random trees ( $n=20$ )	$0.953\pm 0.006$	$0.781\pm 0.028$	$0.449\pm 0.064$
Scale-free ( $n=10$ )	$0.980\pm 0.001$	$0.958\pm 0.002$	$0.793\pm 0.002$
Scale-free ( $n=20$ )	$0.847\pm 0.010$	$0.613\pm 0.022$	$0.331\pm 0.008$

Table 4: Geometry and reconstruction quality by family (mean

\pm

SEM over 3 seeds). Geometry ROC-AUC is the threshold-free edge/non-edge separation; the two F1 columns are degree-oracle and learned-head (medians) reconstruction.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Densities decoded along a straight line between the embeddings of two donuts ( $\alpha=0$ and $\alpha=1$ ). The interior points are interpolated in embedding space, not embedded from samples. Color tracks $\alpha$ .

Figure 6: Embedding arithmetic across horizontal and vertical position. Blue panels are embedded from samples; the orange panel decodes the prediction $\hat{D}=z_{B}+z_{C}-z_{A}$ , which matches $D$ to within the sampling-noise floor.

Figure 7: The same identity across vertical position and width. The prediction (orange) again matches $D$ to within the noise floor.

Figure 8: Where additivity breaks: raising horizontal position, radius, and width together. The prediction $\hat{E}=z_{B}+z_{C}+z_{D}-2z_{A}$ (orange) is visibly thinner than $E$ , because a donut’s thickness is the product of radius and width.

	NLL		SWD
$n$	FM	NF	FM	NF
$16$	$1.819\pm 0.202$	$1.939\pm 0.108$	$0.098\pm 0.001$	$0.091\pm 0.002$
$32$	$1.311\pm 0.028$	$1.319\pm 0.013$	$0.079\pm 0.001$	$0.073\pm 0.000$
$64$	$1.223\pm 0.009$	$1.215\pm 0.006$	$0.072\pm 0.000$	$0.066\pm 0.000$
$128$	$1.214\pm 0.015$	$1.205\pm 0.010$	$0.070\pm 0.001$	$0.064\pm 0.000$
$256$	$1.218\pm 0.017$	$1.212\pm 0.012$	$0.070\pm 0.001$	$0.063\pm 0.000$

Appendix

Table 5: Donut reconstruction quality: held-out NLL and sliced-Wasserstein distance (mean

\pm

SEM over seeds, lower is better) versus embedding sample count

n

	Center		Radius		Width
$n$	FM	NF	FM	NF	FM	NF
$16$	$0.100\pm 0.001$	$0.090\pm 0.003$	$0.035\pm 0.002$	$0.036\pm 0.002$	$0.080\pm 0.006$	$0.074\pm 0.004$
$32$	$0.074\pm 0.001$	$0.063\pm 0.000$	$0.022\pm 0.000$	$0.023\pm 0.002$	$0.055\pm 0.002$	$0.046\pm 0.001$
$64$	$0.064\pm 0.001$	$0.053\pm 0.000$	$0.016\pm 0.000$	$0.017\pm 0.001$	$0.053\pm 0.008$	$0.045\pm 0.005$
$128$	$0.059\pm 0.002$	$0.049\pm 0.000$	$0.013\pm 0.000$	$0.014\pm 0.001$	$0.059\pm 0.010$	$0.053\pm 0.007$
$256$	$0.059\pm 0.003$	$0.048\pm 0.001$	$0.012\pm 0.000$	$0.013\pm 0.001$	$0.066\pm 0.011$	$0.061\pm 0.007$

Appendix

Table 6: Donut parameter-recovery error: center, radius, and width of the reconstructed ring (mean

\pm

SEM over seeds, lower is better) versus embedding sample count

n

Encoder (multi-stage ISAB)
Hyperparameter	Value
Embedding size	$16$
Stages	$3$
Attention heads	$4$
Inducing points	$16$
Feed-forward hidden size	$128$

Appendix

Table 7: Distribution-embedding architectures, hyperparameters, and evaluation protocol on the donut family.

Family	State dim.	Domain and parameter ranges
Polynomials	$1$	$x\in[-4,4]$ , $c_{k}\in[-5,5]$ , $d\in[0,3]$
Damped oscillator	$2M$	$t\in[0,10]$ , $x_{0},v_{0}\in[-2,2]$ , $\gamma\in[0.3,2.5]$ , $\omega~\in~[0.5,3]$ , $k\in[0.5,1.5]$
Lotka–Volterra	$2$	$t\in[0,10]$ , $x_{0},y_{0}\in[0.5,2]$ , $\alpha,\beta,\gamma,\delta\in[0.8,1.5]$

Appendix

Table 8: Function families: state dimension, and domain and parameter ranges.

Encoder (multi-stage ISAB)
Hyperparameter	Value
Embedding size	$64$
Stages	$2$
Attention heads	$4$
Inducing points	$16$
Feed-forward / projection hidden size	$64$

Appendix

Table 9: Function-embedding architecture and training hyperparameters, shared by every family and both encoders.

Figure 9: Function embeddings on the polynomial family. Three polynomials of degree one, two, and three (color-coded). (a) Reconstruction from observed samples: the embedder maps the samples (dots) to an embedding from which the decoder reconstructs $f$ (solid line, ground truth shaded). (b) The differential operator maps each function’s embedding to that of its derivative, $D(\mathrm{embed}(f))\approx\mathrm{embed}(f^{\prime})$ , which the decoder then expands into $f^{\prime}$ (solid line, ground truth shaded). A single embedding thus supports both reconstructing a function and differentiating it.

Family	RMSE	Rel $L^{2}$	Rel Max
Polynomials	$0.8596\pm 0.0165$	$0.0162\pm 0.0007$	$0.0184\pm 0.0008$
degree 0	$0.0041\pm 0.0002$	$0.0010\pm 0.0000$	$0.0017\pm 0.0001$
degree 1	$0.0660\pm 0.0014$	$0.0081\pm 0.0004$	$0.0100\pm 0.0004$
degree 2	$0.4033\pm 0.0246$	$0.0182\pm 0.0013$	$0.0202\pm 0.0014$
degree 3	$2.9649\pm 0.0461$	$0.0376\pm 0.0011$	$0.0415\pm 0.0015$
Damped oscillator	$0.0114\pm 0.0010$	$0.0178\pm 0.0007$	$0.0191\pm 0.0006$

Appendix

Table 10: Embedding reconstruction error (inferred from a sample set) by family, Set Transformer encoder, concatenation fusion, mean

\pm

SEM over 3 seeds. Rel Max is the relative max-abs error.

Family	RMSE	Rel $L^{2}$	Rel Max
Polynomials	$0.5224\pm 0.0682$	$0.0143\pm 0.0011$	$0.0164\pm 0.0017$
degree 0	$0.0055\pm 0.0006$	$0.0016\pm 0.0001$	$0.0030\pm 0.0001$
degree 1	$0.0740\pm 0.0048$	$0.0103\pm 0.0004$	$0.0135\pm 0.0011$
degree 2	$0.3786\pm 0.0375$	$0.0201\pm 0.0019$	$0.0221\pm 0.0027$
degree 3	$1.6315\pm 0.2362$	$0.0251\pm 0.0024$	$0.0270\pm 0.0032$
Damped oscillator	$0.0108\pm 0.0018$	$0.0173\pm 0.0029$	$0.0216\pm 0.0034$

Appendix

Table 11: Embedding reconstruction error (inferred from a sample set) by family, Set Transformer encoder, dot-product fusion, mean

\pm

SEM over 3 seeds. Rel Max is the relative max-abs error.

Encoder (multi-stage ISAB)
Hyperparameter	Trees $10$	Trees $20$	SF $10$	SF $20$	Gridworld
Embedding size	$128$
Stages	$2$
Attention heads	$4$
Inducing points	$32$
Feed-forward hidden size	$64$

Appendix

Table 12: Graph-embedding architectures and hyperparameters per family.

Metric	Trees $10$	Trees $20$	SF $10$	SF $20$	Gridworld
Geometry
ROC-AUC	$0.993\pm 0.001$	$0.953\pm 0.006$	$0.980\pm 0.001$	$0.847\pm 0.010$	$0.958\pm 0.002$
AP	$0.978\pm 0.002$	$0.793\pm 0.016$	$0.956\pm 0.001$	$0.625\pm 0.018$	$0.907\pm 0.001$
Reconstruction (degree oracle)
Precision	$0.984\pm 0.001$	$0.806\pm 0.023$	$0.949\pm 0.002$	$0.649\pm 0.021$	—
Recall	$0.982\pm 0.001$	$0.758\pm 0.032$	$0.967\pm 0.002$	$0.580\pm 0.023$	—

Appendix

Table 13: Graph-embedding quality per family, all metrics (mean

\pm

SEM over 3 seeds).

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 813 papers

Sep 29, 2026

TopoEmbedX: A General Framework for Representation Learning on Topological Domains

Topological structures such as simplicial complexes, hypergraphs, and cell complexes extend standard graph models by modeling higher-order relationships. These structures appear in many modern datasets and require specialized methods for generating meaningful embeddings. In this paper, we introduce TopoEmbedX, a unified framework for embedding a wide range of topological domains into Euclidean spaces. The package brings together several existing topological embedding algorithms---DeepCell, Cell2Vec, CellDiff2Vec, HOLE, and HOGLEE---and introduces five new algorithms: ComplexNetMF, ComplexRep, ComplexRandNE, ComplexWalklets, and ComplexHeat. These algorithms extend well-known graph embedding techniques to higher-order settings using the augmented Hasse graph of a topological domain. TopoEmbedX provides a clear, consistent, and easy-to-use framework for topological representation learning. Experiments show that the embeddings generated by TopoEmbedX support tasks such as classification and regression across multidimensional data.

Florian Frantzen, Ibrahem AlJabea, Ines Henriques-Cadby +3

RWTH Aachen University · Louisiana State University · The University of Manchester +3

Topological Representation Learning Graph Embedding Hypergraph Learning

Ask PDF

Sep 28, 2026

GeoGAE: Scalable Graph-Level Autoencoding via Hyperball Cloud Representations

Embedding structured objects into Euclidean spaces has enabled a wide range of successful machine learning applications. Such objects include words, documents, image patches, time series, and graph nodes. In contrast, embedding entire graphs remains a challenging problem. Existing methods either sustain the original order of the graph nodes or match the output nodes to the input ones, both of which create scalability issues. In this work, we propose a graph representation as a cloud of hyperballs, which allows us to define a specific, typically unique, node ordering. Based on this representation, we propose GeoGAE, an autoencoder, in which the Transformer encoder translates a hyperball cloud into a graph-level embedding, and the Transformer decoder translates the graph-level embedding back into the graph. This formulation enables the model to capture both the global graph structure and local relational patterns. We evaluate our method on multiple graph datasets, spanning various domains. The results demonstrate effectiveness of our method in encoding and reconstructing graphs from their embeddings.

Radosław Nowak, Anna Bielawska, Bogusz Stefańczyk +2

Institute of Theoretical and Applied Informatics Polish Academy of Sciences · IDEAS Research Institute · Faculty of Mathematics, Informatics and Mechanics Warsaw University of Technology

Graph Neural Networks Graph Representation Learning Graph Embedding

Ask PDF

Sep 8, 2026

Topology-induced Operators Reveal Complementary Graph Representations without Training

Graph representation learning has largely focused on designing increasingly sophisticated models to transform graph topology into vector representations, or embeddings. However, the extent to which embedding quality depends on model learning, rather than on the underlying topological transformations, remains unclear. Here, we show that informative embeddings can be derived without complicated model design and gradient-based training. Propagating random features through implicit hierarchical structures induced by random walks and anonymous walks yields embeddings that capture node proximity and structural role, respectively. These two training-free embeddings preserve complementary aspects of graph organization and perform competitively with classic and recent methods across various node-, edge-, and graph-level tasks. They often require substantially less computation, resulting in a favorable quality-efficiency trade-off. Combining the two types of embeddings further improves inference quality of some tasks compared with using either embedding type alone. Our results suggest that informative graph embeddings can arise from carefully chosen topological transformations before any learning operation is applied.

Meng Qin, Jinqiang Cui, Hongwei Zheng +2

Department of Strategic & Advanced Interdisciplinary Research, Pengcheng Laboratory (PCL), Shenzhen, Guangdong, China. · Beijing Academy of Blockchain and Edge Computing, Beijing, China. · LMIB, NLSDE, and School of Artificial Intelligence, Beihang University, Beijing, China. +1

Graph Representation Learning Graph Embedding Topological Representation Learning

Ask PDF

Abstract

Representation Learning

Explore similar work

TopoEmbedX: A General Framework for Representation Learning on Topological Domains

GeoGAE: Scalable Graph-Level Autoencoding via Hyperball Cloud Representations

Topology-induced Operators Reveal Complementary Graph Representations without Training