Benchmarking the performance of complex systems such as rail networks, renewable generation assets and national economies is central to transport planning, regulation and macroeconomic analysis. Classical frontier methods, notably Data Envelopment Analysis (DEA) and Stochastic Frontier Analysis (SFA), estimate an efficient frontier in the observed input-output space and define efficiency as distance to this frontier, but rely on restrictive assumptions on the production set and only indirectly address heterogeneity and scale effects. We propose Geometric Manifold Analysis (GeMA), a latent manifold frontier framework implemented via a productivity-manifold variational autoencoder (ProMan-VAE). Instead of specifying a frontier function in the observed space, GeMA represents the production set as the boundary of a low-dimensional manifold embedded in the joint input-output space. A split-head encoder learns latent variables that capture technological structure and operational inefficiency. Efficiency is evaluated with respect to the learned manifold, endogenous peer groups arise as clusters in latent technology space, a quotient construction supports scale-invariant benchmarking, and a local certification radius, derived from the decoder Jacobian and a Lipschitz bound, quantifies the geometric robustness of efficiency scores. We validate GeMA on synthetic data with non-convex frontiers, heterogeneous technologies and scale bias, and on four real-world case studies: global urban rail systems (COMET), British rail operators (ORR), national economies (Penn World Table) and a high-frequency wind-farm dataset. Across these domains GeMA behaves comparably to established methods when classical assumptions hold, and provides additional insight in settings with pronounced heterogeneity, non-convexity or size-related bias.
Figures & tables
Figure 1: ProMan-VAE architecture. A split-head encoder maps observed inputs and outputs (x,y) to latent technology (z) and inefficiency (u) . The decoder Gθ reconstructs the frontier output from (x,z) , and the realised output is obtained by scaling the frontier with exp(−u) . A monotonicity regulariser encourages the decoder to be weakly increasing in each input dimension.
Percentile
p0
p25
p50
p75
p95
Rcert
0.105
0.227
0.293
0.340
0.392
Table 1: ORR: certification radius percentiles across operator–year observations.
Figure 2: (Left) Distribution of certification radii Rcert(xi) across operator–year observations. A visible left tail indicates cases where performance assessments rely on locally ill-conditioned frontier geometry. (Right) Model-based score versus certification radius. The dashed lines indicate the top decile of the score and the bottom quartile of Rcert , highlighting observations with high point performance but weak robustness guarantees.
Figure 3: Wind farms: learned frontiers vs specification-based toy curves for two representative farms (Farm 2 and 5). Scatter points show observed capacity factor versus hub-height wind speed; red curves show the GeMA frontier (normalised by its 95th percentile per farm); blue curves show simple toy turbine curves constructed from manufacturer cut-in, rated and cut-out speeds. Orange markers indicate specification-based operating points. (Left) Frontier learned without using the turbine threshold (cut-in/rated speeds) as inputs. (Right) Frontier learned with the turbine threshold (cut-in/rated speeds) included as additional inputs, yielding a tighter alignment with the theoretical plateau and a more pronounced decline near cut-out.
Figure 4: COMET: UMAP embeddings of latent technology vectors zi learned by GeMA , coloured by GMM cluster assignment ( k=4 ). (a) Two-dimensional UMAP projection of the latent technology space used for visualising peer groups. (b) Three-dimensional UMAP view of the same latent manifold, illustrating its overall geometry; the projected points in (a) correspond to this surface. (c) DMUs whose outputs lie on the estimated frontier, shown as the subset of points mapped to the boundary of the learned manifold in latent space. These boundary points represent units operating at the highest efficiency given their latent technology, i.e. on the manifold frontier.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Scenario A
Scenario B
Scenario C
Frontier error / Rank corr.
ARI / Rank corr.
Size corr. / Rank corr.
DEA (VRS)
- / 0.655 (0.042)
0.782 (0.024)
- / 0.801 (0.023)
SFA (Translog)
0.216 (0.009) / 0.727 (0.041)
0.803 (0.017)
0.038 (0.022) / 0.818 (0.014)
FDH
- / 0.595 (0.039)
0.701 (0.028)
- / 0.726 (0.027)
CNLS
10.395 (38.537) / 0.350 (0.041)
0.506 (0.039)
0.154 (0.064) / 0.786 (0.056)
ML predictor (RF)
0.405 (0.719) / 0.780 (0.023)
0.767 (0.028)
0.093 (0.024) / 0.778 (0.023)
Appendix
Table 2: Synthetic experiments: Monte Carlo summary of frontier approximation error and rank correlation with true inefficiency across methods and scenarios. Means with standard deviations in parentheses.
Name
Description
Role
Transformation
Operator
Metro system identifier
Entity ID
–
Year
Calendar year
Time ID
–
Staff
Number of staff (FTE)
Input
log(1+x)
Capacity
Rolling stock seating capacity (car seats)
Input
log(1+x)
Stations
Number of stations
Input
log(1+x)
RouteLength
Network Length (km)
Input/Scale
log(1+x)
Appendix
Table 3: COMET variables: definitions and roles.
Name
Description
Role
Transformation
Operator
Train operating company ID
Entity ID
–
Year
Calendar year
Time ID
–
Labour
Number of staff (FTE)
Input
log(1+x)
Route
Route kilometres operated by operators
Input/Scale
log(1+x)
Stock
Train rollingstock in service
Input
log(1+x)
Plan
Numbers of trains planned
Input
log(1+x)
Appendix
Table 4: ORR variables: definitions and roles.
Name
Description
Role
Transformation
country
Country code / identifier
Entity ID
–
year
Calendar year
Time ID
–
rkna
Capital services at constant 2017 national prices
Input
log(rkna/emp)
hc
Human capital index
Input
log(hc)
emp
Number of persons engaged (millions)
Input
log(emp)
rgdpo
Output-side real GDP at chained PPPs (million)
Output
log(rgdpo/hours)
Appendix
Table 5: PWT variables: definitions and roles.
Name
Description
Role
Transformation
asset_num
Wind farm identifier
Entity ID
–
year
Calendar year
Time ID
–
ws_hub
Hub-height wind speed
Input
log(1+x)
temp_air
Air temperature
Input
log(1+x)
pressure_air
Air pressure
Input
log(1+x)
rel_humidity
Relative humidity
Input
log(1+x)
Appendix
Table 6: WF variables: definitions and roles.
Domain
K (tech dim)
Hidden dim
Epochs
Batch size
Learning rate
γu
WF
2
192
300
1024
5×10−4
1.0
PWT
4
192
200
256
3.8×10−3
0.213
ORR
4
128
200
1792
4.7×10−3
0.223
COMET
2
128
200
2048
4.4×10−3
0.201
Appendix
Table 7: ProMan-VAE hyperparameters by domain.
Domain
Activation
Spectral norm
Mono weight
Ranking loss weight
WF
SILU
No
0.0
(if used; else 0)
PWT
SILU
(default)
4.8×10−4
0.271
ORR
GELU
Yes
2.2×10−4
0.298
COMET
GELU
Yes
1.0×10−4
0.056
Appendix
Table 8: Additional architectural and regularisation choices.
Farm
Specification (capacity-weighted)
GeMA frontier (no thresholds)
GeMA frontier (with thresholds)
Site
vcspec
vrspec
vcospec
vcfit
vrfit
vcfit
vrfit
(m/s)
(m/s)
(m/s)
(m/s)
(m/s)
(m/s)
(m/s)
1
3.0
10.1
22.7
0.9
15.1
0.7
15.1
2
2.5
10.5
20.0
0.5
12.2
0.5
11.6
3
3.0
10.4
25.0
0.5
15.7
0.5
16.3
4
3.0
9.5
20.0
0.5
15.8
1.3
12.2
Appendix
Table 9: Wind farms: specification-based (capacity-weighted) vs frontier-fitted cut-in and rated wind speeds. Specification values vcspec,vrspec,vcospec are computed from manufacturer data; fitted values (vcfit,vrfit) are obtained by least-squares on the learned GeMA frontiers, with and without turbine thresholds as inputs. Fitted parameters are obtained post hoc from the learned frontiers and are not directly constrained during training.
Name
N Cap.
Model
Cap.
Ht
Rotor
#
Cut-in
Cut-out
RatedSpeed
Swt.Ar
PD1
PD2
(MW)
(kW)
(m)
(m)
(m/s)
(m/s)
(m/s)
(m 2 )
(W/m 2 )
(m 2 /kW)
F1
75
GW1500/87
1500
85.0
87.0
50
3.0
22
9.9
5890
254.7
3.9
F1
24
H93 L-2.0MW
2000
85.5
93.0
12
3.0
25
10.8
6792.9
294.4
3.4
F2
200
GW3000/140
3000
120.0
140.0
67
2.5
20
10.5
15747
193.9
5.2
F3
49.5
UP86-1500
1500
80.0
86.0
33
3.0
25
10.0
5809
258.2
3.9
F3
49.5
UP82-1500
1500
80.0
82.0
33
3.0
25
10.8
5384
278.6
3.6
Appendix
Table 10: Technical specifications of turbine configurations at each wind farm. Cut-in, rated and cut-out speeds are manufacturer-reported thresholds for individual turbine models.
Figure 5: Additional wind farm power-curve plots. For each farm, scatter points show observed capacity factor versus hub-height wind speed; red curves show the learned GeMA frontier (normalised by its 95th percentile); blue curves show specification-based “toy” turbine curves constructed from capacity-weighted cut-in, rated and cut-out speeds; orange markers highlight the specification-based operating points. The overall shape and operating points are consistent across farms, with high-wind declines indicating early curtailment where frequent high-wind events occur.
Cluster
# Test obs.
Route length
Stations
Capacity
Staff
Passenger-km
Car-km
A
27
120.7 (231.1)
100.5 (120.5)
40.0 (16.7)
3818 (7980)
3107 (5184)
94.4 (172.0)
B
28
186.4 (266.1)
87.0 (190.3)
43.2 (15.3)
5688 (11720)
5532 (13115)
140.1 (281.8)
C
16
141.2 (135.0)
104.0 (88.3)
40.5 (10.4)
5600 (6703)
6660 (14090)
128.9 (211.5)
D
13
115.1 (116.9)
91.0 (59.0)
44.4 (10.8)
4529 (11844)
6201 (7541)
114.7 (164.3)
Appendix
Table 11: COMET (test period 2018–2019): cluster-wise summaries of observable characteristics. Reported as median (interquartile range) over operator–year observations.
Cluster
# Test obs.
Route length (IQR)
DEA θ^ (IQR)
GeMA score (IQR)
RF proxy u (IQR)
A
27
120.7 (231.1)
1.000 (0.050)
-0.255 (0.319)
0.110 (0.283)
B
28
186.4 (266.1)
0.996 (0.073)
-0.447 (0.307)
0.075 (0.269)
C
16
141.2 (135.0)
0.995 (0.101)
-0.310 (0.505)
0.043 (0.185)
D
13
115.1 (116.9)
1.000 (0.043)
-0.555 (0.371)
0.005 (0.179)
Appendix
Table 12: COMET (test period 2018–2019): cluster-wise summaries of observable scale and baseline scores. Reported as median (IQR) over operator–year observations.
Cluster
# Test obs.
maxcpic (IQR)
Share maxcpic≥0.8
Share maxcpic≥0.9
A
27
0.534 (0.195)
11.1%
0.0%
B
28
0.611 (0.305)
21.4%
7.1%
C
16
0.759 (0.346)
43.8%
18.8%
D
13
0.797 (0.272)
46.2%
23.1%
Appendix
Table 13: COMET (test period 2018–2019): posterior cluster assignment confidence from the GMM on latent technology space. Reported as median (IQR) of maxcpic and the share of observations exceeding common confidence thresholds.
A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry. We repurpose and extend dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling, and pair them with finite-difference estimators that recover curvature, reach, and volume at near-ground-truth accuracy in a regime where general-purpose estimators are unreliable or difficult to deploy. The framework is intended as a controlled testbed, useful as a calibration environment for geometric estimators and a sandbox for probing theoretical assumptions. To illustrate its use, we present two application studies, namely assessing the scaling behavior of the bounds of Genovese et al. and Fefferman et al., and tracking the layer-wise geometry of a β-VAE, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. A reference implementation is available at https://github.com/koulakis/manifold-microscope.
Marios Koulakis, Constantin Seibold
Independent Researcher, Germany · Diagnostic and Interventional Radiology, University Clinic Heidelberg, Heidelberg, Germany.
Frontier-model leaderboards now rank systems on economic benchmarks, and those rankings inform what organisations buy and what regulators scrutinise. Whether such benchmarks measure a capability distinct from general test-taking is a question of construct validity that a structural test and a predictive test can answer in opposite ways. We show that they do on a hash-pinned snapshot of a frontier leaderboard with 421 model configurations across twelve benchmarks, four of them economic, of which 103 configurations carry all three sparsely scored economic benchmarks and 96 carry all twelve; four hypotheses and their thresholds were fixed before analysis, and every deviation from the plan is reported. The first factor of a three-factor extraction carries 74.5% of common variance and tracks release date (R^2 = 0.505), and date adjustment lowers its share by 14.9 points. Under the dimensionality rule fixed in advance the economic benchmarks form no factor of their own. A leave-one-benchmark-out test with factors re-estimated inside every fold nevertheless finds that a multi-factor representation predicts held-out economic scores better than a single general index (pooled Delta-MSE 0.037, 95% bootstrap interval [0.019, 0.055]) under the linear learners that fit best, an advantage that reverses for tree learners. Under the linear learners the same representation also predicts the eight other benchmarks better, so the battery carries predictive structure that one index misses and the economic benchmarks share it without forming a distinct factor. Construct validity should therefore be assessed by predictive tests alongside structural ones. We give a two-test protocol for benchmark builders and release the pinned data, the analysis plan and the code.
We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).
Laurent Cheret, Vincent Létourneau, Isar Nejadgholi +3
Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada