Optimal Transport (OT) provides a principled framework for finding the minimum cost coupling between probability distributions. Classical solvers remain the gold standard for this task. At the same time, OT is an ideal testbed for studying whether learning-based amortized models can internalize the structure of OT type problems. Here, ground-truth solutions are available and marginal constraints are explicitly defined. Also, exact discrete solutions have well-characterized sparsity. We describe ToR-OT, a transformer-based model that learns to predict discrete OT plans through autoregressive token regression. We reframe OT into a structured sequence generation task. ToR-OT decodes sparse transport plans as sequences instead of solving an optimization task at test time. We focus on three properties: (a) Zero-shot generalization: pretraining on a large synthetic corpus of OT instances allows a single model to generalize to unseen distribution pairs and heterogeneous cost metrics without retraining, (b) Sparse primal decoding: by exploiting the inherent sparsity in exact discrete OT solutions, the decoder predicts only nonzero entries (avoids dense coupling outputs), (c) Amortized inference: on low-dimensional discrete problems, ToR-OT gives accurate approximate plans and compares favorably with per-instance neural OT baselines. Overall, ToR-OT is not yet a replacement for classical solvers, but a step toward understanding how autoregressive transformers can learn structure of constrained optimization problems like OT.
Figures & tables
Figure 1: Toy discrete OT instance showing the source measure μ , target measure ν , cost C , and an optimal plan π⋆∈R+n×m
Figure 2: ToR-OT architecture. Motivated by Song and Bahri (2025) , we view Optimal Transport as a sparse token-based multi-regression problem. Given a pair of source and target distributions, we train a Transformer to predict the nonzero values of the OT plan π∗ as a sequence of digit tokens.
Figure 3: Test set (top row) results and OOD (bottom row) results. (left) ToR-OT vs NeuralOT, Meta OT and UNOT. On all six costs, our model performs favorably. (right) a sample of the evaluation set, along with its ground truth and predicted OT plan. In the case of the test set, both source and target distributions are randomly sampled, while in the OOD case, they correspond to well-defined distributions, like Beta and Exponential.
Cost
ℓ2
ℓ21/2
ℓ22
ℓ23
Canberra
Gaussian
Test set
88%
91%
95%
93%
91%
64%
OOD set
60%
65%
75%
67%
69%
69%
Table 1: Reduction of ToR-OT’s mean MAPE relative to Meta OT, the strongest baseline, for every cost ( n=30 ).
Figure 4: (A) Violation of marginal constraints in ToR-OT across rows and columns. (B) Relative improvement of ToR-OT’s MAPE over Meta OT, the strongest baseline, under ℓ22 for 100 of the 144 pairs of OOD one-dimensional distributions (all 144 are in Appendix F ). The difference is always positive and reaches up to 99% . X-axis labels: 1: Beta, 2: Laplace, 3: Student’s t, 4: Normal, 5: Lognormal, 6: Cauchy, 7: Triangular, 8: Exponential, 9: Weibull, 10: Gamma. (C) The same for all 49 pairs of OOD two-dimensional distributions. The difference is always positive and reaches up to 99% . X-axis labels: 1’: Circles, 2’: 8 Gaussians, 3’: Moons, 4’: S curve, 5’: Standard Normal, 6’: Swiss roll, 7’: Uniform square. (D) Mean MAPE of every method on the 2D test and OOD sets.
Figure 5: ToR-OT, NeuralOT and Meta OT on a 2D OOD pair (Swiss roll → Standard Normal under ℓ22 ): the 6×6 histograms, the cost matrix and the plans over the 36 flattened bins, on one color scale. The exact same formulation as 1D is used. The only difference is that the 2D model is trained on 2D histograms, with each bin embedded from its 2D coordinate; the six costs are the same as in 1D.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7
Metric
Canonical ToR-OT
ToR-OT + Sinkhorn
Max Marginal Violation
1.15×10−2
1.7×10−4
Prediction Error (MAPE)
1.52×10−1
7.5×10−2
Appendix
Table 3: Improved feasibility and accuracy with a few Sinkhorn iterations on the test set.
Metric
Canonical (Single Pass)
Symmetric (Two Pass Avg.)
Max Marginal Violation
0.0272
0.0218
Prediction Error (MAPE)
0.3025
0.2538
Total Inference Time
1.133 s
2.299 s
Appendix
Table 4: Effect of symmetric two-pass averaging on feasibility, accuracy, and inference time.
n=30
n=100
n=200
Set
Cost
ToR-OT
Meta OT
ToR-OT
Meta OT
ToR-OT
Meta OT
Test
L2
0.009
0.072
0.027
0.436
0.302
0.701
L21/2
0.016
0.180
0.037
0.282
0.375
0.395
L22
0.010
0.221
0.031
0.612
1.770
0.804
L23
0.010
0.148
0.031
0.581
1.767
0.733
Canberra
0.013
0.154
0.032
0.733
0.371
0.945
Appendix
Table 5: Mean MAPE at three support sizes, each method against the solution of the problem it solves (lower is better; bold: the better of the two). ToR-OT is the single 1D model trained on n≤100 ; Meta OT is retrained at every n (mean over three training seeds). Test: 1,000 pairs per cost; OOD: the 144 pairs of the 12 named distributions, binned at each n .
Figure 7: Checkpoints of the 1D model (every 25,000 steps) vs training rows seen: median MAPE against POT’s plan, mean support recall and median largest marginal residual, on 1,000 test pairs and the 864 named-distribution pairs of the corpus’ OOD split.
Figure 8: Small testbed ( n≤30 , the same training budget for every run): median MAPE against POT’s plan vs the fraction of training instances (left) and the number of parameters (right). Line: mean over three training seeds; error bar: standard deviation; dots: the seeds.
Configuration
Parameters
Test set
OOD set
Base recipe
width 256 (base)
9.8M
0.036 ± 0.003
0.113 ± 0.029
Model size
width 64
0.7M
0.068 ± 0.005
0.365 ± 0.022
width 128
2.6M
0.045 ± 0.003
0.222 ± 0.030
width 384
21.7M
0.036 ± 0.002
0.105 ± 0.012
width 512
39.3M
0.034 ± 0.001
0.097 ± 0.015
Training corpus
10% of the instances
9.8M
0.038 ± 0.003
0.102 ± 0.011
Appendix
Table 6: Ablations on the small testbed: median MAPE against POT’s plan, mean ± standard deviation over three training seeds. Each row changes one aspect of the base recipe.
Figure 9: The 12 distributions of the 1D OOD set, numbered as in the improvement heatmaps (Fig. 48 ).
Figure 10: Cost L2 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 11: POT (exact): plans of the 144 OOD pairs, cost L2 .
Figure 12: ToR-OT (ours): plans of the 144 OOD pairs, cost L2 .
Figure 14: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost L2 .
Figure 16: Cost L21/2 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 17: POT (exact): plans of the 144 OOD pairs, cost L21/2 .
Figure 18: ToR-OT (ours): plans of the 144 OOD pairs, cost L21/2 .
Figure 20: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost L21/2 .
Figure 22: Cost L22 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 23: POT (exact): plans of the 144 OOD pairs, cost L22 .
Figure 24: ToR-OT (ours): plans of the 144 OOD pairs, cost L22 .
Figure 26: Meta OT (entropic, ε=0.001 ): plans of the 144 OOD pairs, cost L22 .
Figure 28: Cost L23 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 29: POT (exact): plans of the 144 OOD pairs, cost L23 .
Figure 30: ToR-OT (ours): plans of the 144 OOD pairs, cost L23 .
Figure 32: Meta OT (entropic, ε=0.001 ): plans of the 144 OOD pairs, cost L23 .
Figure 34: Cost Canberra: test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 35: POT (exact): plans of the 144 OOD pairs, cost Canberra.
Figure 36: ToR-OT (ours): plans of the 144 OOD pairs, cost Canberra.
Figure 38: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost Canberra.
Figure 40: Cost Gaussian: test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 41: POT (exact): plans of the 144 OOD pairs, cost Gaussian.
Figure 42: ToR-OT (ours): plans of the 144 OOD pairs, cost Gaussian.
Figure 44: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost Gaussian.
Figure 46: All six costs, OOD set (144 pairs per cost): the five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale).
Figure 47: All six costs, test set (1,000 pairs per cost), as Fig. 46 .
Figure 48: Relative MAPE improvement of ToR-OT over Meta OT, 100(MAPEMeta OT−MAPEToR-OT)/MAPEMeta OT , on each of the 144 OOD pairs (row: source; column: target; distributions numbered as in the first column), for each cost; each method is scored against the solution of its own problem. Green: ToR-OT better; purple: Meta OT better; the color scale is clipped at ±100% . Panel titles: the number of pairs on which ToR-OT is better and the median improvement.