Optimal Transport (OT) provides a principled framework for finding the minimum cost coupling between probability distributions. Classical solvers remain the gold standard for this task. At the same time, OT is an ideal testbed for studying whether learning-based amortized models can internalize the structure of OT type problems. Here, ground-truth solutions are available and marginal constraints are explicitly defined. Also, exact discrete solutions have well-characterized sparsity. We describe ToR-OT, a transformer-based model that learns to predict discrete OT plans through autoregressive token regression. We reframe OT into a structured sequence generation task. ToR-OT decodes sparse transport plans as sequences instead of solving an optimization task at test time. We focus on three properties: (a) Zero-shot generalization: pretraining on a large synthetic corpus of OT instances allows a single model to generalize to unseen distribution pairs and heterogeneous cost metrics without retraining, (b) Sparse primal decoding: by exploiting the inherent sparsity in exact discrete OT solutions, the decoder predicts only nonzero entries (avoids dense coupling outputs), (c) Amortized inference: on low-dimensional discrete problems, ToR-OT gives accurate approximate plans and compares favorably with per-instance neural OT baselines. Overall, ToR-OT is not yet a replacement for classical solvers, but a step toward understanding how autoregressive transformers can learn structure of constrained optimization problems like OT.
Figures & tables
Figure 1: Toy discrete OT instance showing the source measure μ , target measure ν , cost C , and an optimal plan π⋆∈R+n×m
Figure 2: ToR-OT architecture. Motivated by Song and Bahri (2025) , we view Optimal Transport as a sparse token-based multi-regression problem. Given a pair of source and target distributions, we train a Transformer to predict the nonzero values of the OT plan π∗ as a sequence of digit tokens.
Figure 3: Test set (top row) results and OOD (bottom row) results. (left) ToR-OT vs NeuralOT, Meta OT and UNOT. On all six costs, our model performs favorably. (right) a sample of the evaluation set, along with its ground truth and predicted OT plan. In the case of the test set, both source and target distributions are randomly sampled, while in the OOD case, they correspond to well-defined distributions, like Beta and Exponential.
Cost
ℓ2
ℓ21/2
ℓ22
ℓ23
Canberra
Gaussian
Test set
88%
91%
95%
93%
91%
64%
OOD set
60%
65%
75%
67%
69%
69%
Table 1: Reduction of ToR-OT’s mean MAPE relative to Meta OT, the strongest baseline, for every cost ( n=30 ).
Figure 4: (A) Violation of marginal constraints in ToR-OT across rows and columns. (B) Relative improvement of ToR-OT’s MAPE over Meta OT, the strongest baseline, under ℓ22 for 100 of the 144 pairs of OOD one-dimensional distributions (all 144 are in Appendix F ). The difference is always positive and reaches up to 99% . X-axis labels: 1: Beta, 2: Laplace, 3: Student’s t, 4: Normal, 5: Lognormal, 6: Cauchy, 7: Triangular, 8: Exponential, 9: Weibull, 10: Gamma. (C) The same for all 49 pairs of OOD two-dimensional distributions. The difference is always positive and reaches up to 99% . X-axis labels: 1’: Circles, 2’: 8 Gaussians, 3’: Moons, 4’: S curve, 5’: Standard Normal, 6’: Swiss roll, 7’: Uniform square. (D) Mean MAPE of every method on the 2D test and OOD sets.
Figure 5: ToR-OT, NeuralOT and Meta OT on a 2D OOD pair (Swiss roll → Standard Normal under ℓ22 ): the 6×6 histograms, the cost matrix and the plans over the 36 flattened bins, on one color scale. The exact same formulation as 1D is used. The only difference is that the 2D model is trained on 2D histograms, with each bin embedded from its 2D coordinate; the six costs are the same as in 1D.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7
Metric
Canonical ToR-OT
ToR-OT + Sinkhorn
Max Marginal Violation
1.15×10−2
1.7×10−4
Prediction Error (MAPE)
1.52×10−1
7.5×10−2
Appendix
Table 3: Improved feasibility and accuracy with a few Sinkhorn iterations on the test set.
Metric
Canonical (Single Pass)
Symmetric (Two Pass Avg.)
Max Marginal Violation
0.0272
0.0218
Prediction Error (MAPE)
0.3025
0.2538
Total Inference Time
1.133 s
2.299 s
Appendix
Table 4: Effect of symmetric two-pass averaging on feasibility, accuracy, and inference time.
n=30
n=100
n=200
Set
Cost
ToR-OT
Meta OT
ToR-OT
Meta OT
ToR-OT
Meta OT
Test
L2
0.009
0.072
0.027
0.436
0.302
0.701
L21/2
0.016
0.180
0.037
0.282
0.375
0.395
L22
0.010
0.221
0.031
0.612
1.770
0.804
L23
0.010
0.148
0.031
0.581
1.767
0.733
Canberra
0.013
0.154
0.032
0.733
0.371
0.945
Appendix
Table 5: Mean MAPE at three support sizes, each method against the solution of the problem it solves (lower is better; bold: the better of the two). ToR-OT is the single 1D model trained on n≤100 ; Meta OT is retrained at every n (mean over three training seeds). Test: 1,000 pairs per cost; OOD: the 144 pairs of the 12 named distributions, binned at each n .
Figure 7: Checkpoints of the 1D model (every 25,000 steps) vs training rows seen: median MAPE against POT’s plan, mean support recall and median largest marginal residual, on 1,000 test pairs and the 864 named-distribution pairs of the corpus’ OOD split.
Figure 8: Small testbed ( n≤30 , the same training budget for every run): median MAPE against POT’s plan vs the fraction of training instances (left) and the number of parameters (right). Line: mean over three training seeds; error bar: standard deviation; dots: the seeds.
Configuration
Parameters
Test set
OOD set
Base recipe
width 256 (base)
9.8M
0.036 ± 0.003
0.113 ± 0.029
Model size
width 64
0.7M
0.068 ± 0.005
0.365 ± 0.022
width 128
2.6M
0.045 ± 0.003
0.222 ± 0.030
width 384
21.7M
0.036 ± 0.002
0.105 ± 0.012
width 512
39.3M
0.034 ± 0.001
0.097 ± 0.015
Training corpus
10% of the instances
9.8M
0.038 ± 0.003
0.102 ± 0.011
Appendix
Table 6: Ablations on the small testbed: median MAPE against POT’s plan, mean ± standard deviation over three training seeds. Each row changes one aspect of the base recipe.
Figure 9: The 12 distributions of the 1D OOD set, numbered as in the improvement heatmaps (Fig. 48 ).
Figure 10: Cost L2 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 11: POT (exact): plans of the 144 OOD pairs, cost L2 .
Figure 12: ToR-OT (ours): plans of the 144 OOD pairs, cost L2 .
Figure 14: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost L2 .
Figure 16: Cost L21/2 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 17: POT (exact): plans of the 144 OOD pairs, cost L21/2 .
Figure 18: ToR-OT (ours): plans of the 144 OOD pairs, cost L21/2 .
Figure 20: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost L21/2 .
Figure 22: Cost L22 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 23: POT (exact): plans of the 144 OOD pairs, cost L22 .
Figure 24: ToR-OT (ours): plans of the 144 OOD pairs, cost L22 .
Figure 26: Meta OT (entropic, ε=0.001 ): plans of the 144 OOD pairs, cost L22 .
Figure 28: Cost L23 : test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 29: POT (exact): plans of the 144 OOD pairs, cost L23 .
Figure 30: ToR-OT (ours): plans of the 144 OOD pairs, cost L23 .
Figure 32: Meta OT (entropic, ε=0.001 ): plans of the 144 OOD pairs, cost L23 .
Figure 34: Cost Canberra: test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 35: POT (exact): plans of the 144 OOD pairs, cost Canberra.
Figure 36: ToR-OT (ours): plans of the 144 OOD pairs, cost Canberra.
Figure 38: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost Canberra.
Figure 40: Cost Gaussian: test set (top, 1,000 pairs) and OOD set (bottom, 144 pairs). Five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale), and time per instance on one A100 GPU (solid: one instance; hatched: per instance, batched). NeuralOT optimizes each instance separately and its runs shared GPUs, so its times are not comparable (n/a).
Figure 41: POT (exact): plans of the 144 OOD pairs, cost Gaussian.
Figure 42: ToR-OT (ours): plans of the 144 OOD pairs, cost Gaussian.
Figure 44: Meta OT (entropic, ε=0.01 ): plans of the 144 OOD pairs, cost Gaussian.
Figure 46: All six costs, OOD set (144 pairs per cost): the five plan metrics, each method against the solution of its own problem (box: quartiles; line: median; whiskers: 1.5 IQR; outliers not shown; log scale).
Figure 47: All six costs, test set (1,000 pairs per cost), as Fig. 46 .
Figure 48: Relative MAPE improvement of ToR-OT over Meta OT, 100(MAPEMeta OT−MAPEToR-OT)/MAPEMeta OT , on each of the 144 OOD pairs (row: source; column: target; distributions numbered as in the first column), for each cost; each method is scored against the solution of its own problem. Green: ToR-OT better; purple: Meta OT better; the color scale is clipped at ±100% . Panel titles: the number of pairs on which ToR-OT is better and the median improvement.
We propose a novel amortized optimization method for predicting optimal transport (OT) plans across multiple pairs of measures by leveraging Kantorovich potentials derived from sliced OT. We introduce two amortization strategies: regression-based amortization (RA-OT) and objective-based amortization (OA-OT). In RA-OT, we formulate a functional regression model that treats Kantorovich potentials from the original OT problem as responses and those obtained from sliced OT as predictors, and estimate these models via least-squares methods. In OA-OT, we estimate the parameters of the functional model by optimizing the Kantorovich dual objective. In both approaches, the predicted OT plan is subsequently recovered from the estimated potentials. As amortized OT methods, both RA-OT and OA-OT enable efficient solutions to repeated OT problems across different measure pairs by reusing information learned from prior instances to rapidly approximate new solutions. Moreover, by exploiting the structure provided by sliced OT, the proposed models are more parsimonious, independent of specific structures of the measures, such as the number of atoms in the discrete case, while achieving high accuracy. We demonstrate the effectiveness of our approaches on tasks including MNIST digit transport, color transfer, supply-demand transportation on spherical data, and mini-batch OT conditional flow matching.
Minh-Phuc Truong, Khai Nguyen
Hanoi University of Science and Technology · The University of Texas at Austin
Optimal Transport (OT) provides a principled framework for learning transformations between probability distributions from unpaired samples. In many applications, however, a single transformation must map several source distributions to a common target distribution. For example, image restoration might require handling different types of degradation without knowing the degradation of each input at inference time. Simple approaches of pooling the source distributions only encourage alignment with the target at the aggregate level and may leave individual sources misaligned. In our paper, we consider the simultaneous OT problem which formalizes the task of learning a shared transport map that minimizes the average transport cost while aligning each source distribution with a prescribed target. We propose a neural method for solving the simultaneous OT problem by learning a shared transport map that minimizes the average transport cost while aligning each source distribution with a prescribed target. We derive a max-min formulation for learning this map. We illustrate its application to image restoration, where a single model handles multiple degradation types using a common collection of clean target images.
Milena Gazdieva, Kirill Sokolov, Jiawei Chen +2
Applied AI Institute AXXX Moscow, Russia · Lomonosov Moscow State University Moscow, Russia · MIRIAI Moscow, Russia
A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optimal destination. We measure both on Pythia-160m and Pythia-410m, with an exact assignment between consecutive layer clouds, a measured sampling floor, calibration on couplings known to be optimal, and a split of the cost into the common shift of the cloud and the token-specific moves. At the last layer, both models move their tokens where the optimal-transport map sends them, at the optimal cost for Pythia-410m and slightly above it for Pythia-160m. At the first layer they do not. In between, single layers can be judged on cost at only two of ten transitions, and blocks of several layers move the cloud at close to the optimal cost. The agreement at the last layer is much weaker at initialisation (0.64 against 0.86) and grows with training.