Organizations: National Institute of Information and Communications Technology, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan
Optimal Transport (OT) provides a principled framework for comparing and transforming probability distributions while preserving geometric structure. Recently, OT has gained significant attention in machine learning due to its ability to measure discrepancies between distributions, even when their supports do not overlap, making it effective for tasks such as generative modeling, domain adaptation, and transfer learning. Despite its success in fields such as computer vision and natural language processing, OT remains relatively underexplored in speech research. Speech signals present unique challenges, including temporal dynamics, speaker variability, noise, reverberation, and heterogeneous multimodal representations involving audio, text, and visual information. These factors often lead to distribution mismatches, where OT offers a natural framework for alignment and interpretation. This work aims to promote broader adoption of OT in speech processing by: (1) reviewing OT foundations through intuitive physical interpretations and highlighting connections to modern generative models; (2) presenting computational algorithms suitable for deep learning frameworks; and (3) demonstrating OT applications in cross-domain and cross-modal speech tasks, including speech enhancement, automatic speech recognition, language and speaker recognition, and audio spoof detection. We highlight OT's strong potential for addressing distributional variations in real-world speech applications.
Figures & tables
Figure 1: Push-forward measure in probability spaces.
Figure 2: Historical figures in the development of OT theory.
Figure 3: Landmark research activities.
Figure 4: Wide concept connections to OT.
Figure 5: Mass splitting in discrete Kantorovich OT.
Figure 6: OT as graph matching on nodes (a) and edges (b) as conventional definition of OT and GWOT, respectively.
Figure 7: (a) Static OT; (b) Dynamic OT, distribution changes along an efficient path.
Figure 8: Lagrangian (a) and Eularian (b) point of views for fluid dynamics.
Figure 9: Velocity as tangent directions along a particle trajectory (a) and on streamlines of population of particles (b).
Figure 10: Transport-based modeling with a flow map.
Figure 11: Unified concepts in flow mapping via ODEs.
Figure 12: Unified concepts with diffusion and flow matching models via SDEs/ODEs.
Figure 13: Transfer operator point of view for state dynamics.
Figure 14: Distance measure and transform between distributions.
Figure 15: Equal distance between different categories (no information of category geometric relationship).
Figure 16: Geodesic distance in Wasserstein space.
Figure 17: Domain adaptation: solid vertical line denotes the classification boundary trained on source domain data (a), while the dot vertical line represents the classification boundary on target domain data (b).
Figure 18: General computational framework with an OT embedding layer.
Figure 19: SE model framework enhanced with Phone-Fortified Perceptual Loss (PFPL) in a transformed latent space, WD: Wasserstein distance, MAE: mean absolute error.
Figure 20: Discriminator-constrained OT network for SE.
Figure 21: Multi-modal knowledge transfer learning for SE.
Figure 22: Conventional pipeline for ASR.
Figure 23: Cross-modal alignment and mapping for knowledge transfer learning based on OT coupling (acoustic-linguistic modalities).
Figure 24: Cross-modal knowledge transfer for ASR based on OT-matching.
Figure 25: Graph matching OT in Cross-modal knowledge transfer learning for ASR.
Figure 26: Segmental structure of speech in matching between acoustic and linguistic features based on OT.
Figure 27: Temporal order structure of speech in matching between acoustic and linguistic features based on OT.
Figure 28: Entropy regularization of coupling in matching between acoustic and linguistic features based on OT.
Figure 29: Alignment between acoustic and linguistic tokens (indexes of acoustic (Horizontal) and linguistic (Vertical) embedding sequences): (a) cosine similarity matrix; (b) α1=1.0 , α2=1.0 ; (c) α1=0.01 , α2=1.0 .
Figure 30: Cross-domain adaptation for LID based on a Siamese neural network architecture for source (left) and target (right) domain samples.
Figure 31: Visualization of feature representations via t-SNE in cross-domain LID before (a) and after (b) adaptation, lang{1,7}_tr, lang{1,7}_tt denote language IDs 1 and 7 for training and test.
Figure 32: Feature representation in cross-domain speaker recognition before (a) and after (b) adaptation.
In audio generation evaluation, Fréchet Audio Distance (FAD) is a 2-Wasserstein distance with structural constraints for both primitives: the cost is a frozen embedding pullback whose invariance set hides severe artifacts, and the coupling is a Gaussian fit that dilutes rank-1 contamination relative to discrete OT. We propose Optimal Transport Audio Distance (OTAD), which corrects each primitive with one dedicated mechanism -- a residual Riemannian ground-metric adapter for the cost and entropic Sinkhorn optimal transport for the coupling. Across eight encoders under a four-axis protocol, coupling-only comparisons at ε=0.05 show that Sinkhorn's rank-1 sensitivity exceeds FAD's by a factor of 1.9 to 3.6. Furthermore, OTAD achieves a higher mean Spearman correlation with audio-quality MOS (DCASE 2023 Task 7) than baseline metrics. As an intrinsic benefit of the discrete transport plan, OTAD yields per-sample diagnostics with AUROC ≥0.86, a capability that scalar- or kernel-aggregated metrics structurally lack.
Optimal Transport (OT) provides a principled framework for learning transformations between probability distributions from unpaired samples. In many applications, however, a single transformation must map several source distributions to a common target distribution. For example, image restoration might require handling different types of degradation without knowing the degradation of each input at inference time. Simple approaches of pooling the source distributions only encourage alignment with the target at the aggregate level and may leave individual sources misaligned. In our paper, we consider the simultaneous OT problem which formalizes the task of learning a shared transport map that minimizes the average transport cost while aligning each source distribution with a prescribed target. We propose a neural method for solving the simultaneous OT problem by learning a shared transport map that minimizes the average transport cost while aligning each source distribution with a prescribed target. We derive a max-min formulation for learning this map. We illustrate its application to image restoration, where a single model handles multiple degradation types using a common collection of clean target images.
Milena Gazdieva, Kirill Sokolov, Jiawei Chen +2
Applied AI Institute AXXX Moscow, Russia · Lomonosov Moscow State University Moscow, Russia · MIRIAI Moscow, Russia
Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergences may become ineffective when these distributions overlap weakly, which is frequently encountered in imitation learning, offline RL, and deployment under distribution shift. Optimal transport (OT) offers an alternative by measuring the cost of moving probability mass from one distribution to another under a ground cost that encodes task geometry. This survey covers how OT is used inside RL objectives and algorithms. For each method, we identify: the role OT plays, the distributions compared, the OT formulation used, and the treatment of temporal structure. Beyond categorising existing methods, we discuss the motivations behind different OT choices, practical considerations such as cost design and computational challenges, and highlight open problems including scalable trajectory-level transport, principled handling of mass mismatch, and theoretical analysis for OT-regularised RL.