PathFinder: Joint Decompositions of Linked Multimodal Datasets

Ying-Qiu Zheng; Alex Fung; Stephen M Smith; Rogier B Mars; Saad Jbabdi

cs.LGAug 15, 2026

PathFinder: Joint Decompositions of Linked Multimodal Datasets

Authors: Ying-Qiu Zheng, Alex Fung, Stephen M Smith, Rogier B Mars, Saad Jbabdi

Organizations: Oxford Centre for Integrative Neuroimaging University of Oxford Oxford, UK

Abstract

Low-rank matrix decompositions can uncover patterns and structure in data and have a number of different applications across many disciplines. Extensions to "joint" low-rank decompositions have been proposed to link datasets from different modalities. While these methods enable the discovery of common patterns across modalities, they require that all the multimodal data share one or more dimensions. We propose a new analysis method, PathFinder, that enables co-analysis of datasets that do not necessarily all share a dimension. The key insight is that as long as pairs or subgroups of matrices do share some dimension, and that there are one or more paths that link across the data matrices, a global joint decomposition can be sought out. This enables the joint estimation of common patterns across different modalities, species, or scales, where a one-to-one mapping across all data along some dimension is not necessarily available. We show that PathFinder is a general umbrella under which many matrix decomposition methods fall as special cases. It can be used to discover common patterns across disparate datasets and to make predictions for missing data or modalities.

Figures & tables

Figure 1: PathFinder concept. (A) Cortex $\times$ white matter matrices of two species that have incompatible dimensions. (B) Introducing subgroups of matrices that do have a dimension in common links all matrices, acting as a "path". (C) This can generalise to multiple domains and modalities with some modalities potentially missing for some domains.

Algorithm: Alternating Least Square in PathFinder
1:	Initialise $\mathbf{A_{\text{d}}}$ and $\mathbf{S_{\text{m}}}$ for $d=1,2,...D$ and $m=1,2,...M$
2:	while not converged do
	for $d=1,2,...D$
3a:	horizontally concatenate $\mathbf{X}_{\text{d,m}}$ as $\mathbf{\tilde{X}}_{\text{d}}$ and $\mathbf{S}_{\text{m}}$ as $\mathbf{\tilde{S}}$ , for $m$ if $\mathbf{X}_{\text{dm}}$ exists
3b:	set $\mathbf{A_{\text{d}}}\leftarrow\mathbf{\tilde{X}_{\text{d}}}\mathbf{\tilde{S}}(\mathbf{\tilde{S}}^{T}\mathbf{\tilde{S}}+\alpha\mathbf{I})^{-1}$
	for $m=1,2,...M$

Table 1: Pseudo code for the PathFinder alternating least squares algorithm.

Figure 2: Simulation results of a 2 x 3 grid of matrices spanning 2 modalities and 3 domains. Four of the six matrices (blue) are provided to PathFinder for fitting, while two (orange) were marked as missing. (A) Illustration of matrix configuration. (B) Predicted values are plotted against ground truth at noise level $\sigma=0.1$ and $r=20$ . All six matrices gave high reconstruction accuracy $R^{2}$ . (C) The effect of rank $r$ (number of components) at noise level 0.01. Shaded regions indicate the interquartile range over 10 runs. (D) Impact of the regularisation parameter.

Figure 3: Simulation results of cyclic data decomposition. (A) Example of cyclic data and its PathFinder configuration. (B) True and predicted activity. (C) True and reconstructed factors (with or without ICA post-processing).

Figure 4: Example of PathFinder reconstructing missing retinotopy task stimuli. Brain grayordinates and visual field pixels served as domains. The six different retinotopy tasks served as modalities. the RETCCW stimulus (Counter-clockwise rotating wedge) was marked as missing and held out from PathFinder training. The predicted RETCCW matches the actual stimulus.

Figure 5: PathFinder for Dual-Regression. (A) Example matrix configuration for PathFinder as Dual-Regression (B) PathFinder components are similar to dual regression spatial maps, both at the level of the group and individuals. (C) Visualised PathFinder and dual regression maps for the group and an example subject.

Figure 6: PathFinder decomposition of the cortico-basal direct loop. A: PathFinder configuration. B: running a winner-takes-all across components reveals functional circuits across the loop. C: connectivity matrices and their low-rank reconstructed counterparts.

Figure 7: PathFinder Extensions. (A) PathFinder can accommodate not only 2D matrices but also higher-order tensors. (B) An example of non-linear PathFinder using the encoder-decoder architecture. (C) PathFinder can drop the bipartite formulation, i.e., arranging the factors on one side instead of on left and right. This expands the applications for PathFinder, allowing it to run natively on biological networks.

Algorithm: Minibatch Alternate Least Square in PATHFINDER
1:	Initialise $\mathbf{A_{\text{d}}}$ and $\mathbf{S_{\text{m}}}$ for $d=1,2,...D$ and $m=1,2,...M$
2:	while not converged do
	for $d=1,2,...D$
3a:	initialise accumulators for updating $\mathbf{A}_{\text{d}}$ , $\mathbf{C}_{\text{num}}\leftarrow 0$ , $\mathbf{C}_{\text{denom}}\leftarrow 0$
3b:	for $m\in\mathcal{M}_{d}$ , $b\in\{1,2,...B(m)\}$
	set $\mathbf{C}_{\text{num}}\leftarrow\mathbf{C}_{\text{num}}+\mathbf{x^{\text{b}}_{\text{d,m}}}\mathbf{s^{\text{b}}_{\text{m}}}$

Table S1: Pseudo code of Minibatch ALS algorithm to solve PATHFINDER.

Algorithm: Joint SVD
1:	Initialise $\mathbf{U_{\text{d}}}$ , $\mathbf{D}_{d,m}$ , and $\mathbf{V_{\text{m}}}$ for $d=1,2,...D$ and $m=1,2,...M$
	while not converged do
2:	for $(d,m)\in\mathcal{M}$
	set $\mathbf{D}_{d,m}\leftarrow\mathbf{U}_{d}^{T}\mathbf{X}_{d,m}\mathbf{V}_{m}$
3:	for each domain $d=1,2,...D$
	for each component $n=1,2,...r$

Table S2: Pseudo code of the algorithm to solve joint SVD.

Algorithm: Mini-batch Joint SVD
1:	Initialise $\mathbf{U}_{d}$ , $\mathbf{V}_{m}$ for $d=1,\dots,D$ and $m=1,\dots,M$ ;
	set $\mathbf{D}_{d,m}\leftarrow\mathrm{diag}\bigl(\mathbf{U}_{d}^{T}\mathbf{X}_{d,m}\mathbf{V}_{m}\bigr)$ for $(d,m)\in\mathcal{M}$
	while not converged do
2:	for $(d,m)\in\mathcal{M}$
	set $\mathbf{D}_{d,m}\leftarrow\mathrm{diag}\bigl(\mathbf{U}_{d}^{T}\mathbf{X}_{d,m}\mathbf{V}_{m}\bigr)$
3:	for each domain $d=1,\dots,D$

Table S3: Pseudocode for the mini-batch Joint SVD algorithm.

T_{\text{pow}}

is the number of power-iteration steps per component (e.g. 2 in our implementation), and batch_size controls both the number of sampled matrices and the chunk size for matrix-vector products.

\mathcal{D}_{m}

denotes the set of domains for which modality

m

is observed.

Figure S1: Robustness to noise of PathFinder and independent SVD . PathFinder gave higher accuracy when the noise level is high, suggesting a denoising effect by sharing factors across domains and modalities.

Figure S2: (A) Convergence curve of various matrix sizes. Larger matrix size gives faster convergence. (B) Time required for 20 iterations as a function of matrix size.

Explore similar work

Aug 13, 2026

Knowledge-guided Pattern Discovery via Coupled Tensor Factorizations

In order to understand complex systems such as the human metabolome or human brain, different sensing technologies are used, generating complex data. These datasets are often multiway, i.e., with more than two axes of variation such as a subjects by metabolites by time array. While tensor factorizations have successfully revealed interpretable patterns from such complex data, they have so far been mainly data-driven. On the other hand, there is more to data -- there are computational models (of these systems), which are rich sources of prior information. In this paper, we introduce a knowledge-guided approach that brings together data and computational models by jointly analyzing real data and simulated data (generated using a computational model) using coupled tensor factorizations with linear coupling. Our experiments on real metabolomics measurements demonstrate that guiding the analysis of such noisy data with simulated data improves the pattern discovery performance while also revealing potential discrepancies between data and computational models.

Gaute Johannessen, Geert Roelof van der Ploeg, Evrim Acar

Faculty of Mathematics and Natural Sciences, University of Oslo, Oslo · Department of Data Science and Knowledge Discovery, Simula Metropolitan Center for Digital Engineering, Oslo

Tensor Decomposition Scientific ML

Ask PDF

Jul 18, 2026

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Vector Institute, Toronto, Canada +1

Cross-Modal Representation Learning Multimodal Disentangled Representation Learning Representation Disentanglement Multimodal Model Interpretability Multimodal Learning Parameter-Efficient Fine-Tuning

Ask PDF

Aug 11, 2026

FiGuRO: Intrinsic Dimension Estimation for Multi-Modal Data

Determining the complexity, or Intrinsic Dimension (ID), of data is fundamental to efficient and interpretable representation learning. This is particularly challenging in multi-modal settings when trying to learn disentangled representations for shared and private information. Existing techniques leave a critical gap: they are often static, uni-modal, or in the case of contrastive methods, adapt only to the shared ID implicitly. We introduce Fidelity-Guided Rank Optimization (FiGuRO), a framework for approximating the ID of uni- and multi-modal data under constraints of model capacity and hyperparameters. FiGuRO learns the dimensions of low-rank projections using truncated singular value decomposition and an algorithm that determines when to reduce or increase dimension and in which latent space. Disentanglement of shared and private information arises as an emergent property of this optimization, eliminating the need for complex auxiliary loss functions. We demonstrate that FiGuRO outperforms existing ID estimation techniques and is more robust to hyperparameter changes. Across simulations and real-world data, FiGuRO captures distinct ID scales and varying subspace ratios, and decomposes shared and private information successfully. Furthermore, we show that FiGuRO can be applied to modern uni-modal pretrained models, enabling efficient, post-hoc disentanglement of multi-modal representations.

Viktoria Schuster, Sana Tonekaboni, Caroline Uhler

Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Technical University of Denmark, Lyngby, Denmark +1

Intrinsic Dimensionality Representation Learning Multimodal Disentangled Representation Learning

Ask PDF