Canonical Correlation Neural Networks

Andrea Cavallo; Sundeep Prabhakar Chepuri; Charilaos Kanatsoulis; Elvin Isufi

cs.LGNov 4, 2026

Canonical Correlation Neural Networks

Authors: Andrea Cavallo, Sundeep Prabhakar Chepuri, Charilaos Kanatsoulis, Elvin Isufi

Organizations: Delft University of Technology, Delft, Netherlands · Indian Institute of Science, Bangalore, India · Stanford University, CA, USA

Abstract

Relational inductive biases such as grids or graphs allow neural networks with fewer trainable parameters and improved generalization, but require a known structure beforehand. When no structure is available, the covariance matrix provides a principled substitute, and coVariance Neural Networks (VNNs) have shown that using it yields models that are theoretically grounded in Principal Component Analysis, stable to finite-sample estimation errors and parameter-efficient. This framework, however, is limited to single-view data, whereas many applications observe the same phenomenon through multiple views, each with its own in-view covariance and a cross-view covariance linking them. To incorporate these matrices as inductive biases in a principled way, we introduce CAnonical correlation NEural networks (CANEs), architectures that use in-view and cross-view covariances as distinct relational biases to solve downstream tasks. Via spectral analysis, CANEs draw analogies with Canonical Correlation Analysis (CCA), a classical technique to project two-view data onto a correlated subspace. However, while CCA's objective is to compute shared view representations, CANEs are trained to solve a downstream task directly. We bound their output error under finite-sample covariance perturbations and show that it depends on controllable Lipschitz constants of the learned filters in place of the singular-value gap that governs the stability of CCA, thus guaranteeing better robustness. Experiments on seven datasets from brain imaging and motion sensing show that CANEs match or outperform single-view, two-view and CCA-based baselines, often with fewer parameters, and retain their performance when covariances are re-estimated from fewer samples.

Figures & tables

Figure 1: Diagram of the CANE with 2 layers per stage (i.e.,

L_{v}=L_{c}=2

) and filterbanks of size 1 (i.e.,

F_{\ell}=1

\forall\ell

). The input is first processed by the in-view covariance filters independently (in-view covariance stage). Then, the cross-view covariance matrix

\mathbf{\hat{A}}_{12}

is computed to merge the two views, and the cross-view covariance filters further process the data via cross-view correlations, generating one output for each view (cross-view covariance stage). At test time, the matrix

\mathbf{\hat{A}}_{12}

is computed from the training data.

Figure 2: CCA performs two consecutive filtering operations: first, the filter

h_{w}(\hat{\lambda})=\hat{\lambda}^{-1/2}

acts on the covariance eigenvalues of each view to perform whitening (left); then, the high-pass filter

h_{\textnormal{cca}}(\hat{\sigma}^{2})

acts on the cross-view covariance singular values to select the top-

D

directions (right). CAFs replace these operations with learnable polynomial filters: the VF

h_{v}(\hat{\lambda})

in ( 3 ) and the CVF

h_{c}(\hat{\sigma}^{2})

in ( 4 ).

		MAE ( $\downarrow$ )				Accuracy ( $\uparrow$ )
		ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
One view	MLP	7.64 $\pm$ 0.30	8.65 $\pm$ 0.27	8.99 $\pm$ 0.33	3.81 $\pm$ 0.05	71.1 $\pm$ 4.0	90.1 $\pm$ 1.0	72.4 $\pm$ 2.7
	PCA	5.17 $\pm$ 0.09	5.42 $\pm$ 0.05	6.26 $\pm$ 0.09	3.74 $\pm$ 0.12	73.7 $\pm$ 5.0	85.5 $\pm$ 2.7	67.6 $\pm$ 4.0
	SPCA	5.22 $\pm$ 0.06	5.42 $\pm$ 0.10	6.19 $\pm$ 0.13	3.77 $\pm$ 0.17	71.0 $\pm$ 3.2	86.0 $\pm$ 1.6	72.0 $\pm$ 3.0
	VNN	5.16 $\pm$ 0.05	5.24 $\pm$ 0.08	5.81 $\pm$ 0.18	3.25 $\pm$ 0.11	79.1 $\pm$ 3.9	91.9 $\pm$ 0.6	74.6 $\pm$ 4.4
Both views	MLP	10.55 $\pm$ 0.48	9.42 $\pm$ 0.21	9.99 $\pm$ 0.33	3.90 $\pm$ 0.25	91.9 $\pm$ 1.6	95.9 $\pm$ 1.1	78.1 $\pm$ 1.9

Table 1: Performance on age prediction from brain MRI measurements (MAE in years,

\downarrow

) and human action classification from motion sensors (accuracy in %,

\uparrow

). First best and second best results are highlighted. One view results are for the best single view selected on validation.

	ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
VNN, one view	575k	142k	1.25M	32k	103k	497k	119k
VNN, both views	1.12M	279k	2.36M	269k	52k	470k	596k
SDGCCA	699k	699k	699k	2.39M	171k	279k	700k
TOCCA	123k	198k	123k	115k	21k	226k	690k
CANE (Ours)	1.13M	1.14M	281k	67k	56k	230k	351k

Table 2: Number of trainable parameters of the configuration selected on validation for each dataset (Table 1 ).

	MAE ( $\downarrow$ )				Accuracy ( $\uparrow$ )
	ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
in-view	5.07 $\pm$ 0.05	5.04 $\pm$ 0.06	5.61 $\pm$ 0.09	3.19 $\pm$ 0.22	93.1 $\pm$ 1.4	95.6 $\pm$ 1.3	83.8 $\pm$ 2.0
cross-view	5.30 $\pm$ 0.23	5.42 $\pm$ 0.05	5.62 $\pm$ 0.08	3.14 $\pm$ 0.07	93.1 $\pm$ 1.9	96.6 $\pm$ 1.3	79.4 $\pm$ 2.5
full	4.97 $\pm$ 0.10	5.02 $\pm$ 0.08	5.55 $\pm$ 0.07	3.02 $\pm$ 0.14	95.3 $\pm$ 1.1	97.9 $\pm$ 1.0	84.1 $\pm$ 3.7

Table 3: Ablation study on CANEs’ stages. First best and second best results are highlighted.

Figure 3: Results under finite-sample covariance perturbations. Rows 1 and 3 show the task performance, rows 2 and 4 the normalized Frobenius norm of the difference of the generated embeddings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Nodes	Node	Classes	Samples $T$
	( $N_{1}$ , $N_{2}$ )	feat.	Classes	(CV/test)
ADNI1	68, 68	1	NA	801 (640/161)
ADNI2	68, 68	1	NA	1142 (913/229)
ADNI3	68, 68	1	NA	1080 (864/216)
ABIDE	62, 68	1	NA	1035 (828/207)
MHEALTH	6, 6	5	12	5357 (4267/1090)

Appendix

Table 4: Details on datasets. NA means the dataset is used for regression. Samples are split into a pool for 4-fold cross-validation (CV) and a fixed test set.

Figure 6: Time (milliseconds) and allocated memory (MB) for different models on synthetic datasets of increasing dimensionality

N_{1}=N_{2}

Multi-View Learning

Multi-view learning combines complementary representations of the same objects to improve prediction, clustering, and robust data integration.

See all 191 papers

Sep 9, 2026

Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs

This feature article provides an overview of the theoretical foundations for coVariance neural networks (VNNs), i.e., graph neural networks (GNNs) operating on covariance matrices as graphs. Covariance matrices are ubiquitous across domains, and hence, the deployment of GNNs often leverages graphs of pairwise statistical dependencies. Existing theoretical contributions on GNNs consider abstract graph representations and cannot accommodate the data-driven nuances associated with covariance matrices. This tutorial brings into focus various novel theoretical insights via mathematical analyses of VNNs that have broad signal processing implications, including: (i) a conceptual equivalence between VNNs and principal component analysis (PCA)-based information processing; (ii) refined stability bounds on predictive outcomes in the presence of finite sample-induced covariance matrix perturbations; and (iii) refined characterization of transferability of VNNs across multiscale datasets. The theoretical insights discussed herein provide the underlying principles and justification towards adopting VNNs over workhorse PCA-based learning pipelines, in applications where covariance matrices are useful descriptors of data structure. We also convey how impact of these foundational advances permeates to principled designs and applications of learning methods across broad domains where covariance matrices emerge. Notably, we elucidate the conceptual insights facilitated by VNNs to the specific task of characterizing brain age gap for neurodegenerative conditions using neuroimaging datasets, a timely problem in computational neuroscience. Broader impacts to other application domains are discussed as well.

Saurabh Sihag, Andrea Cavallo, Elvin Isufi +2

Department of Electrical and Computer Engineering at the University at Albany, SUNY, Albany, NY · Delft University of Technology, Delft, The Netherlands · Department of Electrical and Computer Engineering at the University of Rochester, Rochester, NY +1

Graph Neural Networks Brain Age Prediction Graph Transfer Learning Transfer Learning

Ask PDF

Jul 29, 2026

TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees

Gradient-boosted trees dominate tabular machine learning, yet canonical correlation analysis has always relied on linear or neural encoders. We propose TreeCCA, the first method to train gradient-boosted tree ensembles end-to-end as CCA encoders, inheriting their plug-and-play reliability: no architecture design, familiar hyperparameters, and strong performance with defaults. The technical enabler is the Eckart-Young (EY) loss, which supplies closed-form per-sample gradients that slot directly into any standard GBT library (XGBoost, LightGBM) as a custom objective. TreeCCA is the first CCA method to combine nonlinear accuracy with native interpretability: every tree split selects one feature, so gain importances reveal which inputs drive cross-view correlation at no extra cost. We demonstrate these properties on synthetic benchmarks, where TreeCCA matches or exceeds Deep CCA (2.61 vs.\ 2.43 on Signed Power; 2.93 vs.\ 2.89 on Hermite), and on a sparse benchmark with zero linear cross-view covariance, where TreeCCA recovers the true support with

\text{Precision@}S = 1.00

p=50

while PMD finds no signal. On the UCI HAR sensor-fusion benchmark, TreeCCA achieves comparable accuracy to Deep CCA at

5\times

lower cost, while XGBoost gain importances directly validate a physics-motivated hypothesis about the data --- an interpretation not readily available with neural encoders. Across five popular tabular multi-view datasets, TreeMCCA consistently matches or exceeds linear CCA in both nonlinear correlation extraction and downstream classification accuracy.

James Chapman

Gradient-Boosted Decision Trees Multi-View Learning Tabular ML Canonical Correlation Analysis

Ask PDF

Oct 5, 2026

Joint Precision Neural Networks: Task-Aware Dependency and Predictive Learning

Exploiting meaningful latent structures from data to solve downstream tasks is a fundamental challenge in signal processing and machine learning. While Principal Component Analysis (PCA) and coVariance Neural Networks (VNNs) successfully leverage the covariance matrix to process data, they inherently capture both direct and indirect correlations. The precision matrix (inverse covariance) overcomes this by explicitly encoding conditional independencies, making it largely studied in graphical lasso and graph topology identification. However, finite-sample precision estimates are notoriously unstable, and regularized estimators remain task-agnostic. In this work, our principal contribution is tackling the challenging problem of task-aware graph inference. We propose Precision Neural Networks-Joint (PNN-Joint), a framework that jointly estimates a sparse, statistically grounded precision matrix alongside graph neural network weights via an alternating optimization scheme. As a foundational framework to support this, we introduce Precision Neural Networks (PNNs), a broader class of graph convolutional networks operating on precision estimators, and establish their spectral connections to PCA and VNNs alongside their stability to finite-sample errors. Extensive empirical evaluations on synthetic data, as well as real-world neuroimaging and motion sensor datasets, demonstrate that PNN-Joint yields highly interpretable task-aware graphs, exhibits remarkable robustness in low-data regimes, and consistently achieves the best or second-best performance among competitors on real-world tasks.

Andrea Cavallo, Samuel Rey, Antonio G. Marques +1

INSY Department, Delft University of Technology · Department of Signal Theory and Communications, King Juan Carlos University

Graph Structure Learning Graph Neural Networks Covariance Estimation

Ask PDF

Abstract

Multi-View Learning

Explore similar work

Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs

TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees

Joint Precision Neural Networks: Task-Aware Dependency and Predictive Learning