Canonical Correlation Neural Networks

Andrea Cavallo; Sundeep Prabhakar Chepuri; Charilaos Kanatsoulis; Elvin Isufi

cs.LGNov 4, 2026

Canonical Correlation Neural Networks

Authors: Andrea Cavallo, Sundeep Prabhakar Chepuri, Charilaos Kanatsoulis, Elvin Isufi

Organizations: Delft University of Technology, Delft, Netherlands · Indian Institute of Science, Bangalore, India · Stanford University, CA, USA

Abstract

Relational inductive biases such as grids or graphs allow neural networks with fewer trainable parameters and improved generalization, but require a known structure beforehand. When no structure is available, the covariance matrix provides a principled substitute, and coVariance Neural Networks (VNNs) have shown that using it yields models that are theoretically grounded in Principal Component Analysis, stable to finite-sample estimation errors and parameter-efficient. This framework, however, is limited to single-view data, whereas many applications observe the same phenomenon through multiple views, each with its own in-view covariance and a cross-view covariance linking them. To incorporate these matrices as inductive biases in a principled way, we introduce CAnonical correlation NEural networks (CANEs), architectures that use in-view and cross-view covariances as distinct relational biases to solve downstream tasks. Via spectral analysis, CANEs draw analogies with Canonical Correlation Analysis (CCA), a classical technique to project two-view data onto a correlated subspace. However, while CCA's objective is to compute shared view representations, CANEs are trained to solve a downstream task directly. We bound their output error under finite-sample covariance perturbations and show that it depends on controllable Lipschitz constants of the learned filters in place of the singular-value gap that governs the stability of CCA, thus guaranteeing better robustness. Experiments on seven datasets from brain imaging and motion sensing show that CANEs match or outperform single-view, two-view and CCA-based baselines, often with fewer parameters, and retain their performance when covariances are re-estimated from fewer samples.

Figures & tables

Figure 1: Diagram of the CANE with 2 layers per stage (i.e.,

L_{v}=L_{c}=2

) and filterbanks of size 1 (i.e.,

F_{\ell}=1

\forall\ell

). The input is first processed by the in-view covariance filters independently (in-view covariance stage). Then, the cross-view covariance matrix

\mathbf{\hat{A}}_{12}

is computed to merge the two views, and the cross-view covariance filters further process the data via cross-view correlations, generating one output for each view (cross-view covariance stage). At test time, the matrix

\mathbf{\hat{A}}_{12}

is computed from the training data.

Figure 2: CCA performs two consecutive filtering operations: first, the filter

h_{w}(\hat{\lambda})=\hat{\lambda}^{-1/2}

acts on the covariance eigenvalues of each view to perform whitening (left); then, the high-pass filter

h_{\textnormal{cca}}(\hat{\sigma}^{2})

acts on the cross-view covariance singular values to select the top-

D

directions (right). CAFs replace these operations with learnable polynomial filters: the VF

h_{v}(\hat{\lambda})

in ( 3 ) and the CVF

h_{c}(\hat{\sigma}^{2})

in ( 4 ).

		MAE ( $\downarrow$ )				Accuracy ( $\uparrow$ )
		ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
One view	MLP	7.64 $\pm$ 0.30	8.65 $\pm$ 0.27	8.99 $\pm$ 0.33	3.81 $\pm$ 0.05	71.1 $\pm$ 4.0	90.1 $\pm$ 1.0	72.4 $\pm$ 2.7
	PCA	5.17 $\pm$ 0.09	5.42 $\pm$ 0.05	6.26 $\pm$ 0.09	3.74 $\pm$ 0.12	73.7 $\pm$ 5.0	85.5 $\pm$ 2.7	67.6 $\pm$ 4.0
	SPCA	5.22 $\pm$ 0.06	5.42 $\pm$ 0.10	6.19 $\pm$ 0.13	3.77 $\pm$ 0.17	71.0 $\pm$ 3.2	86.0 $\pm$ 1.6	72.0 $\pm$ 3.0
	VNN	5.16 $\pm$ 0.05	5.24 $\pm$ 0.08	5.81 $\pm$ 0.18	3.25 $\pm$ 0.11	79.1 $\pm$ 3.9	91.9 $\pm$ 0.6	74.6 $\pm$ 4.4
Both views	MLP	10.55 $\pm$ 0.48	9.42 $\pm$ 0.21	9.99 $\pm$ 0.33	3.90 $\pm$ 0.25	91.9 $\pm$ 1.6	95.9 $\pm$ 1.1	78.1 $\pm$ 1.9

Table 1: Performance on age prediction from brain MRI measurements (MAE in years,

\downarrow

) and human action classification from motion sensors (accuracy in %,

\uparrow

). First best and second best results are highlighted. One view results are for the best single view selected on validation.

	ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
VNN, one view	575k	142k	1.25M	32k	103k	497k	119k
VNN, both views	1.12M	279k	2.36M	269k	52k	470k	596k
SDGCCA	699k	699k	699k	2.39M	171k	279k	700k
TOCCA	123k	198k	123k	115k	21k	226k	690k
CANE (Ours)	1.13M	1.14M	281k	67k	56k	230k	351k

Table 2: Number of trainable parameters of the configuration selected on validation for each dataset (Table 1 ).

	MAE ( $\downarrow$ )				Accuracy ( $\uparrow$ )
	ADNI1	ADNI2	ADNI3	ABIDE	MHEALTH	REALDISP	DSADS
in-view	5.07 $\pm$ 0.05	5.04 $\pm$ 0.06	5.61 $\pm$ 0.09	3.19 $\pm$ 0.22	93.1 $\pm$ 1.4	95.6 $\pm$ 1.3	83.8 $\pm$ 2.0
cross-view	5.30 $\pm$ 0.23	5.42 $\pm$ 0.05	5.62 $\pm$ 0.08	3.14 $\pm$ 0.07	93.1 $\pm$ 1.9	96.6 $\pm$ 1.3	79.4 $\pm$ 2.5
full	4.97 $\pm$ 0.10	5.02 $\pm$ 0.08	5.55 $\pm$ 0.07	3.02 $\pm$ 0.14	95.3 $\pm$ 1.1	97.9 $\pm$ 1.0	84.1 $\pm$ 3.7

Table 3: Ablation study on CANEs’ stages. First best and second best results are highlighted.

Figure 3: Results under finite-sample covariance perturbations. Rows 1 and 3 show the task performance, rows 2 and 4 the normalized Frobenius norm of the difference of the generated embeddings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Nodes	Node	Classes	Samples $T$
	( $N_{1}$ , $N_{2}$ )	feat.	Classes	(CV/test)
ADNI1	68, 68	1	NA	801 (640/161)
ADNI2	68, 68	1	NA	1142 (913/229)
ADNI3	68, 68	1	NA	1080 (864/216)
ABIDE	62, 68	1	NA	1035 (828/207)
MHEALTH	6, 6	5	12	5357 (4267/1090)

Appendix

Table 4: Details on datasets. NA means the dataset is used for regression. Samples are split into a pool for 4-fold cross-validation (CV) and a fixed test set.

Figure 6: Time (milliseconds) and allocated memory (MB) for different models on synthetic datasets of increasing dimensionality

N_{1}=N_{2}

Multi-View Learning

Multi-view learning combines complementary representations of the same objects to improve prediction, clustering, and robust data integration.

See all 191 papers

Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs
Sep 9, 2026Saurabh Sihag, Andrea Cavallo, Elvin Isufi +2Graph Neural Networks Brain Age Prediction
Department of Electrical and Computer Engineering at the University at Albany, SUNY, Albany, NY · Delft University of Technology, Delft, The Netherlands · Department of Electrical and Computer Engineering at the University of Rochester, Rochester, NY +1
TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees
Jul 29, 2026James ChapmanGradient-Boosted Decision Trees Multi-View Learning
Joint Precision Neural Networks: Task-Aware Dependency and Predictive Learning
Oct 5, 2026Andrea Cavallo, Samuel Rey, Antonio G. Marques +1Graph Structure Learning Graph Neural Networks
INSY Department, Delft University of Technology · Department of Signal Theory and Communications, King Juan Carlos University

Sep 9, 2026J/K move · Enter open · S save

Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs

Saurabh Sihag, Andrea Cavallo, Elvin Isufi, Gonzalo Mateos+1

Department of Electrical and Computer Engineering at the University at Albany, SUNY, Albany, NY · Delft University of Technology, Delft, The Netherlands · Department of Electrical and Computer Engineering at the University of Rochester, Rochester, NY+1

Graph Neural Networks Brain Age Prediction Graph Transfer Learning Transfer Learning

Open paper AskPDF arXiv

This feature article provides an overview of the theoretical foundations for coVariance neural networks (VNNs), i.e., graph neural networks (GNNs) operating on covariance matrices as graphs. Covariance matrices are ubiquitous across domains, and hence, the deployment of GNNs often leverages graphs of pairwise statistical dependencies. Existing theoretical contributions on GNNs consider abstract graph representations and cannot accommodate the data-driven nuances associated with covariance matrices. This tutorial brings into focus various novel theoretical insights via mathematical analyses of VNNs that have broad signal processing implications, including: (i) a conceptual equivalence between VNNs and principal component analysis (PCA)-based information processing; (ii) refined stability bounds on predictive outcomes in the presence of finite sample-induced covariance matrix perturbations; and (iii) refined characterization of transferability of VNNs across multiscale datasets. The theoretical insights discussed herein provide the underlying principles and justification towards adopting VNNs over workhorse PCA-based learning pipelines, in applications where covariance matrices are useful descriptors of data structure. We also convey how impact of these foundational advances permeates to principled designs and applications of learning methods across broad domains where covariance matrices emerge. Notably, we elucidate the conceptual insights facilitated by VNNs to the specific task of characterizing brain age gap for neurodegenerative conditions using neuroimaging datasets, a timely problem in computational neuroscience. Broader impacts to other application domains are discussed as well.

Abstract

Multi-View Learning

Explore similar work

Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs

TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees

Joint Precision Neural Networks: Task-Aware Dependency and Predictive Learning