From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

Paris A. Karakasis; Walid Saad

cs.ITOct 20, 2026

From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

Authors: Paris A. Karakasis, Walid Saad

Organizations: Institute for Advanced Computing, Bradley Department of Electrical and Computer Engineering, Virginia Polytechnic Institute and State University, Alexandria, VA 22305, USA

Abstract

Many problems in signal processing, machine learning, and neuroimaging provide several jointly observed measurements of one process, each mixing a component common to all of them with a component private to itself, and require the common component to be recovered without supervision. This problem has been studied in several communities, largely independently. Statisticians proposed canonical correlation analysis (CCA), introduced by Hotelling in 1936. Information theorists proposed two notions of common information, due to Gács and Körner and to Wyner. Engineers developed subspace methods for sensor arrays and multiuser communications to design multiview receivers. This tutorial presents the main results and tools each community developed for this problem in a single exposition, and explains the connections that emerge where they meet, distinguishing exact equivalences from special cases and analogies. We develop the information-theoretic foundation from first principles. We then introduce CCA and its extension to several views, followed by the engineering thread from blind multiuser detection to CCA as a blind receiver. Next, we turn to nonlinear CCA, covering kernel, deep, and probabilistic formulations, the conditions under which the common factor is identifiable, and connections to self-supervised learning. We conclude with a discussion of further extensions and open directions. The exposition is self-contained, so that readers from each of these communities can follow the parts developed by the others.

Figures & tables

Figure 1: Two neighboring cells, each served by a multi-antenna base station. A user near the center of a cell is received strongly at its own station and is barely heard at the other, so only one link is drawn for it. The two cell-edge users are received at low power by both stations, so each of them carries a link to BS-1 and a link to BS-2, and they are the challenging case for decoding.

Work	Info. Theory	Statistics	Sig. Proc.	Multi-View CCA	Nonlinear CCA	Deep CCA	Identif.
Borga [ 14 ]	Partial	✓	Partial
Hardoon et al. [ 15 ]		✓		Partial	✓
Uurtio et al. [ 16 ]		✓			✓	Partial
Wang et al. [ 17 ]		✓		Partial	Partial	Partial	Partial
Yang et al. [ 23 ]		✓		✓	✓	✓	Partial
Yu and Tan [ 24 ]	✓

Table 1: Scope of existing treatments of common information and CCA. A check mark means the topic is covered substantively, and Partial means it is touched on without being developed.

Term	Meaning
Shared structure	Informal term for whatever the views have in common, used only before one of the definitions below is adopted.
Common information	A number, measured in bits and computed from the joint distribution of the views, namely the Gács–Körner or the Wyner common information. The number says how much the views share, not what they share.
Common factor	A latent random vector shared by all views in a multiview generative model, such that the views are conditionally independent given the common factor. Whatever else enters a view is private to that view and forms its private factor.
Common subspace	The column space of the matrix whose rows are the values of the common factor at the training samples.
Common representation	What the outputs of the learned encoders of the views are made to agree on.

Table 2: Terminology used throughout the tutorial.

Figure 2: Two perception systems at an intersection label the same objects with different label sets. A third system maps the labels of System A through

f

and the labels of System B through

g

to a common label set. The finest labeling on which the two maps always agree is light vehicle, heavy vehicle and person.

Figure 3: One speaker recorded by two microphones. Each recording contains the voice, which is common to both, and the noise of its own microphone. The recordings are dependent only through the voice, so given the voice the two recordings are conditionally independent.

Figure 4: Partitions of the sample space $\Omega=[0,1]^{2}$ , with outcome $\omega=(\omega_{1},\omega_{2})$ , into the atoms of $\sigma(X)$ and $\sigma(Y)$ , the sigma-algebras generated by $X$ and $Y$ , of their join $\sigma(X)\vee\sigma(Y)=\sigma(X,Y)$ , and of their meet $\sigma(X)\cap\sigma(Y)$ . Each color marks one atom, that is, the set of all outcomes at which the random variable generating the partition takes one particular value. The join is the coarsest partition that refines both $\sigma(X)$ and $\sigma(Y)$ . The meet is the finest partition coarser than both, and the five atoms of the meet index the values of the common part $W^{*}$ .

Figure 5: Joint PMF

P_{XY}

in the case

K_{GK}(X;Y)=I(X;Y)=C(X;Y)

, with rows and columns permuted so that

P_{XY}

is block-diagonal. The block in blue corresponds to

W^{*}=1

, the block in orange to

W^{*}=2

, and darker shades indicate larger probabilities. Each block equals

P_{W^{*}}(w)

times the outer product of the conditional PMFs

P_{X|W^{*}=w}

and

P_{Y|W^{*}=w}

, as the arrows indicate, which reflects the conditional independence of

X

and

Y

given

W^{*}

. The two blocks occupy disjoint rows and disjoint columns, so each of

X

and

Y

determines the common part

W^{*}

from its own value. Hence,

X

always reveals the block in which

Y

lies, but nothing more about

Y

, and vice versa.

Figure 6: The binary symmetric channel with crossover probability

a_{0}

that relates

X

and

Y

in the doubly symmetric binary source. Each bit of

X

is received correctly as

Y

with probability

1-a_{0}

and flipped with probability

a_{0}

Figure 7: Wyner’s optimal common variable

W

for the doubly symmetric binary source, shown as the intermediate variable in a cascade of two binary symmetric channels, each with crossover probability

a_{1}=\tfrac{1}{2}\big(1-\sqrt{1-2a_{0}}\big)

. Since

2a_{1}(1-a_{1})=a_{0}

, the cascade is equivalent to the single binary symmetric channel of Figure 6 , and

X

and

Y

are conditionally independent given

W

Panel of Figure 8 — Figure 8: The Gray–Wyner source coding network. An encoder observes $(X^{n},Y^{n})$ and describes them to two receivers using a common link of rate $R_{0}$ and two private links of rates $R_{1}$ and $R_{2}$ . Wyner’s common information $C(X;Y)$ is the minimum $R_{0}$ among achievable rate triples with $R_{0}+R_{1}+R_{2}=H(X,Y)$ .

Panel of Figure 9 — Figure 9: The naive Bayes model. A single latent variable $W$ generates every observation, and no edge joins the $X_{k}$ to one another, so they are mutually independent once $W$ is given. This is the conditional independence ( 17 ) that Wyner’s construction requires, and writing it out on a discrete joint PMF gives the nonnegative tensor factorization above, with one rank-one factor per value of $W$ .

Panel of Figure 10 — Figure 10: The single-component case, drawn for $\mathbf{X}_{1},\mathbf{X}_{2}\in\mathbb{R}^{3\times 2}$ . Blue refers to view 1 and green to view 2, for the planes, the circles, and the vectors. The constraints confine each canonical variate to the unit sphere of $\mathbb{R}^{N}$ , $\mathbb{S}^{N-1}$ , and to the column space of its own view. Hence, the two feasible sets are the circles where the sphere intersects those subspaces, and $\theta$ is the angle between a feasible pair. Minimizing $\|\mathbf{X}_{1}\mathbf{q}_{1}-\mathbf{X}_{2}\mathbf{q}_{2}\|_{2}^{2}$ closes that angle as far as the subspaces allow. Here, two planes in $\mathbb{R}^{3}$ share a line, so the circles intersect at the two antipodal points marked in red, each corresponding to an optimum, where $\theta=0$ and $\sigma_{1}=1$ . For visualization purposes, the figure ignores centering. With $N=3$ , centered columns lie in the two-dimensional subspace orthogonal to $\mathbf{1}$ , so two centered data matrices of rank two would share the same column space.

Panel of Figure 11 — Figure 11: Both canonical pairs at once, drawn for $\mathbf{X}_{1},\mathbf{X}_{2}\in\mathbb{R}^{3\times 2}$ . Along the line the two subspaces share, the canonical variates coincide, so $\theta_{1}=0$ and $\sigma_{1}=1$ , and this direction spans $\mathrm{col}(\mathbf{X}_{1})\cap\mathrm{col}(\mathbf{X}_{2})$ . The second pair cannot be brought into agreement, since each variate is confined to its own subspace and must be uncorrelated with the first, so it settles at the angle $\theta_{2}$ with $\sigma_{2}=\cos\theta_{2}<1$ . What CCA reports as common is the part of the spectrum at $\sigma_{i}=1$ . As in Figure 10 , the figure ignores centering.

Receiver	Designed Knowledge	Propagated Knowledge	Cost
CDMA, single antenna
Zero-forcing [ 104 ]	all signatures	none	detected users spread, rate $1/M$
Linear MMSE, closed form [ 105 ]	all signatures	received amplitudes, noise variance	detected users spread, rate $1/M$
Subspace blind multiuser detection [ 7 ]	own signature	none	detected users spread, rate $1/M$
Multiple antennas
Pilot-based MMSE (Section 4.3 )	pilots	none	pilot overhead, rate $1-N_{p}/N$

Table 3: Prior information required by each receiver, and its cost.

Panel of Figure 12 — Figure 12: The two views of the speech example over the same time interval. ( a ) The acoustic view, summarized by the first four mel-frequency cepstral coefficients (MFCCs), a standard description of the short-time spectrum of the sound. ( b ) The articulatory view, namely the trajectories of sensors on the tongue, the upper and lower lips, and the soft palate. The two views bear no obvious resemblance to each other, yet both change at the same moments, since the phoneme being produced shapes both.

Panel of Figure 13 — Figure 13: The two systems of assumptions, drawn for $K=2$ . Each view is produced from the common factor and its own private factor through the mixing $\bm{v}_{k}$ of ( 116 ). In ( a ) the three latent factors are mutually independent, which is ( 118 ) and the assumption of Lyu et al. In ( b ) two further edges are added, so each private factor may depend on the common one, which is ( 122 ) and the assumption of Karakasis and Sidiropoulos. The views stay conditionally independent given $\mathbf{g}$ in both cases, since every path between $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ passes through $\mathbf{g}$ .

Canonical Correlation Analysis

See all 10 papers

Feb 27, 2026

Provable Subspace Identification of Nonlinear Multi-view CCA

We investigate the identifiability of nonlinear canonical correlation analysis (CCA) in a multi-view setup, in which each view is generated by applying an unknown nonlinear map to a linear mixture of shared latent variables plus view-private noise. Rather than pursuing exact unmixing, which is known to be ill-posed under general nonlinear mixing, we instead reframe multi-view CCA as a basis-invariant subspace identification problem. Under suitable latent priors and spectral separation conditions, we prove that the pairwise population CCA objective recovers correlated signal subspaces up to view-wise orthogonal ambiguity. For

N \geq 3

views, their multi-view aggregation provably isolates the jointly correlated subspaces shared across all views while eliminating view-private variation. We further establish finite-sample statistical consistency guarantees by translating the concentration of empirical cross-covariances into explicit subspace error bounds via spectral perturbation theory. Experiments on synthetic and rendered image datasets support our theoretical findings and illustrate the necessity of the assumed conditions.

Zhiwei Han, Stefan Matthes, Hao Shen

Chair of Data Processing, Technical University of Munich, Germany · fortiss GmbH, Munich, Germany

Representation Identifiability Parameter Identifiability Multi-View Learning Statistical Learning Theory Canonical Correlation Analysis

Ask PDF

Oct 7, 2026

Unpaired Canonical Correlation Analysis

Canonical Correlation Analysis (CCA) is a fundamental method for multiview shared space learning. However, its strict reliance on paired data poses a significant limitation, as such data is often difficult to obtain or entirely unavailable. In this paper, we present Unpaired CCA (UCCA), a novel method that learns linear projections to maximize the correlation of the true underlying pairing without access to any paired samples during training. We first establish theoretical results connecting the Quadratic Assignment Problem (QAP) to CCA. Leveraging these theoretical insights, we derive a practical method to maximize correlation exclusively from unpaired data. To the best of our knowledge, UCCA is the first approach to learn maximally correlated projections in a strictly unpaired setting. We validate UCCA on real-world multi-modal datasets, demonstrating that it significantly outperforms recent unpaired alignment baselines in recovering the underlying true correlation. This work fills a critical gap between traditional statistical multiview learning and the growing field of unpaired data learning.

Nir Ben-Ari, Ronen Talmon, Uri Shaham

Department of Computer Science and AI Bar-Ilan University Ramat-Gan, Israel · Electrical and Computer Engineering Technion Haifa, Israel

Canonical Correlation Analysis Multi-View Learning Cross-Modal Alignment Multimodal Learning

Ask PDF

Oct 6, 2025

Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors

In this work, we establish the sufficient conditions under which nonlinear Canonical Correlation Analysis (CCA) recovers ground-truth latent factors up to an affine transformation. By transporting the analysis from the observation space to the source space, we extend classical statistical results on orthogonal polynomial expansions of bivariate distributions to representation learning, proving affine identifiability under specific distributional priors. We formally demonstrate that whitening is strictly necessary to ensure the boundedness and well-conditioning of the learned mappings. Furthermore, we bridge the gap between theory and practice by proving that ridge-regularized empirical CCA converges to its population counterpart in the finite-sample regime. Finally, our findings provide a rigorous theoretical foundation explaining the empirical success of recent correlation-based non-contrastive learning methods. Experiments on synthetic and rendered image datasets, alongside systematic ablations, validate the predicted recovery behavior and illustrate the failure modes that arise when the assumptions are violated.

Zhiwei Han, Stefan Matthes, Hao Shen

fortiss GmbH, Munich, Germany · Technical university of Munich

Representation Learning Parameter Identifiability Statistical Learning Theory Canonical Correlation Analysis

Ask PDF

Abstract

Canonical Correlation Analysis

Explore similar work

Provable Subspace Identification of Nonlinear Multi-view CCA

Unpaired Canonical Correlation Analysis

Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors