From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

Paris A. Karakasis; Walid Saad

Figure 1: Two neighboring cells, each served by a multi-antenna base station. A user near the center of a cell is received strongly at its own station and is barely heard at the other, so only one link is drawn for it. The two cell-edge users are received at low power by both stations, so each of them carries a link to BS-1 and a link to BS-2, and they are the challenging case for decoding.

Work	Info. Theory	Statistics	Sig. Proc.	Multi-View CCA	Nonlinear CCA	Deep CCA	Identif.
Borga [ 14 ]	Partial	✓	Partial
Hardoon et al. [ 15 ]		✓		Partial	✓
Uurtio et al. [ 16 ]		✓			✓	Partial
Wang et al. [ 17 ]		✓		Partial	Partial	Partial	Partial
Yang et al. [ 23 ]		✓		✓	✓	✓	Partial
Yu and Tan [ 24 ]	✓

Table 1: Scope of existing treatments of common information and CCA. A check mark means the topic is covered substantively, and Partial means it is touched on without being developed.

Term	Meaning
Shared structure	Informal term for whatever the views have in common, used only before one of the definitions below is adopted.
Common information	A number, measured in bits and computed from the joint distribution of the views, namely the Gács–Körner or the Wyner common information. The number says how much the views share, not what they share.
Common factor	A latent random vector shared by all views in a multiview generative model, such that the views are conditionally independent given the common factor. Whatever else enters a view is private to that view and forms its private factor.
Common subspace	The column space of the matrix whose rows are the values of the common factor at the training samples.
Common representation	What the outputs of the learned encoders of the views are made to agree on.

Table 2: Terminology used throughout the tutorial.

Figure 2: Two perception systems at an intersection label the same objects with different label sets. A third system maps the labels of System A through

f

and the labels of System B through

g

to a common label set. The finest labeling on which the two maps always agree is light vehicle, heavy vehicle and person.

Figure 3: One speaker recorded by two microphones. Each recording contains the voice, which is common to both, and the noise of its own microphone. The recordings are dependent only through the voice, so given the voice the two recordings are conditionally independent.

Figure 4: Partitions of the sample space

\Omega=[0,1]^{2}

, with outcome

\omega=(\omega_{1},\omega_{2})

, into the atoms of

\sigma(X)

and

\sigma(Y)

, the sigma-algebras generated by

X

and

Y

, of their join

\sigma(X)\vee\sigma(Y)=\sigma(X,Y)

, and of their meet

\sigma(X)\cap\sigma(Y)

. Each color marks one atom, that is, the set of all outcomes at which the random variable generating the partition takes one particular value. The join is the coarsest partition that refines both

\sigma(X)

and

\sigma(Y)

. The meet is the finest partition coarser than both, and the five atoms of the meet index the values of the common part

W^{*}

.

Figure 5: Joint PMF

P_{XY}

in the case

K_{GK}(X;Y)=I(X;Y)=C(X;Y)

, with rows and columns permuted so that

P_{XY}

is block-diagonal. The block in blue corresponds to

W^{*}=1

, the block in orange to

W^{*}=2

, and darker shades indicate larger probabilities. Each block equals

P_{W^{*}}(w)

times the outer product of the conditional PMFs

P_{X|W^{*}=w}

and

P_{Y|W^{*}=w}

, as the arrows indicate, which reflects the conditional independence of

X

and

Y

given

W^{*}

. The two blocks occupy disjoint rows and disjoint columns, so each of

X

and

Y

determines the common part

W^{*}

from its own value. Hence,

X

always reveals the block in which

Y

lies, but nothing more about

Y

, and vice versa.

Figure 6: The binary symmetric channel with crossover probability

a_{0}

that relates

X

and

Y

in the doubly symmetric binary source. Each bit of

X

is received correctly as

Y

with probability

1-a_{0}

and flipped with probability

a_{0}

.

Figure 7: Wyner’s optimal common variable

W

for the doubly symmetric binary source, shown as the intermediate variable in a cascade of two binary symmetric channels, each with crossover probability

a_{1}=\tfrac{1}{2}\big(1-\sqrt{1-2a_{0}}\big)

. Since

2a_{1}(1-a_{1})=a_{0}

, the cascade is equivalent to the single binary symmetric channel of Figure 6 , and

X

and

Y

are conditionally independent given

W

.

Figure 8: The Gray–Wyner source coding network. An encoder observes

(X^{n},Y^{n})

and describes them to two receivers using a common link of rate

R_{0}

and two private links of rates

R_{1}

and

R_{2}

. Wyner’s common information

C(X;Y)

is the minimum

R_{0}

among achievable rate triples with

R_{0}+R_{1}+R_{2}=H(X,Y)

.

Figure 9: The naive Bayes model. A single latent variable

W

generates every observation, and no edge joins the

X_{k}

to one another, so they are mutually independent once

W

is given. This is the conditional independence ( 17 ) that Wyner’s construction requires, and writing it out on a discrete joint PMF gives the nonnegative tensor factorization above, with one rank-one factor per value of

W

.

Figure 10: The single-component case, drawn for

\mathbf{X}_{1},\mathbf{X}_{2}\in\mathbb{R}^{3\times 2}

. Blue refers to view 1 and green to view 2, for the planes, the circles, and the vectors. The constraints confine each canonical variate to the unit sphere of

\mathbb{R}^{N}

,

\mathbb{S}^{N-1}

, and to the column space of its own view. Hence, the two feasible sets are the circles where the sphere intersects those subspaces, and

\theta

is the angle between a feasible pair. Minimizing

\|\mathbf{X}_{1}\mathbf{q}_{1}-\mathbf{X}_{2}\mathbf{q}_{2}\|_{2}^{2}

closes that angle as far as the subspaces allow. Here, two planes in

\mathbb{R}^{3}

share a line, so the circles intersect at the two antipodal points marked in red, each corresponding to an optimum, where

\theta=0

and

\sigma_{1}=1

. For visualization purposes, the figure ignores centering. With

N=3

, centered columns lie in the two-dimensional subspace orthogonal to

\mathbf{1}

, so two centered data matrices of rank two would share the same column space.

Figure 11: Both canonical pairs at once, drawn for

\mathbf{X}_{1},\mathbf{X}_{2}\in\mathbb{R}^{3\times 2}

. Along the line the two subspaces share, the canonical variates coincide, so

\theta_{1}=0

and

\sigma_{1}=1

, and this direction spans

\mathrm{col}(\mathbf{X}_{1})\cap\mathrm{col}(\mathbf{X}_{2})

. The second pair cannot be brought into agreement, since each variate is confined to its own subspace and must be uncorrelated with the first, so it settles at the angle

\theta_{2}

with

\sigma_{2}=\cos\theta_{2}<1

. What CCA reports as common is the part of the spectrum at

\sigma_{i}=1

. As in Figure 10 , the figure ignores centering.

Receiver	Designed Knowledge	Propagated Knowledge	Cost
CDMA, single antenna
Zero-forcing [ 104 ]	all signatures	none	detected users spread, rate $1/M$
Linear MMSE, closed form [ 105 ]	all signatures	received amplitudes, noise variance	detected users spread, rate $1/M$
Subspace blind multiuser detection [ 7 ]	own signature	none	detected users spread, rate $1/M$
Multiple antennas
Pilot-based MMSE (Section 4.3 )	pilots	none	pilot overhead, rate $1-N_{p}/N$

Table 3: Prior information required by each receiver, and its cost.

Figure 12: The two views of the speech example over the same time interval. ( a ) The acoustic view, summarized by the first four mel-frequency cepstral coefficients (MFCCs), a standard description of the short-time spectrum of the sound. ( b ) The articulatory view, namely the trajectories of sensors on the tongue, the upper and lower lips, and the soft palate. The two views bear no obvious resemblance to each other, yet both change at the same moments, since the phoneme being produced shapes both.

Figure 13: The two systems of assumptions, drawn for

K=2

. Each view is produced from the common factor and its own private factor through the mixing

\bm{v}_{k}

of ( 116 ). In ( a ) the three latent factors are mutually independent, which is ( 118 ) and the assumption of Lyu et al. In ( b ) two further edges are added, so each private factor may depend on the common one, which is ( 122 ) and the assumption of Karakasis and Sidiropoulos. The views stay conditionally independent given

\mathbf{g}

in both cases, since every path between

\mathbf{x}_{1}

and

\mathbf{x}_{2}

passes through

\mathbf{g}

.

From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

Abstract

Figures & tables

Canonical Correlation Analysis

Provable Subspace Identification of Nonlinear Multi-view CCA

Unpaired Canonical Correlation Analysis

Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors

Abstract

Canonical Correlation Analysis

Explore similar work

Provable Subspace Identification of Nonlinear Multi-view CCA

Unpaired Canonical Correlation Analysis

Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors