Conditional Moment Kernels: Joint Modeling of Expected Returns and Covariances in Unbalanced Panels

Damir Filipovic; Paul Schneider

stat.MEOct 29, 2024

Conditional Moment Kernels: Joint Modeling of Expected Returns and Covariances in Unbalanced Panels

Authors: Damir Filipovic, Paul Schneider

Organizations: École Polytechnique Fédérale de Lausanne and Swiss Finance Institute · Università della Svizzera italiana and Swiss Finance Institute

Abstract

We propose the conditional moment kernel (CMK), a joint model of the conditional mean and covariance matrix of asset returns given their characteristics. A single kernel function of the characteristics gives the means and second moments of all returns. The implied covariance matrices are positive semidefinite for any number and composition of assets, so the model applies directly to unbalanced panels. Its estimation is a convex program whose size does not depend on the number of assets. A floor at the scale of the idiosyncratic noise keeps the estimated systematic covariance away from zero where the data cannot resolve it, and a trace penalty selects the number of factors. A closed form, the CMK estimator, approximates the solution. For fixed features and hyper-parameters, we prove identification, consistency and finite-sample bounds. In out-of-sample tests on US stocks from 1972 to 2021, the CMK estimator fits the conditional moments significantly better than instrumented principal component analysis (IPCA) with the same characteristics. Almost all of the gain is in the covariance matrix, and nonlinear functions of the characteristics add to it. The implied unconditionally mean--variance efficient portfolios earn annualized Sharpe ratios of 4.0 to 5.1 before costs, close to those of IPCA and declining over time.

Figures & tables

model	# factors	$R^{2}$ (%)	$R^{2,2}$ (%)	score	total $R^{2}$ (%)	adj. total $R^{2}$ (%)	$\rho^{\bm{f}}$ (%)	diff. lin.	$t$	diff. Gauss.	$t$
CMK, linear, $m=20$	10.6	0.20	0.82	763	14.0	13.5	18.4	-54.3	-8.1	-114.5	-7.0
CMK, linear, $m=40$	15.3	0.23	0.84	791	15.1	14.1	19.9	-26.3	-8.2	-86.6	-6.3
CMK, linear, $m=95$	22.0	0.29	0.87	817	17.3	15.0	22.8	0.0	–	-60.2	-5.4
CMK, linear, $m$ selected	21.9	0.29	0.87	817	17.2	15.0	22.7	-0.1	-0.6	-60.3	-5.4
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=152$	21.5	0.35	0.96	844	19.5	15.9	24.2	26.2	6.8	-34.0	-4.3
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=202$	30.5	0.36	0.99	857	21.1	16.2	26.5	39.2	5.9	-21.0	-4.3

Table 1: Out-of-sample statistical performance on the 594 test months from July 1972 to December 2021. “# factors” is the average rank of the excess over the floor (an upper bound for the additive kernel, Online Appendix D ), and for IPCA the average rank of the factor covariance. “score” is the score differential, and

\rho^{\bm{f}}

the projected variance share. “diff. lin.” and “diff. Gauss.” are the average monthly score differences to the linear kernel with

m=95

and to the selected additive kernel, with Newey–West

t

-statistics. Every CMK row selects

(\eta,\lambda^{\rm sy})

by validation. IPCA has

K

factors and the same instruments. “linear + Gaussian” is the additive kernel, and

\ell=1\times

and

0.5\times

stand for

\ell_{0}

and

\ell_{0}/2

Figure 1: Rolling out-of-sample performance over 60-month windows. Top: annualized Sharpe ratio of the UMVE portfolio for the additive kernel (bandwidth and

m

selected), the linear kernel (

m=95

), IPCA (

K=60

) and the equally weighted portfolio. Middle: score differential. Bottom: score difference between the CMK specifications and IPCA. Shaded: the 1987 crash, the dot-com bubble, the global financial crisis and the COVID-19 crash, dated by Adrian et al. (2023) .

Step	Score	Change	$t$	SR
IPCA, $K=60$	785			4.55
IPCA, $K=95$	793	$+8.0$	$5.2$	4.49
Moment step, full feature space, Tikhonov $\varrho=10^{-3}$	808	$+15.5$	$2.4$	4.48
Noise correction: subtract $\hat{u}\bm{G}^{+}_{t,\varrho}$	798	$-10.4$	$-4.0$	4.44
Floor at $\eta=1$	814	$+16.3$	$4.1$	4.79
Select $(\eta,\lambda^{\rm sy})$ : CMK estimator	817	$+3.6$	$2.6$	4.91

Table 2: Decomposition of the gain over IPCA. Each row changes one step, from IPCA with

K=60

to the CMK estimator in the full linear feature space,

m=95

. Score differential, its change from the row above with the paired Newey–West

t

-statistic, and the annualized Sharpe ratio (SR) of the UMVE portfolio. Rows 3 to 5 have

\lambda^{\rm sy}=0

, rows 3 and 4 have

\eta=0

model	SR pred.	SR UMVE	$t$	SR CMVE	$p$ vs $1/N$	$p$ vs IPCA	intercept (%)	$t$ (int.)	adj. $R^{2}$ (%)	turnover	SR net 0	SR net 10	SR net 25	SR net 50	SR net 100
CMK, linear, $m=20$	5.72	3.96	14.4	3.77	0.00	0.07	37.7	13.5	3.0	0.74	3.87	3.59	3.16	2.41	0.81
CMK, linear, $m=40$	6.74	4.37	16.3	4.12	0.00	0.47	42.0	15.2	2.7	0.78	4.20	3.90	3.43	2.61	0.88
CMK, linear, $m=95$	8.53	4.91	18.7	4.48	0.00	0.02	47.8	17.7	2.0	0.93	4.91	4.52	3.91	2.85	0.61
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=295$	8.55	5.08	19.4	4.61	0.00	0.01	49.6	18.2	1.3	0.85	4.90	4.56	4.04	3.15	1.29
CMK, linear + Gaussian ( $\ell=0.5\times$ ), $m=295$	8.62	4.97	18.6	4.52	0.00	0.01	48.3	17.2	2.2	0.85	4.85	4.51	3.98	3.07	1.16
CMK, linear + Gaussian, $(\ell,m)$ selected	8.89	5.05	19.0	4.54	0.00	0.01	49.2	17.6	1.7	0.89	4.95	4.60	4.07	3.16	1.25

Table 3: Portfolios on the 594 test months. Predicted unconditional Sharpe ratio

\sqrt{12\,\bar{\theta}/(1-\bar{\theta})}

of ( 45 ), with the expectation in

\bar{\theta}

replaced by the average over the test months. Realized annualized Sharpe ratios of the UMVE portfolio (SR UMVE) and of the CMVE portfolio (SR CMVE). Newey–West

t

-statistic of the mean UMVE return and

p

-values of the Ledoit and Wolf (2008) test of its Sharpe ratio against

1/N_{t}

and against IPCA with

K=60

. Intercept (annualized, in percent), its

t

-statistic and adjusted

R^{2}

of the regression of the UMVE return, scaled to 10% annual volatility, on the five factors of Fama and French (2015) . Turnover and the net Sharpe ratios refer to the UMVE portfolio at unit gross exposure,

\sum_{i}|w_{t,i}|=1

: average monthly turnover

\sum_{i}|w_{t,i}-w_{t-1,i}|

, and Sharpe ratios before costs (“SR net 0”) and net of proportional costs of 10 to 100 basis points of turnover.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure C.1 : Simulation study. (a) Mean squared error

{\mathbb{E}}\|\bm{U}^{\ast}_{T}-\bm{U}^{\rm pop}\|^{2}

and its 95% quantile over 100 simulated panels per

T

for the Frobenius weight and the Gram-whitened weight (

m^{\rm sy}=20

\lambda^{\rm sy}=\lambda^{\rm id}=0

\eta=0

), with the fitted log-log slopes, and a plug-in evaluation of

\sigma^{2}/(\alpha^{2}T)

with simulated constants. (b) Number of factors and (c) squared distance to

\bm{U}^{\rm pop}

along the regularization path

\lambda^{\rm sy}

, for the Frobenius weight and for the Gram-whitened weight with

\eta=0

and

\eta=1

, for one panel of

T=108

and one of

T=1000

months. The dotted line in (b) is the true rank three.

DGP	model	$R^{2}$ (%)	$R^{2,2}$ (%)	score	total $R^{2}$ (%)	# factors	SR pred.	SR UMVE	SR CMVE
well specified	$m=5$	0.72	0.93	64.6	12.3	2.9	1.31	0.86	0.84
	$m=10$	0.92	1.44	85.3	16.5	3.8	1.51	0.86	0.84
	$m=20$	1.13	2.59	133.6	25.0	4.3	1.87	0.75	0.73
	population	1.31	2.67	134.5	25.0	3.0	1.12	1.27	1.27
misspecified	$m=5$	0.62	0.89	62.7	12.1	3.0	1.37	0.88	0.85
	$m=10$	0.75	1.33	81.3	15.9	4.1	1.53	0.78	0.77

Appendix

Table C.1 : Out-of-sample protocol on simulated data. Measures of Section 7.2 (in percent where indicated), average number of factors, and annualized predicted and realized Sharpe ratios of the UMVE and CMVE portfolios over 612 test months, for the solution of the CMK program with the Gram-whitened weight, with

m^{\rm sy}=5,10,20

(

(\eta,\lambda^{\rm sy})

by validation,

\lambda^{\rm id}=0

, re-estimation on 108 months) and for the population moments of ( C.1 ), under the well-specified DGP and under the misspecified DGP with three nonlinear features.

period	model	months	$R^{2}$ (%)	$R^{2,2}$ (%)	score	SR UMVE	SR $1/N$	diff. score	$t$
full	CMK, linear + Gaussian, $(\ell,m)$ selected	594	0.38	1.03	878	5.05	0.50	–	–
full	CMK, linear, $m=95$	594	0.29	0.87	817	4.91	0.50	-60.2	-5.4
full	IPCA, $K=60$	594	0.29	0.84	785	4.55	0.50	-93.2	-4.7
full	IPCA, $K=20$	594	0.28	0.83	774	4.74	0.50	-103.6	-5.2
1972–1989	CMK, linear + Gaussian, $(\ell,m)$ selected	210	0.38	1.97	1000	6.93	0.36	–	–
1972–1989	CMK, linear, $m=95$	210	0.28	1.94	945	6.84	0.36	-55.5	-3.2

Appendix

Table D.1 : Sub-periods. Predictive

R^{2}

and

R^{2,2}

, score differential and annualized Sharpe ratios of the UMVE and the equally weighted portfolio in the full sample, in three sub-periods and in the 57 crash months of Figure 1 , for the selected additive-kernel specification, the linear kernel with

m=95

and IPCA with

K=20

and

K=60

. The last two columns give the average monthly score difference to the selected additive-kernel specification and its Newey–West

t

-statistic.

specification	# factors	$R^{2}$ (%)	$R^{2,2}$ (%)	score	total $R^{2}$ (%)	SR UMVE	SR CMVE	diff. score	$t$	$p$ SR
linear $m=95$ : $(\eta,\lambda^{\rm sy})$ selected (Table 1 )	22.0	0.29	0.87	817	17.3	4.91	4.48	–	–	–
linear $m=95$ : $\eta=0$ , $\lambda^{\rm sy}=0$	45.1	0.29	0.84	798	17.3	4.44	4.22	-19.9	-4.7	0.00
linear $m=95$ : $\eta=1$ , $\lambda^{\rm sy}=0$	28.9	0.29	0.84	814	17.3	4.79	4.52	-3.6	-2.6	0.13
linear $m=95$ : $\eta=0$ , $\lambda^{\rm sy}$ selected	41.6	0.29	0.85	798	17.3	4.43	4.34	-19.8	-4.7	0.00
linear $m=95$ : moment step, $\hat{\bm{F}}$ in every direction	94.2	0.29	0.84	808	17.3	4.48	4.27	-9.5	-4.8	0.00
linear $m=95$ : identity-shaped floor, $(\eta,\lambda^{\rm sy})$ selected	27.5	0.29	0.86	808	17.3	4.63	4.03	-9.4	-8.4	0.13

Appendix

Table D.2 : Robustness. Out-of-sample measures of Table 1 , annualized Sharpe ratios, and the average monthly score difference to the reference row of the block with its Newey–West

t

-statistic and the

p

-value of the Ledoit and Wolf (2008) test of equal UMVE Sharpe ratios. First block, in three panels: the regularizers, for the linear kernel with

m=95

and the additive kernel with

m=295

at both bandwidths, with the CMK estimator of Table 1 as reference, the CMK estimator without regularizers, the floor alone, the penalty alone, the moment step that adds

\hat{\bm{F}}

\bm{V}

in every direction instead of flooring the excess, and the identity-shaped floor

\eta(\tr\hat{\bm{F}}/m)\bm{I}_{m}

of Section 4.3 , with

(\eta,\lambda^{\rm sy})

selected and at

\eta=1

\lambda^{\rm sy}=0

(for the additive kernel at

\ell=\ell_{0}

). Second block: the weight and the shape of the floor together, for the linear kernel with

m=95

on every fourth window, the January, May and September test months, the Gram-whitened weight and the scalar weight

N_{t}^{-1/2}\bm{I}_{N_{t}+1}

with the floor

\eta\hat{\bm{F}}

and with the identity-shaped floor,

(\eta,\lambda^{\rm sy})

selected, and the scalar weight without regularizers, computed by the solver of Appendix B (its rows carry no count of the excess). Third block: the weight. At

m=95

the Browne weight against the Gram-whitened weight, both without regularizers. At

m=40

on every second window, the Gram-whitened, the Frobenius (first-moment weight

\kappa

) and the Browne weights, all without regularizers, against the Gram-whitened weight with

(\eta,\lambda^{\rm sy})

selected. Fourth block: the protocol, without re-estimation on the validation months (with the corresponding rows of Table 1 as references), with a validation window of 36 months and a training window of 72 months, with one validation month and 107 training months, and with the smallest 20% of stocks by market capitalization removed from estimation and evaluation. Fifth block: the one-step estimator ( 35 ), computed by the solver of Appendix B , against the CMK estimator, at

\eta=1

and

\lambda^{\rm sy}=0

, on every fourth window. 594 test months unless noted.

specification	months	score (0)	factors (0)	$R^{2,2}$ (0)	score (0.5)	factors (0.5)	$R^{2,2}$ (0.5)	score (1)	factors (1)	$R^{2,2}$ (1)	score (2)	factors (2)	$R^{2,2}$ (2)
linear $m=95$ , floor $\eta\hat{\bm{F}}$	594	798	45.1	0.84	813	35.9	0.84	814	28.9	0.84	807	19.8	0.84
linear $m=95$ , identity-shaped floor	594	798	45.1	0.84	801	24.0	0.84	783	17.4	0.83	755	11.2	0.76
linear + Gaussian ( $\ell=1\times$ ) $m=295$ , floor $\eta\hat{\bm{F}}$	594	845	90.3	0.83	854	77.4	0.80	854	66.7	0.77	848	50.3	0.70
linear + Gaussian ( $\ell=1\times$ ) $m=295$ , identity-shaped floor	594	845	90.3	0.83	622	62.6	-25.29	566	53.8	-104.62	507	41.0	-424.24

Appendix

Table D.3 : The shape of the floor along the multiplier grid at

\lambda^{\rm sy}=0

. Score differential, number of factors and predictive

R^{2,2}

(in percent) of the floor

\eta\hat{\bm{F}}

and of the identity-shaped floor

\eta(\tr\hat{\bm{F}}/m)\bm{I}_{m}

at the multipliers

0

\tfrac{1}{2}

1

and

2

, for the linear kernel with

m=95

and for the additive kernel with

m=295

\ell=\ell_{0}

, on the 594 test months.

	GMV volatility				Variance ratio		Score, difference to linear $m=95$
	1972–2021	$t$	1985–2021	$t$	GMV	UMVE	Cov.	$t$	Mean	$t$
CMK, linear, $m=20$	9.06	1.5	8.35	1.6	1.59	1.95	$-51.9$	$-7.7$	$-2.4$	$-6.1$
CMK, linear, $m=40$	8.87	0.7	8.21	1.2	1.71	2.01	$-24.5$	$-7.5$	$-1.9$	$-5.7$
CMK, linear, $m=95$	8.71		7.97		1.93	2.46
CMK, linear + Gaussian, $(\ell,m)$ selected	9.03	1.8	8.35	1.9	2.43	3.09	$+58.3$	5.2	$+1.9$	4.5
IPCA, $K=20$	9.01	0.8	8.55	2.7	4.27	1.73	$-42.0$	$-4.6$	$-1.7$	$-4.3$

Appendix

Table D.4 : Covariance forecasts. Annualized out-of-sample volatility (in percent) of the GMV portfolio over the 594 test months and over the 444 test months with returns from January 1985 on, with the Newey–West

t

-statistic of the paired difference of squared demeaned returns to the linear kernel with

m=95

. Variance ratio: sample variance of the GMV and the UMVE return divided by its predicted variance. Covariance part and mean part of the score differential, as differences to the linear kernel with

m=95

, with Newey–West

t

-statistics. Every CMK row selects

(\eta,\lambda^{\rm sy})

by validation, and the selected additive kernel also selects the bandwidth and

m

, as in Table 1 . A positive

t

-statistic of the GMV volatility means a higher variance than the linear kernel with

m=95

. Variance ratios and score parts are over the 594 test months. The IPCA rows come from a rerun whose score differentials differ from those of Table 1 by at most 0.3 points.

model	# factors	$R^{2}$ (%)	$R^{2,2}$ (%)	score	total $R^{2}$ (%)	adj. total $R^{2}$ (%)	diff. lin.	$t$	diff. Gauss.	$t$	SR UMVE	max DD (%)	turnover	SR net 0	SR net 25
CMK, linear, $m=20$	11.9	0.17	1.36	870	17.2	16.7	-79.8	-8.4	-136.3	-7.5	3.95	-13.6	0.88	3.88	2.90
CMK, linear, $m=40$	17.7	0.18	1.34	898	18.6	17.5	-51.2	-7.4	-107.7	-6.7	4.22	-21.3	0.87	4.11	3.13
CMK, linear, $m=122$	26.0	0.28	1.40	949	22.5	19.4	0.0	–	-56.5	-5.3	4.98	-13.0	0.93	4.89	3.78
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=179$	22.1	0.34	1.52	973	24.8	20.3	23.8	6.5	-32.7	-4.5	5.18	-12.9	0.92	5.12	4.12
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=229$	29.2	0.35	1.52	984	26.5	20.7	34.5	5.4	-21.9	-4.8	5.39	-14.1	0.91	5.26	4.25
CMK, linear + Gaussian ( $\ell=1\times$ ), $m=322$	46.4	0.36	1.52	998	29.3	21.0	48.6	4.1	-7.8	-2.7	5.57	-13.5	0.89	5.39	4.32

Appendix

Table D.5 : The panel of Jensen et al. (2023) . Out-of-sample measures of Table 1 , the annualized Sharpe ratio of the UMVE portfolio, its maximum drawdown (“max DD”, in percent and negative, the largest loss from a peak of the cumulative return, compounded at 10% annual volatility), its average monthly turnover at unit gross exposure and its Sharpe ratio at unit gross exposure before costs (“SR net 0”) and net of proportional transaction costs of 25 basis points of turnover, as in Table 3 , on the panel described in the text. The linear kernel with

m=122

is the full linear feature space, the reference of the column “diff. lin.”, the additive kernel with

m=322

the full additive feature space, and the selected additive kernel the reference of the column “diff. Gauss.”. 647 test months, February 1972 to December 2025.

Step	Score	Change	$t$	SR
IPCA, $K=60$	895			4.93
Moment step, full feature space, Tikhonov $\varrho=10^{-3}$	929	$+33.9$	$3.9$	4.73
Noise correction: subtract $\hat{u}\bm{G}^{+}_{t,\varrho}$	915	$-13.6$	$-4.9$	4.71
Floor at $\eta=1$	943	$+27.7$	$5.2$	5.03
Select $(\eta,\lambda^{\rm sy})$ : CMK estimator	949	$+6.4$	$4.0$	4.98

Appendix

Table D.6 : Decomposition of the gain over IPCA on the second panel, as in Table 2 , from IPCA with

K=60

to the CMK estimator in the full linear feature space,

m=122

. Score differential, its change from the row above with the paired Newey–West

t

-statistic, and the annualized Sharpe ratio (SR) of the UMVE portfolio. Rows 2 to 4 have

\lambda^{\rm sy}=0

, rows 2 and 3 have

\eta=0

Quantitative Finance

Quantitative finance uses mathematical models, statistics, and machine learning to value assets, measure risk, and guide trading and portfolio decisions.

See all 191 papers

End-to-End Neural Shrinkage of Indefinite Pairwise Correlation Matrices for Small-Cap-Inclusive Portfolios
Aug 31, 2026Christian Bongiorno, Lorenzo VillasseroQuantitative Finance Portfolio Optimization
Université Paris-Saclay, CentraleSupélec · Mathématiques et Informatique pour la Complexité et les Systèmes · Gif-sur-Yvette, France +2
The Decision Geometry of Covariance Estimation for the Global Minimum-Variance Portfolio under Heavy Tails
Jun 25, 2026Xavier FonsecaRegret Minimization Portfolio Optimization
Academy for AI, Games and Media, Breda University of Applied Sciences, Breda, The Netherlands.
A Hybrid Gaussian Process Regression Framework for Stable Volatility-Covariance Estimation: Evidence from Global Equity Indices
May 17, 2026Ujjwala VadrevuQuantitative Finance Financial Time Series

Aug 31, 2026J/K move · Enter open · S save

End-to-End Neural Shrinkage of Indefinite Pairwise Correlation Matrices for Small-Cap-Inclusive Portfolios

Christian Bongiorno, Lorenzo Villassero

Université Paris-Saclay, CentraleSupélec · Mathématiques et Informatique pour la Complexité et les Systèmes · Gif-sur-Yvette, France+2

Quantitative Finance Portfolio Optimization Covariance Estimation

Open paper AskPDF arXiv

Small-cap-inclusive equity universes contain recently listed and intermittently traded securities, so enforcing a common look-back discards a substantial fraction of the available information. Pairwise-complete estimation preserves the longest overlap for each asset pair, but the resulting correlation matrix can be indefinite because its entries are computed on different samples. This prevents direct use in Markowitz optimization and falls outside the assumptions of standard random-matrix shrinkage. We adapt a rotation-invariant neural covariance estimator to this setting. The model computes mask-aware marginal moments and a pairwise correlation matrix proxy, processes its signed spectrum, and uses a bidirectional gated recurrent unit conditioned on factor-aligned effective sample lengths derived from the overlap matrix and eigenvector loadings. It maps all eigenvalues, including negative ones, to a positive inverse spectrum. The reconstructed covariance is positive definite and is trained end-to-end to minimize five-day realized global-minimum-variance risk. We evaluate 26 expanding-window models from 2000 to 2025 on up to 1,500 U.S. equities in a closing-auction simulator with point-in-time selection, commissions, financing, corporate actions, and market impact. Across the 26-year out-of-sample period, the neural estimator reduces annualized five-day volatility by approximately 20% and increases the Sharpe ratio by approximately 40% relative to the next-best covariance estimator. These improvements are consistent across realized risk, risk-adjusted performance, and drawdown control, remain after the modeled execution frictions, and are supported by a 99.9% Model Confidence Set that retains only the neural estimator.

Abstract

Quantitative Finance

Explore similar work

End-to-End Neural Shrinkage of Indefinite Pairwise Correlation Matrices for Small-Cap-Inclusive Portfolios

The Decision Geometry of Covariance Estimation for the Global Minimum-Variance Portfolio under Heavy Tails

A Hybrid Gaussian Process Regression Framework for Stable Volatility-Covariance Estimation: Evidence from Global Equity Indices