Universal Distributed Feature Extraction Complementary to Side Information via H-Score

Yuan Xinjie; Dier Tang; Shao-Lun Huang

cs.ITOct 12, 2026

Universal Distributed Feature Extraction Complementary to Side Information via H-Score

Authors: Yuan Xinjie, Dier Tang, Shao-Lun Huang

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Department of Mathematics, The University of Hong Kong, Hong Kong, China

Abstract

This paper considers a distributed estimation system where edge nodes transmit averaged features to a center node for parameter estimation. A key challenge arises when the center node possesses auxiliary side information: how to design local extractors that are universally optimal for any downstream task, while avoiding redundancy with the side information. To tackle this problem, we orthogonally decompose the joint function space and demonstrate in the weak-dependence regime that the modal decomposition of the canonical dependence kernel's conditional-additive component yields optimal, task-independent features. We construct a distributed nested H-score to learn this decomposition and show that, under a limited feature budget, the mean-squared error (MSE) gap to the best task-specific features is determined solely by the task's alignment with the discarded, least informative modes, and vanishes to leading order for every square-integrable task once the budget is sufficient.

Figures & tables

Panel of Figure 1 — Fig. 1: System model: node $k$ transmits the block average $U^{(k)}$ of its local features, and the center node, which alone knows $S$ , forms the estimate $\hat{\psi}(U,S)$ .

Fig. 2: The nested H-score network for the modal decomposition of the Markov-additive component

\pi_{\mathsf{M}}

and the conditional-additive component

\pi_{\mathsf{C}}

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 804 papers

Fourier Preconditioning for Neural Feature Learning
Jul 2, 2026Preston Pitzer, Anish Pradhan, Harpreet S. DhillonFrequency-Domain Feature Learning Predictive Representation Learning
On the Power of Source Screening for Learning Shared Feature Extractors
Feb 18, 2026Leo Muxing Wang, Connor Mclaughlin, Lili SuShared Representation Learning Representation Learning
Northeastern University
Decentralized Multitask Learning over Learned Task Graphs
Aug 27, 2026Zirui Wan, Stefan VlaskiGraph Structure Learning Multi-Task Learning
Department of Electrical and Electronic Engineering, Imperial College London, UK

Jul 2, 2026J/K move · Enter open · S save

Fourier Preconditioning for Neural Feature Learning

Preston Pitzer, Anish Pradhan, Harpreet S. Dhillon

Frequency-Domain Feature Learning Predictive Representation Learning Invariant Representation Learning

Open paper AskPDF arXiv

Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.

Abstract

Representation Learning

Explore similar work

Fourier Preconditioning for Neural Feature Learning

On the Power of Source Screening for Learning Shared Feature Extractors

Decentralized Multitask Learning over Learned Task Graphs