Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network

We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.

Large deep neural networks are costly to store and deploy because inference must move and evaluate many parameters. This paper studies \emph{Automatically Differentiable Nonlinear Tensor Networks} (ADNTNs), compact differentiable weight generators for replacing selected dense, convolutional, and attention layers. An ADNTN maps a small set of trainable tensor cores to a full weight tensor through hierarchical contractions and learnable nonlinearities; the generated layer is then used as an ordinary linear or convolutional operator. We investigate three multilayered topologies: Tree Tensor Networks, augmented Tree Tensor Networks with boundary disentanglers, and MERA-style multi-scale decoders. Compared with flat brick-wall automatically differentiable tensor networks, these hierarchies provide logarithmic-depth communication between tensorised modes and optional lateral mixing, which can improve long-range structure without large increases in stored parameters. We give a unified forward--adjoint formulation showing how reverse-mode automatic differentiation computes pre-activation adjoints and contracted-environment gradients for all trainable cores. The formulation supports task losses, reconstruction losses, distillation, quantisation-aware terms, batching, and modern optimisers. Proof-of-concept experiments on selected AlexNet and VGG-16 layers on CIFAR-10 datasets achieve per-layer parameter-compression ratios from about $2{,}000\times$ to $430{,}000\times$. Several VGG-16 compressed models match or slightly exceed the dense baseline, whereas AlexNet shows moderate degradation under more restrictive redundancy. These results indicate that nonlinear tensor-network generators are a promising structured route to compact pattern-recognition models, while also showing that contraction schedules and hardware-aware implementations remain essential for practical speedups.

Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly understood. In this work, we precisely characterize how the function space spanned by the features of a two-layer neural network evolves during gradient descent training. We prove that, in the high-dimensional proportional regime, after a large gradient step the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance. This induces a data-adaptive kernel that reshapes the function space and modifies its spectral structure. Our analysis reveals that feature learning can be interpreted as a distributional transformation in either parameter space or input space, equivalently as the introduction of a target-dependent kernel. In particular, it selectively amplifies eigenvalues aligned with the target direction and mixes leading eigenfunctions, coupling the top radial mode with a target-aligned quadratic harmonic. Overall, our results provide a precise function-space perspective on early-stage feature learning: rather than just rescaling a fixed kernel, gradient descent induces a data-adaptive deformation that preferentially enhances directions aligned with the signal in the data.