cs.LGNov 5, 2026

Gradient Boosting for Efficient Rank Discovery in Tensor Network Kernel Machines

Authors: Albert Saiapin, Kim Batselier

Organizations: Delft Center for Systems and Control Delft University of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Tensor network kernel machines (TNKMs) combine nonlinear features with low-rank tensor representations to model complex relationships efficiently. However, selecting an appropriate tensor rank remains challenging, as conventional rank-search methods require training multiple candidate models, increasing computational costs. In this work, we focus on a canonical polyadic (CP) tensor network parameterization and propose GB-CPKM, a gradient boosting framework that builds a higher-capacity predictor by sequentially adding fixed-rank learners. This approach provides a new perspective on rank selection by incrementally increasing model capacity, with ensemble size selected based on validation performance rather than through explicit rank search. Experiments on real-world regression benchmarks show that GB-CPKM trains up to 6x faster and uses up to 2x less peak memory than a conventional rank-selection method based on cross-validation, while achieving comparable predictive performance. These results highlight the potential of gradient boosting to improve the computational efficiency and scalability of tensor-network learning.

Figures & tables

Figure 1: Comparison of the training procedures for CP tensor network kernel machines (CPKMs) using gradient boosting (GB) and rank selection via validation (CV). (a) GB-CPKM incrementally builds an ensemble by training fixed-rank R^\hat{R} base learners and concatenating their factor matrices along the rank dimension to represent the combined model. The ensemble size MM is selected based on validation performance. (b) In contrast, CV-CPKM independently trains candidate models with different CP ranks and selects the rank that achieves the lowest validation error. In the diagrams, each base learner model is represented by DD factor matrices of size I×R^I\times\hat{R} , where II is the number of rows and R^\hat{R} is the CP rank. Different colors distinguish the trainable factor matrices. Factors in gray are fixed and are not updated.
Figure 2: Ablation study of base-learner rank in GB-CPKM on four real-world datasets, with each column corresponding to a dataset. The top row shows cumulative training time (in seconds), while the bottom row reports validation MSE on a logarithmic scale. Across the evaluated datasets, higher-rank base learners achieve better predictive performance while also reducing cumulative training time. However, rank selection should also account for memory requirements, which may constrain the use of higher-rank models in practice.
Figure 3: Comparison of GB-CPKM (red) and CV-CPKM (blue) on four real-world datasets, with each column corresponding to a dataset. From top to bottom, the rows show cumulative training time (in seconds), test MSE, and peak memory consumption (in MiB), plotted against the number of considered candidate CP ranks. Solid lines indicate the mean across 10 runs, and shaded regions represent ±\pm 1 standard deviation. GB-CPKM consistently requires less training time and peak memory than CV-CPKM, while achieving comparable predictive performance. The test MSE of GB-CPKM is similar to that of CV-CPKM, with overlapping variability bands in some cases, and approaches comparable values across the evaluated datasets. These results highlight the potential of GB-CPKM to reduce computational costs without substantially compromising predictive accuracy.
   MSE (test) ↓\downarrowTime (sec.) ↓\downarrowMemory (MiB) ↓\downarrow
DataNDCV-CPKMGB-CPKMGB-CPKM(H)CV-CPKMGB-CPKMCV-CPKMGB-CPKM
Energy76880.0026 ±0.000.0043±0.000.0038±0.000.14±0.020.02 ±0.01534444
Concrete103080.1618 ±0.050.1889±0.040.1801±0.050.18±0.000.06 ±0.01572461
Kin8nm819280.1077 ±0.010.1721±0.030.1209±0.011.28±0.030.33 ±0.02865551
Naval11934160.0000 ±0.000.0074±0.000.0010±0.001.78±0.060.36 ±0.021090630
Precipitation62847430.9419 ±0.040.9454±0.040.9448±0.0488.59±11.848.35 ±1.0528851156
Table 1: Performance comparison of GB-CPKM and CV-CPKM across regression benchmarks. We report test mean squared error (MSE), training time (seconds), and peak memory consumption (MiB). Each experiment is repeated over 10 runs, with results reported as mean ±\pm 1 standard deviation. NN denotes the number of samples, and DD denotes the input dimensionality. Overall, GB-CPKM trains up to 6×6\times faster and uses up to 2×2\times less peak memory, on average, than CV-CPKM, while achieving comparable test MSE. The hybrid variant, GB-CPKM(H), further improves predictive performance on most datasets by performing an additional full ALS sweep on the combined model.
More in this topic

Tensor Networks

Tensor networks represent high-dimensional data and quantum states as interconnected low-rank tensors to enable efficient computation, learning, and simulation.

See all 55 papers

Explore similar work

CardsList

  1. Bayesian Tensor Train Kernel Machines with Automatic Model Complexity Inference

    Nov 4, 2026Afra Kilic, Kim BatselierKernel MethodsBayesian Inference

    Delft Center for Systems and Control, Delft University of Technology, Delft, 2628 CN, The Netherlands

  2. Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression

    Mar 27, 2026Haoming Shi, Eric C. Chi, Hengrui LuoModel SelectionTensor Decomposition

    Department of Statistics, Rice University · School of Statistics, University of Minnesota

  3. Tensor Network Kernel Machines: A JAX Framework for Machine Learning and Nonlinear System Identification

    Aug 7, 2026Albert Saiapin, Kim BatselierTensor NetworksNonlinear System Identification

    Delft Center for Systems and Control, TU Delft, Netherlands