Iterative tensor network transformations for element-wise evaluation of elementary and filtering functions
Authors: Xiao Wang, Tomohiro Hashizume, Pia Siegl, Dieter Jaksch
Organizations: Clarendon Laboratory, University of Oxford, Parks Road, Oxford OX1 3PU, United Kingdom · The Hamburg Centre for Ultrafast Imaging, Luruper Chaussee 149, Hamburg 22761, Germany. · Institute for Quantum Physics, University of Hamburg, Luruper Chaussee 149, Hamburg 22761, Germany · Institute of Software Methods for Product Virtualization, German Aerospace Center (DLR), Nöthnitzer Straße 46b, 01187 Dresden, Germany
Tensor networks are powerful formats for compressing large-scale data. However, their application to general data processing has been limited by the difficulty of performing nonlinear operations. Here, we introduce iterative tensor network transformations (ITNTs), a general algorithmic framework for the element-wise evaluation of elementary and nonlinear filtering functions on data encoded as tensor trains (TTs), a class of tensor networks. Our approach operates entirely in the compressed domain, enabling efficient computation on exponentially large datasets while maintaining a controlled computational cost. We demonstrate its power in two key areas: (I) evaluating highly nonlinear elementary and filtering functions on a 3D reactive flow field, enabling high-fidelity reaction rate computation and region filtering, and (II) finding extrema in complex optimization problems, such as solving Max-SAT instances on spaces up to 270 configurations. These results establish ITNT as a foundational tool that provides tensor network methods with the capability for general-purpose data science and large-scale optimization.
Figures & tables
Figure 1: Overview of the results. (a) Schematic of the Iterative tensor network transformations (ITNTs) methodology. The framework enables the evaluation of highly complex nonlinear transformations, T , directly on TT-encoded data Ψ defined on array indices j∈{0,1,…,N−1=2n−1} . This is achieved by leveraging 2n parallel evaluations of the iteratively converging Newton-Raphson method (bottom-right panels). In these panels, the red curve represents the nonlinear function being iteratively solved, while the blue and green lines illustrate the successive tangent-linear approximations during the iterative process, starting from the blue dot. This convergence allows the transformation to be applied while maintaining the data in a compressed format. (b) Validation of the framework on a simulated methane/air jet flame ( 26 ) . The ITNT method is used to compute the highly nonlinear Arrhenius reaction rate ( exp(−Ec/T) ). The plot shows the iso-rate surface at 10% of the maximum reaction rate, computed directly from a compressed temperature field T [K] using a normalized reaction energy of Ec=17865.2 [K]. The comparison uses a TT bond dimension of χ=200 , requiring only 0.4The results demonstrate high-fidelity reconstruction, with ITNT achieving enclosed volume relative error approximately 100 times lower than that of TCI, representing a significant improvement in capturing the flame’s topological features. (c) Schematic of combinatorial optimization utilizing ITNT. The framework uses iterative nonlinear operations, specifically self-multiplication, to amplify and sharpen dominant configurations, such as the global maximum ( Eopt ) and sub-optimal peaks ( Esub ), within the problem-encoded energy landscape. Simultaneously, TT truncation (depicted by the lightning bolt) prunes away the rugged bulk spanned by the low-energy configurations by discarding the subspace corresponding to smaller singular values. This allows the algorithm to isolate optimal solutions while maintaining the data in a compressed TT format.
Figure 2: Nonlinear post processing with ITNTs. (a) Convergence comparison between ITNT and TCI for the Arrhenius equation (Eq. ( 3 )) computation. While TCI suffers from instability around χ=200 , ITNT demonstrates a drop in error in the same region, consistently converging towards the exact result with increasing χ . (b) Relative ℓ2 error of the 1/T field computed exactly ( Ψexact ) from the original field and by using the ITNT algorithm ( ΨTT ) for χ=50 , 100 , 200 , 300 , and 500 (light to dark, left panel), and the same error at the 10th iteration for different bond dimensions. In the right panel, the blue dashed line indicates 1/χ decay as a guide to the eye. (c) The position of the maximum temperature is indicated by a red dot and the orange region represents the isothermal surface at 90 % of the maximum temperature ( 0.9max{T} ). (d) Relative error of the maximum temperature, 1−max{TTT}/max{Texact} , found using ITNT for χ=50 , 100 , 200 , 300 , and 500 (light to dark). (e) Relative error of the volume of the region where T>0.9max{T} for χ=50 , 100 , 200 , 300 , and 500 (light to dark). Here, the value of 0.9max{T} is computed with the maximum-finding ITNT and the volume is determined by constructing the filter with sign-transformation ITNT.
Figure 3: Solving Max-3-SAT problems with ITNTs. (a) Truncation error at each bond in the zip-up self-multiplication algorithm. In order to solve 70 variables 700 clauses Max-3SAT problem, 10 self-multiplications are performed, and for each self-multiplication is computed with the zip-up algorithm with on-site truncation, while fixing χ=700 . Each color corresponds to a different self-multiplication iteration (iterations 1 to 10, light to dark) (b) Deflation result for the max 3-SAT problem s3v70c700-1 . Three configurations with an energy of E with E=679 are found after the i=1655 th , the i=6442 nd , and the i=30186 th extremum-finding ITNT steps. (c) Scaling of the minimum bond dimension for successful optimization and solution certification. Here, we generate 100 random n -variable, M=αn -clause Max-3SAT problems for n=10 to 20 , at clause densities α=8 and α=3 . Orange dashed lines show the minimum bond dimension required for the maximum search to return the correct optimum after P=⌈log2(nM)⌉+1 self-multiplication steps. The blue line shows the minimum bond dimension required for the truncation budget in Method V.7 to be obeyed after P steps. The required bond dimension grows with n , saturating the bond dimension ∼2n/2 (black dotted line).
Figure 4: ITNTs for element-wise nonlinear operations in TT representation. (a) Extrema detection with TT. i Starting from a TT representation of Ψ (black curve), repeated element-wise self-multiplication sharpens the state into a multi-peak profile Φ , where its integral is dominated by extremal configurations (red curve). ii A peak is then located by a binary fixation procedure. This involves integrating out all unfixed sites and choosing the local configuration that maximizes the summed value. Starting from every index not being fixed, in the leftmost panel, a comparison between j2=0 (orange) and j2=1 (cyan) is performed, identifying j2=1 as a candidate for the optimal configuration. In the subsequent next panel, j1=0 (orange) and j1=1 (cyan) are compared and j1=0 is selected, while the integrated out region from the fixation of j2=0 is indicated in gray. Continuing these steps with a random sequence of local tensors identifies the index I of the maximum (orange, right most panel). iii The deflation process then removes the identified peak to ensure that subsequent applications find the true maximum. (b) Element-wise evaluation of sign function. A highly nonlinear, non-analytic element-wise filter is realized through the composition of iterative applications of the functions F (Eq. ( 1 )) and G (Eq. ( 16 )) that produce an element-wise sign function Sign{f}[j]=Sign{f[j]} . As an illustrative example, the blue curve shows the output of ITNT for the input field f(x)=x (black dashed line) after applying G twice followed by repeated iterations of F . The red shaded region marks the regime of input values where the composed iteration F(…F(G(G(f)))) does not converge. The curves are evaluated in the full-array representation solely to visualize the mechanism of the procedure.
Figure S1 : Result of the Sign-function transformation via ITNT of the function defined in Eq. S8 for different number of roots Nroot and different qubit numbers n . For all examples we consider Niter=20 and an SVD tolerance of 10−8 . Sign transformation of the function with the following properties: (a) n=14 , Nroot=7 , χmax=20 . (b) n=14 , Nroot=7 , χmax=25 . (c) n=18 , Nroot=7 , χmax=25 . (d) n=18 , Nroot=23 , χmax=75 . For Nroot=23 , a noiseless sign transformation requires χmax=75 .
Figure S2 : Scaling of the element-wise evaluation of sign function via ITNT (a) Element-wise evaluation of sign function y=Sign{fω(x)} (red) using ITNT of a function y=fω(x) (black) ω=1.0 . (b) Required bond dimension in dependence on the number of tensors n for ω=0.5 . Considered are the maximal intermediate bond-dimension χpeak (crosses) used by the algorithm if the bond dimension is not restricted and the SVD tolerance is fixed at 10−8 . Furthermore, we consider the final bond dimension χfinal (circles) required to represent y accurate up to the order of the SVD tolerance of 10−8 with respect to the L2 -error. Finally, there is the cutoff bond dimension χmin (squares) which defines a lower cutoff bound, required to reach the result with ITNT up to this accuracy while reaching the bond dimension χfinal within 200 iteration steps. (c) Required bond dimensions for different numbers of roots Nroots by using ω=[0.5,1.0,1.5,2.0] corresponding to Nroots=[7,13,19,25] and n=18 to converge within a maximum of 200 iteration steps. The bond dimensions are defined as for (b). (d) The number of iteration steps Niter needed to reach the final compressed result with respect to the chosen maximal cutoff bond dimension χmax for ω=0.5 for n=14,18,22 (light to dark). All simulations were implemented using the zip-up algorithm for point-wise multiplication.
Figure S3 : The bond dimension of the sign-transformed TT after Niter steps of iteration. The qubit number is n=18 with root number Nroot=7 , and χmax denotes the maximum bond dimension allowed during TT operations (e.g., element-wise products), with the SVD tolerance fixed at 10−8 . For χmax=28 , the convergence is reached at Niter≈64 . For χmax≳35 , the required iteration count reduces to Niter≈34 .
n
Nroot
χmax
time (sec) of ITNT-based sign tranforms
time (sec) of standard sign transform.
22
7
45
3
1.6
22
19
100
7
1.6
22
51
250
65
1.6
24
7
45
3.5
6.5
24
19
100
8.5
6.5
24
51
250
84
6.5
Table 1: Comparison of runtime performance across different qubit numbers n , root numbers Nroot , and chosen cutoff bond dimensions χmax .
Figure S4 : Accuracy and scaling benchmarks. (a) Binary-discard deflation for locating extrema of a 12×12 classical Ising model. The algorithm is performed for 100 runs (each including 800 extremum-finding ITNT steps i ), and the cumulative best value at each step is shown, with the percentages indicating the fraction of runs that attain the corresponding value. (b) A single run of the extremum-finding ITNT steps. We fix χ=409 and iteratively self-multiply E that encodes the energy of all the configurations 8 times. In the resulting TT, we identified 800 sub-leading peaks, and found the extremum ∣E∣max∼33.35 after the i=324 th step. Here, the configuration returned by the i th deflation step is denoted as σ(i) with corresponding index σ(i) , and E[σ(i)] is the corresponding energy of the found configuration.
Figure S5 : Extremum-finding ITNT without preprocessing. Shown is a single instance of the results obtained from extremum-finding ITNT steps without performing the self-multiplication preprocessing procedure. Clearly, in comparison to Fig. S4 b., the values of the found configurations are significantly lower, indicating a failure to detect the global extremum.
Figure S6 : Energy matching. Energy matching E(σ)=0 for the 144 -qubit model is conducted and 104 candidate are identified. The best match, ∣E(σbest)∣∼1.38×10−4 , is found after the i=784 th step. The second-best match, ∣E(σnext-best)∣∼1.81×10−4 , is found after the i=5813 rd deflation. During the self-multiplication, the bond dimension is fixed at 818 for 14 iterations (each taking ∼104 seconds). During the following deflation, the bond dimension is truncated to 1900 whenever it reaches 2000 , yielding a per-extremum-finding ITNT runtime of ∼60 seconds.
Tensor networks represent high-dimensional data and quantum states as interconnected low-rank tensors to enable efficient computation, learning, and simulation.
We introduce the tubal tensor train (TTT) decomposition, a tensor-network model that combines the t-product algebra of the tensor singular value decomposition (T-SVD) with the low-order core structure of the tensor train (TT) format. For an order-(N+1) tensor with a distinguished tube mode, the proposed representation consists of two third-order boundary cores and N−2 fourth-order interior cores linked through the t-product. As a result, for bounded tubal ranks, the storage scales linearly with the number of modes, in contrast to direct high-order extensions of T-SVD. We present two computational strategies: a sequential fixed-rank construction, called TTT-SVD, and a Fourier-slice alternating scheme based on the alternating two-cores update (ATCU). We also state a TT-SVD-type error bound for TTT-SVD and illustrate the practical performance of the proposed model on image compression, video compression, tensor completion, and hyperspectral imaging.
Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1
Large deep neural networks are costly to store and deploy because inference must move and evaluate many parameters. This paper studies Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), compact differentiable weight generators for replacing selected dense, convolutional, and attention layers. An ADNTN maps a small set of trainable tensor cores to a full weight tensor through hierarchical contractions and learnable nonlinearities; the generated layer is then used as an ordinary linear or convolutional operator. We investigate three multilayered topologies: Tree Tensor Networks, augmented Tree Tensor Networks with boundary disentanglers, and MERA-style multi-scale decoders. Compared with flat brick-wall automatically differentiable tensor networks, these hierarchies provide logarithmic-depth communication between tensorised modes and optional lateral mixing, which can improve long-range structure without large increases in stored parameters. We give a unified forward--adjoint formulation showing how reverse-mode automatic differentiation computes pre-activation adjoints and contracted-environment gradients for all trainable cores. The formulation supports task losses, reconstruction losses, distillation, quantisation-aware terms, batching, and modern optimisers. Proof-of-concept experiments on selected AlexNet and VGG-16 layers on CIFAR-10 datasets achieve per-layer parameter-compression ratios from about 2,000× to 430,000×. Several VGG-16 compressed models match or slightly exceed the dense baseline, whereas AlexNet shows moderate degradation under more restrictive redundancy. These results indicate that nonlinear tensor-network generators are a promising structured route to compact pattern-recognition models, while also showing that contraction schedules and hardware-aware implementations remain essential for practical speedups.
Andrzej Cichocki, Michal Wietczak
Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
Tree tensor networks (TTNs) are widely used in low-rank approximation and quantum many-body simulation. In this work, we present a formal analysis of the quotient geometry underlying the TTN parameter space. Our framework allows for arbitrary horizontal distributions, and we develop efficient first- and second-order optimization algorithms that exploit this geometry. Additionally, we devise a backpropagation algorithm for training TTNs in a kernel learning setting. We validate our methods through numerical experiments on a representative digit classification task and reveal an important tradeoff between two different horizontal distributions that are available for TTNs: while one offers cleaner geometric statements, the other ultimately leads to more efficient algorithms.
Marius Willner, Marco Trenti, Dirk Lebiedz
Chair of Mathematical Data Science, University of Augsburg, Augsburg, Germany · Tensor AI Solutions GmbH, Pfaffenhofen a.d. Roth, Germany · Institute for Numerical Mathematics, University of Ulm, Ulm, Germany