Fast matrix multiplication can be described as searching for low-rank decompositions of the matrix--multiplication tensor. We design a neural architecture, StrassenNet, which reproduces the Strassen algorithm for 2×2 multiplication. Across many independent runs the network always converges to a rank-7 tensor, thus numerically recovering Strassen's optimal algorithm. We then train the same architecture on 3×3 multiplication with rank r∈{19,…,23}. Our experiments reveal a clear numerical threshold: models with r=23 attain significantly lower validation error than those with r≤22, suggesting that r=23 could actually be the smallest effective rank of the matrix multiplication tensor 3×3. We also sketch an extension of the method to border-rank decompositions via an ε--parametrisation and report preliminary results consistent with the known bounds for the border rank of the 3×3 matrix--multiplication tensor.
Figures & tables
Figure 1. Tensor neural network which computes the row by product matrix multiplication with the classical algorithm.
Figure 2. Tensor neural network which computes the row by product matrix multiplication with the Strassen algorithm.
Figure 3. Final sub-network which computes the Strassen algorithm.
Hyperparameter
Value
Matrix size
n=3 (i.e., 3×3 matrices)
Rank
r
Training samples per epoch
10,000
Validation samples
10,000
Batch size
32
Epochs
60
Table 1. Summary of training hyperparameters.
Figure 4. Plot of the training (a) and validation (b) losses of a batch of experiments for matrices with entries clamped in [−1,1] and values of rank r between 19 and 23 . In blue, orange, red, green and yellow the mean and standard deviation of the losses for r=23,22,21,20,19 , respectively.
Figure 5. Histogram showing the mean μ and standard deviation σ of validation losses for matrices with entries limited to [−1,1] and rank values r between 19 and 23 . The value μ for r=23 , in blue, is 0,0025802 , while σ is 0,0050602 ; for r=22 , in orange, μ=0,012773 and σ=0,007641 ; for r=21 , in red, μ=0,029102 and σ=0,0056912 ; for r=20 , in green, μ=0,034695 and σ=0,0054031 ; for r=19 , in yellow, μ=0,047196 and σ=0,0085363 .
The current best bounds on the matrix multiplication exponent ω are obtained through a refinement of the laser method called combination loss analysis (Duan et al., 2022; Williams et al., 2024; Alman et al., 2025). In this note, we address the optimization problem at the core of this approach and propose several improvements. First, we reformulate the optimization problem allowing us to solve it in a larger setting than was previously possible. Second, we leverage recent advances in machine learning to design a new optimization algorithm for this problem. Finally, we refine the resulting optimization algorithm with AlphaEvolve. Our combined approach yields an upper bound of ω < 2.371177, improving the previous best bound of 2.371339.
Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder--Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2--7.8% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.
Ilya Koziev, Ivan Oseledets
DAIMLD, Moscow, Russia · Artificial Intelligence Research Institute, Moscow, Russia
We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications. We examine ten points in the algorithm design space: two geometries for rank-r matrices, three geometries for rank-r partial isometries, and block-matrix variants of these five, where factors are shared across block-rows and block-columns. We apply our methods to the multihead attention parameters in small language models. After tuning learning rates, our methods do not conclusively outperform an AdamW baseline. Our implementations are available online.