BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation"

Vladimer Khasia

cs.LGMar 5, 2026

BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation"

Authors: Vladimer Khasia

Organizations: Independent Researcher

Abstract

The activation memory required for exact backpropagation scales linearly with network depth, context length, and feature dimensionality, forming an O(L * BN ) spatial bottleneck (where B is the sequence-batch cardinality and N is the feature dimension). This constraint historically throttles the scaling of deep neural networks. While randomized automatic differentiation attempts to mitigate this, it historically suffers from catastrophic variance. In this paper, we introduce BASIS (Balanced Activation Sketching with Invariant Scalars), an efficient backpropagation algorithm that fully decouples activation memory from the batch and sequence dimensions. BASIS propagates the exact error signal (dX) to preserve flawless gradient flow, but computes the weight updates (dW) using massively compressed rank-R tensors. To solve the foundational instability of sketched gradients, we propose two novel mechanisms: Balanced Hashing, which strictly eliminates off-diagonal collision variance, and Invariant Scalars, a principled bias-variance tradeoff that deterministically preserves the exact continuous energy norm of the spatial geometry. Theoretically, BASIS reduces activation memory to O(L * RN ) and heavily decreases the backward pass matrix-multiplication footprint. Empirically, training a GPT architecture for 50,000 steps validates our theoretical guarantees: at R = 32, BASIS achieves parity with (and marginally outperforms) exact backpropagation validation loss (6.575 vs. 6.616), acting as an implicit regularizer. Remarkably, the stabilized magnitude trajectory allows the model to converge smoothly even under extreme spatial compression (R = 1), proving the extreme robustness of the estimator. The code is available at https://github.com/VladimerKhasia/basis

Figures & tables

Method	Network Activation Memory	Forward Time	Backward Time ( $\mathbf{dX}$ )	Backward Time ( $\mathbf{dW}$ )	Estimator Property
Exact Backprop	$\mathcal{O}(L\cdot BN)$	$\mathcal{O}(BNM)$	$\mathcal{O}(BNM)$	$\mathcal{O}(BNM)$	Exact
BASIS (Ours)	$\mathbf{\mathcal{O}(L\cdot RN)}$	$\mathcal{O}(BNM)$	$\mathcal{O}(BNM)$	$\mathbf{\mathcal{O}(RNM)}$	Biased (Norm-Stabilized)

Table 1: Theoretical Complexity for an

L

-layer network. (Context

B

, Feature Dims

N\times M

, Rank

R\ll B

)

Hyperparameter	Configuration
Architecture	Generative Transformer (GPT)
Vocabulary Size ( $V$ )	50,257
Sequence Length ( $T$ )	64
Embedding Dimension ( $d_{\text{model}}$ )	64
Attention Heads ( $N_{\text{head}}$ )	2
Transformer Layers ( $N_{\text{layer}}$ )	2

Table 2: Architectural and Optimization Hyperparameters

Method	Rank ( $R$ )	Seq. Memory Compression	Final Train Loss	Final Val Loss
Exact Backpropagation	N/A	1x (None)	6.718	6.616
BASIS (Ours)	64	1x (Sanity Check)	6.680	6.573
BASIS (Ours)	32	2x	6.735	6.575
BASIS (Ours)	16	4x	6.731	6.675
BASIS (Ours)	8	8x	6.798	6.762
BASIS (Ours)	1	64x (Maximal)	7.247	7.221

Table 3: Terminal Loss Metrics at Step 50,000 across Compression Regimes

Figure 1: Validation loss trajectories over 50,000 optimization steps for varying sketch ranks. The $R=32$ trajectory (purple) perfectly superimposes the exact baseline (blue), while $R=16$ (red) and $R=8$ (green) display graceful degradation with highly correlated descent geometries. The $R=1$ trajectory (orange) proves that monotonic convergence is robustly preserved even under 64x spatial compression.

Backpropagation

Backpropagation computes neural-network gradients by propagating errors backward through layers, enabling end-to-end training.

See all 44 papers

Jul 18, 2026

Backpropagation-Free Trunk Training via the Split Forward Gradients

Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a

16

M-parameter GPT-2-style model reaches validation perplexity

387

, compared with

668

for the frozen-trunk control and

2{,}885

for a matched pure forward-gradient baseline (backpropagation reaches

150

). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches

60.5\%

on CIFAR-10 and

35.2\%

on CIFAR-100 with a heavy-head design. It reduces peak memory by up to

35\%

relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.

Tian Qin, Wei-Min Huang

Math department Lehigh University Bethlehem, PA 18015, USA

Backpropagation Gradient Descent Neural Network Optimization Neural Network Training Dynamics Memory-Efficient Training

Ask PDF

Feb 16, 2026

Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation

In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations thereof during backpropagation. We provide a theoretical analysis of the trade-off between the number of epochs needed to achieve a target precision and the cost reduction for each epoch. We then identify specific unbiased estimates of vector-jacobian products for which we establish desirable optimality properties of minimal variance under sparsity constraints. Finally we provide in-depth experiments on multi-layer perceptrons, BagNets and Visual Transfomers architectures. These validate our theoretical results, and confirm the potential of our proposed unbiased randomized backpropagation approach for reducing the cost of deep learning.

Killian Bakong, Laurent Massoulié, Edouard Oyallon +1

DI-ENS · Inria Paris & DI ENS, ENS, PSL University · Inria, ARGO, CMAP +2

Backpropagation Deep Learning Optimization Memory-Efficient Training

Ask PDF

May 2, 2026

Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

Training large language models (LLMs) is highly memory-intensive, as training must store not only weights and optimizer states but also intermediate activations for backpropagation. While existing memory-efficient methods largely focus on gradients and optimizer states, activation compression is less well established due to the lack of LLM-tailored theory and guarantees. In this work, we develop a theoretical framework showing that activation compression is safe for linear operators when activation compression is unbiased, but problematic for nonlinear ones. We further derive gradient variance bound and establish convergence guarantees for applying activation compression to all linear operators under the standard

L

-smoothness assumption, showing that it does not change the convergence rate. Guided by the theory, we propose an activation-gradient co-compression method that reuses low-rank activation factors to compress linear-layer gradients without extra computation or additional gradient error. We conduct extensive experiments on Qwen and LLaMA models using a pretraining benchmark and multiple fine-tuning benchmarks to validate our theory and demonstrate competitive performance of our method in both accuracy and compression efficiency. We provide our code in the supplementary material for reproducibility.

Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3

Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1

LLM Compression Gradient Compression Low-Rank Compression Activation Compression LLM Training Neural Network Compression+1

Ask PDF

Abstract

Backpropagation

Explore similar work

Backpropagation-Free Trunk Training via the Split Forward Gradients

Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation

Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm