cs.LGNov 12, 2026

Beyond a Single Sign: BiTE for Branch-Preserving Binary Threshold Expansion

Authors: Siddharth Gupta, Mikail Yayla, Akash Kumar

Organizations: Ruhr University Bochum, Germany

Abstract

Binary neural networks (BNNs) reduce model storage and replace many arithmetic operations with efficient bitwise computations, but the single-sign activation limits their representational capacity. Each preactivation direction is compared with one threshold and reduced to one of two values, discarding variation within each sign region. We propose BiTE (Branch-Preserving Binary Threshold Expansion), which evaluates each feature direction against KK ordered thresholds and retains the resulting binary responses as separate inputs to the following learned operator. BiTE can distinguish up to K+1K+1 ordered activation states while allowing the following operator to learn a separate transformation for each threshold response. It also enables a controllable trade-off between the number of independently constructed feature directions and the number of ordered responses extracted from each direction. On BinaryViT, BiTE improves mean final-epoch Top-1 accuracy from 74.60±0.30%74.60\pm0.30\% to 75.83±0.15%75.83\pm0.15\% on CIFAR-10 and from 42.80±0.46%42.80\pm0.46\% to 44.07±0.32%44.07\pm0.32\% on CIFAR-100, with nearly unchanged trainable parameter counts. A matched uniform-collapse control reaches 75.33±0.15%75.33\pm0.15\% on CIFAR-10, supporting the value of retaining the threshold responses separately relative to the evaluated scalar collapse. Binary VGG-16 experiments further show a favorable observed parameter--accuracy trade-off, including a configuration that achieves 0.640.64 percentage points higher mean accuracy with 21.8%21.8\% fewer parameters than conventional width scaling. Overall, these results support branch-preserving threshold expansion as an effective way to improve binary representations and the observed parameter--accuracy trade-off.

Figures & tables

Explore similar work

Aug 6, 2026

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference. For binary activations, the dot products become sign-controlled additions or subtractions, but the number of operations is unchanged. Indeed, every neuron or output channel still accumulates all of its input, even though only the sign will be retained, which is often wasteful. As the accumulation progresses, the running partial sum frequently drifts so far from zero that its final sign becomes highly predictable long before the last term is reached; every contribution evaluated after that point changes the value of the sum but not the final output activation. This paper turns this observation into a post-training early-stopping mechanism. We characterize the behavior of the running accumulations on the training dataset and use this information to predict the final sign as soon as possible. No model parameter is retrained. We count the number of operations under an idealized ordering of weights. On VGG11 applied to the CIFAR-10 dataset, the method removes 86.6%86.6\% of the accumulation terms of the deepest convolution for a 0.370.37-point accuracy drop, and 25%25\% of the full-network arithmetic when used on the three deepest convolutions simultaneously, for a 1.361.36-point drop.
Aug 2, 2026

BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks

Binarizing a polynomial Kolmogorov--Arnold Network (KAN) not only changes parameter precision, but also alters the function space available to each layer. When activations are restricted to −1,+1{-1,+1}, all even powers reduce to 11 and all odd powers reduce to xx, causing the elementwise polynomial basis to collapse to constant and first-order responses. We refer to this structural failure as Spatial Orthogonality Collapse. Our proposed BiKAN addresses this critical issue by augmenting each binary KAN layer with selected degree-2 Walsh characters. Fixed circular channel rolls generate pairwise parities, and learned binary projections mix them using the same XNOR--popcount operations as the remaining W1A1 paths. This restores explicit pairwise coordinates without learned routing or multiplier-based feature generation. Experiments on CIFAR-10 confirms that removing parity reduces accuracy by 1.231.23 points over five paired seeds (p=0.003p=0.003), the gain increases as width decreases, and accuracy improves monotonically as more parity planes are added. At an equal ∼\sim11.9M-parameter budget, parity outperforms conventional widening by 3.093.09 points (p<10−4p<10^{-4}). At W1A1, BiKAN reaches 99.48%99.48\%, 84.38%84.38\%, and 55.81%55.81\% on MNIST, CIFAR-10, and CIFAR-100, respectively. Post-route Zynq-7020 FPGA results show that the repair remains hardware-efficient; the convolutional design cuts DSP usage from 164 to 72 and estimated compute-core latency from 401 to 54.8 ms, while the power-of-two-aware dense design achieves zero-DSP inference with a 0.03-point accuracy loss. The BiKAN implementation is available at https://github.com/OSU-STARLAB/BiKAN.
Jun 26, 2026

Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen. We study an orthogonal axis: when and where binarization is enforced during training. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output, using stochastic partial masks with soft refresh. StoMPP delivers two complementary benefits. As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth (ResNet-50 BNN: +18.0/+13.5/+3.8 on CIFAR-10/100/ImageNet), and the pattern holds across ResNet-18/34/50, MobileNetV2, and BERT fine-tuning. Composed with surrogate gradients by applying STE only to frozen entries, it reaches +27.1/+19.8/+17.7 over vanilla STE on the same setting. Underlying both regimes is a single mechanistic finding: progression order is decisive. Forward layerwise progression prevents depth collapse, reverse progression collapses to near-chance, and binary-weight networks (without binary activations) are insensitive to order. We trace this asymmetry to activation-induced gradient blockades: a committed binary activation severs gradient flow upstream, and ordering controls when these blockades form. To isolate the progression's contribution from any benefit conferred by STE, we conduct all ablations in the STE-free regime; the resulting characterization (schedule, refresh, ordering, dynamics) thus reflects the progression itself rather than its interaction with surrogate gradients.