Binary neural networks (BNNs) reduce model storage and replace many arithmetic operations with efficient bitwise computations, but the single-sign activation limits their representational capacity. Each preactivation direction is compared with one threshold and reduced to one of two values, discarding variation within each sign region. We propose BiTE (Branch-Preserving Binary Threshold Expansion), which evaluates each feature direction against K ordered thresholds and retains the resulting binary responses as separate inputs to the following learned operator. BiTE can distinguish up to K+1 ordered activation states while allowing the following operator to learn a separate transformation for each threshold response. It also enables a controllable trade-off between the number of independently constructed feature directions and the number of ordered responses extracted from each direction. On BinaryViT, BiTE improves mean final-epoch Top-1 accuracy from 74.60±0.30% to 75.83±0.15% on CIFAR-10 and from 42.80±0.46% to 44.07±0.32% on CIFAR-100, with nearly unchanged trainable parameter counts. A matched uniform-collapse control reaches 75.33±0.15% on CIFAR-10, supporting the value of retaining the threshold responses separately relative to the evaluated scalar collapse. Binary VGG-16 experiments further show a favorable observed parameter--accuracy trade-off, including a configuration that achieves 0.64 percentage points higher mean accuracy with 21.8% fewer parameters than conventional width scaling. Overall, these results support branch-preserving threshold expansion as an effective way to improve binary representations and the observed parameter--accuracy trade-off.
Figures & tables
Figure 1: From a single sign to branch-preserving threshold expansion. (a) A conventional sign activation produces one binary response per feature direction. (b) Two ordered thresholds produce three activation states, but uniform collapse forces both threshold crossings to share the same downstream effect. (c) BiTE retains the threshold responses as separate inputs, enabling threshold-specific downstream transformations.
Dataset
Method
Top-1 (%)
CIFAR-10
BinaryViT
74.60±0.30
BinaryViT + BiTE
75.83±0.15
CIFAR-100
BinaryViT
42.80±0.46
BinaryViT + BiTE
44.07±0.32
Table 1: BinaryViT results on CIFAR-10 and CIFAR-100. Values are final-epoch Top-1 accuracy on the CIFAR test set, reported as sample mean ± sample standard deviation over seeds 0, 1, and 2.
Method
Top-1 (%)
DeSign
89.04±0.18
IE-Net
91.89±0.05
IE-Net + BiTE
92.06±0.13
Table 2: Secondary evaluation on binary ResNet-18 and CIFAR-10. Values report the highest test accuracy observed during training as sample mean ± sample standard deviation over three runs. IE-Net and IE-Net with BiTE use matched runs, whereas DeSign is independently trained.
Figure 2: Parameter–accuracy trade-off on VGG-16/CIFAR-10. (a) Highest test-set accuracy observed during training and (b) final-epoch test accuracy. Points report the mean over three runs, with error bars showing the sample standard deviation. Nondominance is determined from the mean parameter–accuracy coordinates among the evaluated configurations; connecting lines are visual guides only.
Variant
Thresholds
Responses
Top-1 (%)
Single-sign baseline
One at 0
Single
74.60±0.30
Collapsed-state control
Learned
Collapsed
75.33±0.15
Fixed-threshold BiTE
Fixed
Preserved
75.63±0.25
BiTE
Learned
Preserved
75.83±0.15
Table 3: Mechanism controls on BinaryViT/CIFAR-10. Values are final-epoch Top-1 accuracy on the CIFAR test set, reported as sample mean ± sample standard deviation over seeds 0, 1, and 2.
Figure 3: Effective rank of BinaryViT representations. Effective rank of BiTE and the collapsed-state control (a) immediately after threshold expansion and (b) after reduction to the same intermediate dimensionality. Curves show one matched seed-0 checkpoint pair evaluated on identical held-out inputs; they are not averages over runs.
Configuration
Params
Top-1 (%)
Top-5 (%)
Single-sign baseline
22.492 M
59.952
82.464
BiTE, layers 0–6
22.476 M
59.392
82.240
BiTE, layers 0–6, WDt=0
22.476 M
59.488
82.600
BiTE, layers 0–2, WDt=0
22.488 M
60.016
82.648
Table 4: Preliminary BinaryViT results on ImageNet. Each configuration is represented by one run. Top-1 and Top-5 are reported from the checkpoint with the highest validation Top-1 accuracy. WDt=0 denotes zero weight decay on threshold parameters.