Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks

Youngsung Kim

cs.LGMar 14, 2026

Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks

Authors: Youngsung Kim

Organizations: Department of Artificial Intelligence and Department of Electrical and Computer Engineering, Inha University, Incheon, Korea.

Abstract

In neural network models, soft mixtures of fixed candidate components (e.g., logic gates and sub-networks) are often used during training for stable optimization, while hard selection is typically used at inference. This raises questions about training-inference mismatch. We analyze this gap by separating forward-pass computation (hard selection vs. soft mixture) from stochasticity (with vs. without Gumbel noise). Using logic gate networks as a testbed, we observe distinct behaviors across four methods: Hard-ST achieves zero selection gap by construction; Gumbel-ST achieves near-zero gap when training succeeds but suffers accuracy collapse at low temperatures; Soft-Mix achieves small gap only at low temperature via weight concentration; and Soft-Gumbel exhibits large gaps despite Gumbel noise, confirming that noise alone does not reduce the gap. We propose CAGE (Confidence-Adaptive Gradient Estimation) to maintain gradient flow while preserving forward alignment. On logic gate networks, Hard-ST with CAGE achieves over 98% accuracy on MNIST and over 58% on CIFAR-10, both with zero selection gap across all temperatures, while Gumbel-ST without CAGE suffers a 47-point accuracy collapse.

Figures & tables

Panel of Figure 1 — Figure 1: Training-inference gap depends primarily on forward-pass structure. Left: Mixture methods (top) compute weighted sums, enabling hedging; hard selection methods (bottom) commit to a single gate, matching inference. Right-Top: Inference uses argmax selection. Right-Bottom: Peak selection gap ( $2\times 2$ factorial, MNIST-Binary, $L$ =6, $\tau$ =2.0). Hard methods achieve near-zero gap; soft methods show 24–40% gap, with Gumbel noise amplifying the gap.

Method	$\tau$ =0.05	$\tau$ =0.1	$\tau$ =0.5	$\tau$ =1.0	$\tau$ =2.0
Soft-Mix	0.0 / 0	0.0 / 0	0.0 / 0.1	0.0 / 0.1	0.6 / 24
Soft-Gumbel	0.7 / 2	0.2 / 2	0.2 / 7	0.2 / 17	2.1 / 40
Hard-ST	0 / 0	0 / 0	0 / 0	0 / 0	0 / 0
Gumbel-ST	$-$ 40 †	$-$ 59 †	0.0 / 0.1	0.0 / 0.3	0.0 / 1.1
Hard-ST+ CAGE	0 / 0	0 / 0	0 / 0	0 / 0	0 / 0
Gumbel-ST+ CAGE	0.0 / 4.5	0.1 / 9	0.0 / 3.5	0.0 / 1.6	0.0 / 2.2

Table 1: Selection gap (%) on MNIST-Binary (

L

=6). Format: Final / Peak (omitted for training failures marked with

\dagger

). Hard-ST achieves exactly zero gap at all temperatures. Soft methods show temperature-dependent gaps. Negative values indicate training failure (training accuracy below deployment). Bold : zero selection gap.

Method	MNIST	M-Binary	C-10-Binary
Soft-Mix	97.1 / 1.4	97.2 / 1.1	50.7 / 7.5
Soft-Gumbel	85.9 / 12.0	85.6 / 12.2	52.6 / 5.4
Hard-ST	97.1 / 1.3	97.1 / 1.1	50.9 / 7.2
Gumbel-ST	52.1 / 46.3	51.0 / 47.3	18.9 / 39.5
Hard-ST+ CAGE	98.4 / 0.1	98.1 / 0.0	57.6 / 0.5
Gumbel-ST+ CAGE	98.4 / 0.1	98.2 / 0.1	58.0 / 0.6

Table 2: Temperature robustness: worst accuracy / range (%) across

\tau\in\{0.05,0.1,0.5,1.0,2.0\}

L

=6. Gumbel-ST suffers catastrophic failure at low

\tau

across all datasets; CAGE eliminates this sensitivity entirely. Bold : worst performance (failure case); underline : best range (most robust).

Panel of Figure 2 — Figure 2: Main results (MNIST-Binary, $L$ =6, $\tau$ =1.0 unless varied). (a) Selection gap during training: Hard-ST achieves exactly zero; soft methods show non-zero gaps. (b) Test accuracy vs temperature: Hard-ST and CAGE remain stable; Gumbel-ST fails at low $\tau$ . (c) Iterations to convergence: Hard-ST converges significantly faster than soft methods. (d) Gate commitment by layer: Hard-ST commits; Soft-Gumbel hedges, especially in early layers.

Figure 3: CAGE adaptation dynamics (MNIST-Binary,

L{=}6

). (a) Backward temperature over training:

\tau_{b}

decreases from

\tau_{\max}

as confidence rises. (b) EMA confidence over training: increases from uniform (

\sim

0.25) toward commitment (

\sim

0.9). Results averaged over

n{=}3

random seeds.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Discrete Choices	Training	Inference	Gap?
Logic Gate Networks ( Petersen et al., 2022 ; Kim, 2024 )	Boolean gates	Soft mixture	Hard (argmax)	Yes
NAS (DARTS) ( Liu et al., 2019 )	Operations	Soft mixture	Hard (argmax)	Yes
NAS (SNAS/GDAS) ( Xie et al., 2019 ; Dong & Yang, 2019 )	Operations	Hard (sample)	Hard (argmax)	Small
MoE (traditional) ( Jacobs et al., 1991 )	Expert networks	Soft gating	Soft gating	No
MoE (sparse) ( Fedus et al., 2022 )	Expert networks	Hard (top-k)	Hard (top-k)	No
VQ-VAE ( Van Den Oord et al., 2017 )	Codebook vectors	Hard (nearest)	Hard (nearest)	No

Appendix

Table 3: Training-inference gap in discrete selection. Domains using soft training with hard inference suffer from mismatch; those using consistent selection avoid it.

Parameter	MNIST	CIFAR-10
Neurons per layer	64,000	128,000
Layers	4, 5, 6	4, 5, 6
Gates per node	16	16
Batch size	512	512
Learning rate	0.01	0.05
Optimizer	Adam	Adam

Appendix

Table 4: Training configuration.

	MNIST		MNIST-Binary		CIFAR-10-Binary
Method	Peak Gap	Acc Range	Peak Gap	Acc Range	Peak Gap	Acc Range
Soft-Mix	23%	1.4%	24%	1.1%	9%	7.5%
Soft-Gumbel	40%	12.0%	40%	12.2%	3.5%	5.4%
Hard-ST	0.4% ∗	1.3%	0%	1.1%	0%	7.2%
Gumbel-ST	62% †	46.3%	61% †	47.3%	16% †	39.5%
Hard-ST+CAGE	0.3% ∗	0.1%	0%	0.0%	0%	0.5%

Appendix

Table 5: Cross-dataset summary (

L

=6). Peak selection gap (%) during training and accuracy range (%) across

\tau\in\{0.05,0.1,0.5,1.0,2.0\}

. Hard-ST achieves zero selection gap and minimal accuracy range across all datasets. Gumbel-ST suffers catastrophic training failure at low

\tau

. Bold : zero selection gap; underline : best accuracy range.

Method	L	$\tau$ =0.05	$\tau$ =0.1	$\tau$ =0.5	$\tau$ =1.0	$\tau$ =2.0
Soft-Mix	4	$+0.3$ [+0.2,+0.4]	$+0.2$ [+0.1,+0.3]	$+0.2$ [+0.1,+0.3]	$+0.1$ [+0.0,+0.3]	$+0.2$ [+0.2,+0.5]
	5	$+0.3$ [+0.2,+0.4]	$+0.2$ [+0.1,+0.3]	$+0.2$ [+0.1,+0.3]	$+0.1$ [+0.1,+0.3]	$+0.3$ [+0.2,+1.5]
	6	$+0.3$ [+0.1,+0.4]	$+0.2$ [+0.1,+0.3]	$+0.2$ [+0.1,+0.3]	$+0.1$ [+0.1,+0.3]	$+0.7$ [+0,+23]
Soft-Gumbel	4	$+0.6$ [-31,+2]	$+0.2$ [-15,+2]	$+0.3$ [-0.1,+2.3]	$+0.3$ [+0,+10]	$+0.5$ [+0,+21]
	5	$-0.2$ [-33,+1]	$+0.2$ [-33,+2]	$+0.3$ [+0.0,+2.9]	$+0.4$ [+0,+16]	$+1.1$ [+1,+33]
	6	$+0.2$ [-29,+4]	$+0.6$ [-25,+4]	$+0.4$ [-0,+9]	$+0.4$ [+0,+16]	$+1.7$ [+1,+40]

Appendix

Table 6: Gap trajectory on MNIST. Format: Final [min, max] (%). Range computed over last 80% of training. Hard-ST shows stable near-zero gap; Soft-Gumbel shows large transient peaks that increase with

\tau

Method	L	$\tau$ =0.05	$\tau$ =0.1	$\tau$ =0.5	$\tau$ =1.0	$\tau$ =2.0
Soft-Mix	4	$+0.0$ [-0.1,+0.0]	$+0.0$	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.2]	$+0.0$ [-0.1,+0.3]
	5	$+0.0$	$+0.0$	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+1.6]
	6	$+0.0$	$+0.0$	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.6$ [+0,+24]
Soft-Gumbel	4	$+0.3$ [-31,+1]	$+0.1$ [-14,+1]	$+0.1$ [-0.3,+1.8]	$+0.2$ [+0,+9]	$+0.2$ [+0,+23]
	5	$-0.1$ [-35,+1]	$-0.5$ [-31,+2]	$+0.2$ [-0.4,+4.3]	$+0.2$ [+0,+18]	$+0.7$ [+0,+35]
	6	$+0.7$ [-30,+2]	$+0.2$ [-25,+2]	$+0.2$ [-1,+7]	$+0.2$ [+0,+17]	$+2.1$ [+2,+40]

Appendix

Table 7: Gap trajectory on MNIST-Binary. Format: Final [min, max] (%). With binary inputs, computation gap = 0, so this isolates pure selection gap. Hard-ST achieves exactly 0% at all temperatures.

Method	L	$\tau$ =0.05	$\tau$ =0.1	$\tau$ =0.5	$\tau$ =1.0	$\tau$ =2.0
Soft-Mix	4	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.3,+0.3]	$+0.1$ [-0.3,+0.5]
	5	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.3,+0.3]	$-0.1$ [-0.2,+1.5]
	6	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.1,+0.1]	$+0.0$ [-0.2,+0.3]	$+0.3$ [-0,+9]
Soft-Gumbel	4	$-0.2$ [-2.3,+0.4]	$+0.0$ [-2.0,+0.4]	$+0.0$ [-0.7,+0.5]	$+0.0$ [-0.3,+0.3]	$+0.1$ [-0.2,+0.4]
	5	$+0.0$ [-2.5,+0.3]	$-0.2$ [-2.6,+0.4]	$+0.0$ [-0.8,+0.7]	$+0.0$ [-0.3,+0.5]	$+0.1$ [-0.3,+0.4]
	6	$-0.3$ [-3.5,+0.2]	$+0.2$ [-2.3,+0.4]	$+0.0$ [-0.7,+0.5]	$+0.0$ [-0.3,+0.5]	$+0.0$ [-0.3,+0.4]

Appendix

Table 8: Gap trajectory on CIFAR-10-Binary. Format: Final [min, max] (%). Hard-ST achieves exactly 0% gap; Gumbel-ST fails at low

\tau

but less severely than on MNIST.

Figure 4: Gap heatmaps on MNIST. Hard-ST shows uniform near-zero gap (light colors) across all conditions. Gumbel-ST exhibits catastrophic negative gap at low $\tau$ (dark blue = training failure where deployment outperforms training). CAGE eliminates this failure, restoring uniform small gap.

Temperature sensitivity (accuracy)
Claim	Comparison	$t$	$p$	$d$	Stat.	Prac.
Gumbel-ST fails at low $\tau$	$\tau$ =0.05 vs $\tau$ =2.0	$-$ 36.16	$<$ .001	31.21		✓
Hard-ST is $\tau$ -robust	$\tau$ =0.05 vs $\tau$ =2.0	$-$ 14.53	.005	16.26	**	–
CAGE effectiveness
CAGE fixes Gumbel-ST	Gumbel-ST vs +CAGE ( $\tau$ =0.05)	$-$ 38.81	$<$ .001	31.28		✓
Selection gap comparisons (at $\tau$ =2.0)

Appendix

Table 9: Statistical significance of key claims (MNIST-Binary,

L

=6,

n

=3 seeds). Effect size

d

is Cohen’s

d

( Cohen, 1988 ) .

Claim	MNIST	MNIST-Binary	CIFAR-10-Binary
Gumbel-ST fails at low $\tau$
Hard-ST is $\tau$ -robust	** †	** †	*** †
CAGE fixes Gumbel-ST

Appendix

Table 10: Cross-dataset verification of key claims (

L

=6, significance level).

Figure 5: Gate usage distribution (MNIST-Binary,

L

=6,

\tau

=1.0). Hard-ST methods show pronounced XOR/XNOR preference (

\sim

11% each vs.

\sim

6% expected under uniform selection), while soft methods distribute usage more uniformly. This suggests hard selection encourages commitment to more expressive Boolean operations.

Index	Gate	Boolean	Soft Formula	Hard Formula
0	FALSE	$0$	$0$	$0$
1	AND	$a\land b$	$ab$	$\mathds{1}[a>0.5]\land\mathds{1}[b>0.5]$
2	A $\land\neg$ B	$a\land\neg b$	$a(1-b)$	$\mathds{1}[a>0.5]\land\mathds{1}[b\leq 0.5]$
3	A	$a$	$a$	$\mathds{1}[a>0.5]$
4	$\neg$ A $\land$ B	$\neg a\land b$	$(1-a)b$	$\mathds{1}[a\leq 0.5]\land\mathds{1}[b>0.5]$
5	B	$b$	$b$	$\mathds{1}[b>0.5]$

Appendix

Table 11: All 16 two-input Boolean gates with soft (differentiable) and hard (Boolean) formulas.

Distribution	Unary	AND/OR	XOR
Binary $\{0,1\}$ (lower bound)	0%	0%	0%
MNIST (91% binary-like)	$\sim$ 2%	$\sim$ 3%	$\sim$ 5%
Uniform $[0,1]$	25%	21.9%	16.7%
At boundary $a{=}0.5$ (upper)	50%	75%	50%

Appendix

Table 12: Computation gap for representative gates under different input distributions.

Category	Gates	Gap (%)
Constants	FALSE, TRUE	0.0
XOR-type	XOR, XNOR	16.7
AND/OR-type	AND, OR, NAND, NOR, A $\land\neg$ B, $\neg$ A $\land$ B, A $\lor\neg$ B, $\neg$ A $\lor$ B	21.9
Unary	A, B, $\neg$ A, $\neg$ B	25.0

Appendix

Table 13: Expected computation gap for all 16 gates on Uniform [0,1] inputs.

Pixel Range	MNIST [0,1]	MNIST-Binary
Exactly 0	80.9%	86.7%
$(0,0.1)$	1.7%	0.0%
$[0.1,0.9]$	8.7%	0.0%
$(0.9,1.0]$	8.8%	13.3%
Binary-like ( $<0.1$ or $>0.9$ )	91.3%	100.0%

Appendix

Table 14: MNIST pixel value distribution.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

Task-Aware Discretization of Differentiable Logic Gate Networks
Sep 27, 2026Thore GerlachEfficient Neural Network Inference Differentiable Logic Gate Networks
Independent Researcher
On the Depth Scalability of Logic Gate Networks
Jul 19, 2026Taegun An, Dohun kim, Haebeom Lee +1Neural Network Optimization Credit Assignment
Jung Woonoh IT & General Education Center, Korea University 145, Anam-ro, Seongbuk-gu, Seoul 02841, Republic of Korea
Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks
Jul 10, 2026Wout Mommen, Lars Keuninckx, Matthias Hartmann +2Neural Network Optimization Differentiable Logic Gate Networks
imec, Kapeldreef 75, Leuven, 3001, , Belgium · Vrije Universiteit Brussel, Peinlaan 2, Elsene, 1050, , Belgium

Sep 27, 2026J/K move · Enter open · S save

Task-Aware Discretization of Differentiable Logic Gate Networks

Thore Gerlach

Independent Researcher

Efficient Neural Network Inference Differentiable Logic Gate Networks Differentiable Logic

Open paper AskPDF arXiv

Differentiable logic gate networks (DLGNs) enable gradient-based training of highly efficient Boolean networks by relaxing discrete logic gates during training and discretizing them for inference. Standard approaches make this discretization decision locally, typically through argmax selection and confidence- or entropy-based convergence criteria. We show that local discretization can be task-suboptimal even for globally optimal relaxed solutions, with high gate confidence providing no general guarantee, and derive bounds relating task-aware gate selection to tractable interventions in the relaxed network. Motivated by these results, we study first-order downstream task information for progressive discretization and characterize when this local approximation is reliable. Experiments on convolutional DLGNs reveal a strong locality dependence: first-order scores become unreliable when directly optimized over nonlocal interventions, but accurately assess local argmax decisions for progressive freezing.

Abstract

Neural Network Optimization

Explore similar work

Task-Aware Discretization of Differentiable Logic Gate Networks

On the Depth Scalability of Logic Gate Networks

Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks