Differentiable logic networks (DLNs) are interpretable, edge-friendly classifiers that learn both their logic operators and their connections through gradient descent, achieving accuracy competitive with multi-layer perceptrons at orders-of-magnitude lower inference cost. Their accuracy, however, is bounded by the problems of vanishing gradients, scalability, and lack of an ecosystem, leaving a gap relative to conventional neural networks on several datasets. We introduce knowledge distillation for DLNs to narrow this gap. We distill from a trained multi-layer perceptron teacher into a DLN student, integrating the distillation objective into a two-phase training loop without altering the architecture or the inference procedure. The teacher is discarded after training. Hence, the distilled model retains the interpretability, sparsity, and inference cost of a conventionally trained DLN. We investigate this approach through four variants: a joint hyperparameter search that optimizes the distillation weight and temperature alongside the architecture, a grid search over these two hyperparameters with the architecture fixed, and two single-phase variants that employ distillation during only the neuron-function or only the connection phase. On 20 public tabular classification tasks, distillation improves balanced-class accuracy at negligible cost to gate-level operation counts and model size, though the benefit varies by dataset. Performance improvements of up to 15% are observed, in some cases surpassing the teacher itself.
Figures & tables
Figure 1: A simple example of our DLN. The model takes binarized continuous variables and categorical variables through a ThresholdLayer. Then, the binary vectors are passed to the LogicLayers. We concatenate the inputs with a ThresholdLayer at every intermediate LogicLayer, i.e. except the first and last one. At the end, the SumLayer determines the input’s class.
Figure 2: KD flowchart for DLNs.
Original
Knowledge Distillation
Data set
MLP (teacher)
DLN
Joint HPO
Grid Search
Neuron Phase
Connection Phase
BankChurn
85.20
78.64
78.13
77.48
75.91
79.03
BreastCancer
77.80
65.55
67.37
67.37
62.89
63.17
Cardio
99.20
97.72
99.07
97.88
97.30
97.80
CKD
99.00
100.0
98.65
98.39
98.33
100.0
Cirrhosis
58.80
45.62
47.20
51.49
47.97
53.99
Table 1: Balanced-class test accuracies for DLNs obtained using KD.
Original
Knowledge Distillation
Data set
MLP (teacher)
DLN
Joint HPO
Grid Search
Neuron Phase
Connection Phase
BankChurn
7.67M
1.6K
4.6K
3.7K
2.6K
2.8K
BreastCancer
93.7M
358
1.3K
358
211
1.8K
Cardio
13.1M
109.2K
6.4K
6.5K
6.0K
4.9K
CKD
29.7M
5.1K
3.7K
1.6K
2.6K
2.7K
Cirrhosis
9.10M
2.8K
2.1K
5.1K
3.2K
1.3K
Table 2: Number of basic logic gate-level operations for inference, assuming float16 for floating-point and int16 for integer.
Original
Knowledge Distillation
Data set
MLP (teacher)
DLN
Joint HPO
Grid Search
Neuron Phase
Connection Phase
BankChurn
12.4K
44
162
99
58
70
BreastCancer
151.0K
82
511
83
58
683
Cardio
21.1K
1839
179
250
296
206
CKD
47.6K
283
91
105
62
73
Cirrhosis
14.7K
38
37
135
123
23
Table 3: Number of high-level operations for inference, assuming float16 for floating-point and int16 for integer.
Figure 3: The difference in model size for the Diabetes dataset can be observed. The top model is the original DLN model, and the bottom model is the distilled DLN model.
Figure 4: An example heatmap for the HeartFailure dataset for grid search. The percentage improvement in validation accuracy compared to the baseline DLN validation accuracy is recorded over 25 combinations of α and T .
Figure 5: A comparison of training pipelines and time using the AdultIncome dataset. (a) This is the original DLN training pipeline, which takes about 13 hours to achieve 82.64% accuracy. (b) The KD pipeline that does not employ hyperparameter search takes 3 hours to achieve a similar accuracy to (a).
Figure 6: An example DLN model for the HepatitisC dataset. Given some feature values for an example diseased patient, the triggered paths are indicated in red. It is possible to provide a personalized interpretation to why this patient is labeled as diseased by tracing back the red lines to specific conditions (e.g., ALP > 40.63).
Neural networks (NNs) achieve outstanding performance in many domains; however, their decision processes are often opaque and their inference can be computationally expensive in resource-constrained environments. We recently proposed Differentiable Logic Networks (DLNs) to address these issues for tabular classification based on relaxing discrete logic into a differentiable form, thereby enabling gradient-based learning of networks built from binary logic operations. DLNs offer interpretable reasoning and substantially lower inference cost. We extend the DLN framework to supervised tabular regression. We first redesign the final output layer (the SumLayer) to support continuous targets. More critically, we find the original two-phase training procedure used for classification is suboptimal for regression, and thus develop a unified, single-stage optimization procedure. We also demonstrate that temperature annealing of the network's differentiable relaxations is decisive for achieving stable convergence and high accuracy. We evaluate the resulting model on 15 public regression benchmarks, comparing it with modern neural networks and classical regression baselines. Regression DLNs match or exceed baseline accuracy while preserving interpretability and fast inference. Our results show that DLNs are a viable, cost-effective alternative for regression tasks, especially where model transparency and computational efficiency are important.
Chang Yue, Niraj K. Jha
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA
Differentiable logic networks (DLNs) have shown promising results in tabular domains by combining accuracy, interpretability, and computational efficiency. In this work, we apply DLNs to the domain of TSC for the first time, focusing on univariate datasets. To enable DLN application in this context, we adopt feature-based representations relying on Catch22 and TSFresh, converting sequential time series into vectorized forms suitable for DLN classification. Unlike prior DLN studies that fix the training configuration and vary various settings in isolation via ablation, we integrate all such configurations into the hyperparameter search space, enabling the search process to select jointly optimal settings. We then analyze the distribution of selected configurations to better understand DLN training dynamics. We evaluate our approach on 51 publicly available univariate TSC benchmarks. The results confirm that classification DLNs maintain their core strengths in this new domain: they deliver competitive accuracy, retain low inference cost, and provide transparent, interpretable decision logic, thus aligning well with previous DLN findings in the realm of tabular classification and regression tasks.
Chang Yue, Niraj K. Jha
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA
Differentiable logic gate networks (DLGNs) enable gradient-based training of highly efficient Boolean networks by relaxing discrete logic gates during training and discretizing them for inference. Standard approaches make this discretization decision locally, typically through argmax selection and confidence- or entropy-based convergence criteria. We show that local discretization can be task-suboptimal even for globally optimal relaxed solutions, with high gate confidence providing no general guarantee, and derive bounds relating task-aware gate selection to tractable interventions in the relaxed network. Motivated by these results, we study first-order downstream task information for progressive discretization and characterize when this local approximation is reliable. Experiments on convolutional DLGNs reveal a strong locality dependence: first-order scores become unreliable when directly optimized over nonlocal interventions, but accurately assess local argmax decisions for progressive freezing.