ZeroPoint: Lightweight Energy-Based Decision Model
Abstract
Energy-based models (EBMs) assign a scalar energy to every joint value of the observed and predicted variables. In decision models, the prediction is chosen from a finite list of options provided with each input. Inference then reduces to comparing the energies of the options, and training lowers the energy of the correct option relative to the others. A complete decision must also account for the possibility that no option is acceptable. However, the energies learned by a standard EBM define only relative compatibility: adding the same input-dependent quantity to all energies leaves the predictions, the Gibbs distribution, and the usual training losses unchanged. The model therefore has no learned reference level against which to decide whether any option fits. We propose ZeroPoint, an EBM that makes the empty answer a state of the model with energy zero for every input. This fixed state provides an explicit reference for the energies of all listed options, whose non-negative depths measure their compatibility with the input relative to the empty answer. The options and the empty answer share one distribution, in which an option at zero depth receives no probability. The decision to select an option or to abstain is thus part of ordinary inference, trained with the negative log-likelihood loss, rather than a separate rejection rule. The energy function is a small multilingual encoder that reads the input together with all the options and is invariant to their order. Ablations show that most of the benefit comes from training with examples in which no option fits. With the same encoder and data, the fixed zero separates inputs with and without an acceptable option slightly better than a learned rejection class and clearly better than a thresholded softmax, while leaving the choice among options unchanged.
Figures & tables
| System | Correct | Ranking | Silent | Order | Latency | Runs on |
|---|---|---|---|---|---|---|
| decisions | mistakes | changes | (median) | |||
| ZeroPoint | 53.6 | 43.8 | 7.8 | 0.0 | 42 ms | CPU |
| Strands Decider 2B | 58.6 | 58.4 | 0.0 | 7.3 | 93 ms a | GPU |
| Clef-flash | 53.3 | 56.2 | 5.8 | 0.0 | 156 ms | GPU |
| Laya | 49.9 | 55.6 | 4.9 | 13.1 | 593 ms | CPU |
| Semantic Router | 19.4 | 38.9 | – | 0.6 | 17 ms | CPU |
| System | Correct decisions | Ranking | Silent mistakes |
|---|---|---|---|
| Strands Decider 2B | 5.0 [ 8.6, 1.6] | 14.7 [ 20.2, 9.3] | +7.8 [+5.3, +10.4] |
| Clef-flash | +0.2 [ 3.2, +3.8] | 12.4 [ 17.8, 7.3] | +2.0 [ 0.9, +4.9] |
| Laya | +3.7 [+0.1, +7.3] | 11.8 [ 16.4, 6.9] | +2.9 [ 0.4, +6.0] |
| Semantic Router | +34.1 [+30.1, +38.0] | +4.9 [+0.2, +10.0] | – |
| Zero-shot NLI | +32.9 [+28.7, +36.8] | +2.4 [ 2.4, +7.8] | 14.4 [ 18.7, 10.4] |
| Claude Haiku 5.5 | 3.4 [ 6.8, 0.1] | – | +6.7 [+4.2, +9.3] |
| Correct decisions | Silent mistakes | |||||
|---|---|---|---|---|---|---|
| System | 2,000 | Removed | Unrelated | 2,000 | Removed | Unrelated |
| ZeroPoint | 76.7 | 22.3 | 61.7 | 4.7 | 18.7 | 0.0 |
| Strands Decider 2B | 88.0 | 21.7 | 66.0 | 0.0 | 0.0 | 0.0 |
| Clef-flash | 85.3 | 19.3 | 55.3 | 8.0 | 9.3 | 0.0 |
| Laya | 72.3 | 19.3 | 58.0 | 14.0 | 0.7 | 0.0 |
| Semantic Router | 32.3 | 12.3 | 13.7 | – | – | – |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Test (AUROC) | Fixed zero | Softmax | Softmax + none | Other |
|---|---|---|---|---|
| Tickets, queue removed, within each queue | 0.679 | 0.556 | 0.599 | 0.615 a |
| the same, queues never removed in training | 0.672 | – | 0.472 | – |
| Tickets, queue removed, second experiment | 0.730 | 0.514 | – | 0.488 b |
| Out-of-scope questions written by people | 0.987 | 0.963 | – | 0.962 b |
| Foreign texts (film reviews, newsgroup posts) | 0.612 | 0.596 | – | 1.000 b |
| Encoder | Output layer | Correct | Ranking | AUROC | FPR95 | Silent | False | ECE |
|---|---|---|---|---|---|---|---|---|
| decisions | mistakes | rejections | ||||||
| Released | Fixed zero, | 69.7 | 76.0 | 0.861 | 48.3 | 5.2 | 23.8 | 15.4 |
| Constant empty weight, | 69.2 | 75.3 | 0.850 | 63.5 | 5.8 | 23.4 | 16.5 | |
| Learned rejection class | 69.6 | 75.9 | 0.859 | 50.3 | 3.9 | 21.7 | 15.1 | |
| Softmax with threshold | 61.3 | 75.9 | 0.711 | 84.9 | 28.9 | 29.8 | 27.1 | |
| Fixed zero, ReLU depths | 69.3 | 72.3 | 0.850 | 81.0 | 3.8 | 19.8 | 15.6 |
| Encoder | Fixed zero minus | Correct decisions | AUROC | FPR95 |
|---|---|---|---|---|
| Released | Constant empty weight, | +0.4 [+0.1, +0.7] ∗ | +0.011 [+0.010, +0.013] ∗ | 15.6 [ 17.2, 13.8] ∗ |
| Learned rejection class | +0.1 [ 0.2, +0.4] | +0.002 [+0.001, +0.002] ∗ | 2.0 [ 3.0, 0.8] ∗ | |
| Softmax with threshold | +8.6 [+7.5, +9.7] ∗ | +0.150 [+0.138, +0.162] ∗ | 36.5 [ 39.3, 33.7] ∗ | |
| Fixed zero, ReLU depths | +0.3 [ 0.1, +0.7] | +0.005 [+0.004, +0.005] ∗ | 4.4 [ 6.2, 2.8] ∗ | |
| Original | Constant empty weight, | +0.1 [ 0.1, +0.2] | +0.001 [+0.000, +0.001] ∗ | 0.2 [ 0.6, +0.4] |
| Learned rejection class | +0.6 [+0.1, +1.2] | +0.003 [+0.001, +0.005] ∗ | +0.5 [ 0.7, +1.6] |
| System | Correct | Ranking | Silent | FPR95 | AUROC | Order | Latency |
|---|---|---|---|---|---|---|---|
| decisions | mistakes | changes | (median) | ||||
| ZeroPoint | 63.6 | 68.5 | 6.9 | 53.6 | 0.808 | 0.0 | 21 ms |
| Laya | 59.8 | 72.2 | 21.4 | 60.1 | 0.794 | 14.9 | 383 ms |
| Semantic Router | 34.6 | 56.8 | – | 69.0 | 0.740 | 0.2 | 9 ms |
| Zero-shot NLI | 28.6 | 50.6 | 30.2 | 87.1 | 0.666 | 0.0 | 1,316 ms |
| Test | Options | Correct decisions | Ranking | Silent mistakes | AUROC |
|---|---|---|---|---|---|
| CLINC150 | 20 | 90.5 | 95.3 | 0.7 | 0.973 |
| 50 | 83.5 | 92.0 | 6.0 | 0.934 | |
| 100 | 74.0 | 85.3 | 4.7 | 0.915 | |
| 150 | 68.2 | 83.0 | 7.0 | 0.898 | |
| MTOP (Spanish) | 20 | 53.3 | 60.0 | 4.7 | 0.659 |
| 50 | 34.7 | 42.7 | 13.3 | 0.564 |
| Tickets, queue removed | Tickets, unrelated texts | ||||||
| Layers trained | Examples | Ranking | Silent | AUROC | Ranking | Silent | AUROC |
| None (released) | – | 24.3 | 19.3 | 0.508 | 28.7 | 0.3 | 0.993 |
| All | 100 | 36.0 | 6.7 | 0.652 | 44.7 | 2.7 | 0.918 |
| All | 300 | 38.0 | 2.0 | 0.692 | 42.0 | 1.7 | 0.952 |
| All | 1,000 | 39.7 | 0.0 | 0.705 | 46.7 | 0.3 | 0.947 |
| Last 4 of 12 | 300 | 36.0 | 0.0 | 0.594 | 46.7 | 0.0 | 0.991 |
| Source of variation | Model | Measurement | Result |
|---|---|---|---|
| Training seed (2 seeds) | preliminary | mean , 8 tests | 1.4 pts, 0.005 AUROC |
| Order (5 orders, 1,500 cases) | checkpoint | max | 0 |
| CPU threads (1, 2 and 4) | checkpoint | max | |
| GPU against CPU | checkpoint | max | |
| Float16 and int8 storage | released | decisions changed | 11 of 7,537 |
| Operating system, Python | released | max |
Energy-Based Models
Energy-based models assign scalar energies to configurations, defining probabilities and guiding generation, inference, prediction, and associative memory.
See all 99 papers