Organizations: MIPT, Dolgoprudny, 141701, Russia · NRC “Kurchatov Institute", 123182, Moscow, Russia · Institute for Information Transmission Problems, Moscow 127994, Russia · ITEP, B.Cheremushkinskaya, 25, 177259, Moscow, Russia
Many scientific problems in the context of machine learning can be reduced to the search of polynomial answers in appropriate variables. The Hevisidization of arbitrary polynomial is actually provided by one-and-the same two-layer expression. What prevents the use of this simple idea is the fatal degeneracy of the Heaviside and sigmoid expansions, which traps the steepest-descent evolution at the bottom of canyons, close to the starting point, but far from the desired true minimum. This problem is unavoidable and can be formulated as a peculiar uncertainty principle -- the sharper the minimum, the smoother the canyons. It is a direct analogue of the usual one, which is the pertinent property of the more familiar Fourier expansion. Standard machine learning software fights with this problem empirically, for example, by testing evolutions, originated at randomly distributed starting points and then selecting the best one. Surprisingly or not, phenomena and problems, encountered in ML application to science are pure scientific and belong to physics, not to computer science. On the other hand, they sound slightly different and shed new light on the well-known phenomena -- for example, extend the uncertainty principle from Fourier and, later, wavelet analysis to a new peculiar class of nearly singular sigmoid functions.
Figures & tables
Figure 1: Two different stable network parameters states (shown are about 150 ordered values), as the results of training for the same dataset but with different initial values
Figure 2: Matching of 400 training target determinant values and a trained network output
Figure 3: Mismatch of the output returned by the same network, but taken for another set of 400 testing values
Figure 4: The minimal curve γ for L(a,b;1,1)
Figure 5: The look of the plot of L(a,b) in the region a∈[−0.5,3.5],b∈[−0.5,3.5] for K=12 .
Figure 6: σ(x)−x for α=0.2,1.0,1.1
Figure 7: dependency on the sign of ξ
Figure 8: dependency on ϵ=1,0.5,0.4
Figure 9: Training with 10 samples. Left: values of b0,w0,b2,w2 , middle: values of w1 (close to diagonal), right: matching training sample (orange y(a) completely overlap blue original values of a )
Figure 10: Training with 40 samples. Left: values of b0,w0,b2,w2 (practically the same as for 10 samples), middle: values of w1 (essentially moved away from diagonal), right: matching training sample (orange y(a) completely overlap blue original values of a )
Figure 11: Evolution of 3×3 -determinant model output (top) and some learnable parameters (bottom) after 2000 and 6000 training steps correspondingly: the case of initializing parameters according to determinant ansatz - essential decreasing loss values, minor corrections of network weights
Figure 12: Evolution of 3×3 -determinant model output (top) and some learnable parameters (bottom) after 2000 and 6000 training steps correspondingly: the case of randomly initialized parameters - high remaining loss value, active changes of network weights
Traditional machine learning relies on explicit models and domain assumptions, often limiting flexibility and interpretability. We introduce a "model-free" framework of instance-based learning using surprisal (information theoretic uncertainty) to directly analyze and perform inferences from raw data, eliminating distribution modeling, reducing bias, and enabling efficient updates including direct edits and deletion of training data. By quantifying relevance through uncertainty, the approach enables generalizable inference across tasks including generative inference, causal discovery, anomaly detection, and time series forecasting. Our approach emphasizes traceability, interpretability, and data-driven decision making, offering a unified, human-understandable framework for machine learning, can be adapted to virtually all machine learning tasks, achieving competency across all and state-of-the-art performance on anomaly detection and some supervised tasks. The mathematical foundations create a "physics" of information, which enable these techniques to apply effectively to a wide variety of complex data types, including missing data. Our results suggest that this may be a viable foundation upon which to build scalable machine learning and artificial intelligence that can maintain human understandability of the underlying mechanics.
Christopher J. Hazard, Michael Resnick, Jacob Beel +7
Reliable uncertainty evaluation is essential for assessing the trustworthiness of machine learning (ML) predictions, particularly in safety-critical applications. This paper explores conceptual analogies between uncertainty evaluation in ML and measurement science, drawing on the evolution of metrology from error-based approaches to the framework established by the Guide to the Expression of Uncertainty in Measurement (GUM). Particular attention is devoted to the distinction between process-level and result-level quality and to the limitations of the conventional classification of ML uncertainty into aleatoric and epistemic components. It is argued that this dichotomy, similarly to the traditional distinction between random and systematic measurement errors, is inherently model-dependent and provides an insufficient basis for a unified uncertainty evaluation framework. Further analogies between machine learning and measurement field of knowledge are identified concerning distribution shifts, anomalous operating conditions, and the validity of uncertainty evaluation. Building on the proposed considerations, the paper advocates a metrology-inspired approach based on the systematic identification, quantification, and propagation of uncertainty contributions. By extending the methodology already established in measurement science to ML, this approach could offer a potentially rigorous, consistent, and transferable framework for evaluating the trustworthiness of individual predictions.
Dario Petri, Luca Mari
Department of Industrial Engineering of the Università di Trento, Trento, Italy · Department of Philosophy of the Università Cattolica di Milano, Milano, Italy
Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the standard information-theoretic framework. However, in practice, two critical issues arise: entanglement (AU and EU are highly correlated) and epistemic collapse (EU magnitude shrinks with increasing model capacity). We analyze this framework on a functional level and discover that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities. We characterize how this infeasible region scales with the number of classes and Monte Carlo samples N (e.g., from ensembles with N members), revealing it is bounded by AU≤log(2)/N. Crucially, the infeasible region's boundary helps explain epistemic collapse: when model confidence is high, AU>EU is guaranteed by this fundamental structural limitation. Our findings show that increasing ensemble size mitigates epistemic collapse by reducing the infeasible area. Lastly, we caution against interpreting AU and EU as independent quantities in low AU regimes, since we show they are coupled when AU≤log(2)/N.
Jakob Lønborg Christensen, Christian F. Baumgartner, Morten Rieger Hannemose +2
DTU Compute Technical University of Denmark · Faculty of Health Sciences and Medicine University of Lucerne