A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring)

Christopher J. Hazard; Michael Resnick; Jacob Beel; Jack Xia; Cade Mack; Dominic Glennie; Matthew Fulp; David Maze

cs.LGOct 26, 2025

A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring)

Authors: Christopher J. Hazard, Michael Resnick, Jacob Beel, Jack Xia, Cade Mack, Dominic Glennie, Matthew Fulp, David Maze, +2 more

Organizations: Howso Incorporated

Abstract

Traditional machine learning relies on explicit models and domain assumptions, often limiting flexibility and interpretability. We introduce a "model-free" framework of instance-based learning using surprisal (information theoretic uncertainty) to directly analyze and perform inferences from raw data, eliminating distribution modeling, reducing bias, and enabling efficient updates including direct edits and deletion of training data. By quantifying relevance through uncertainty, the approach enables generalizable inference across tasks including generative inference, causal discovery, anomaly detection, and time series forecasting. Our approach emphasizes traceability, interpretability, and data-driven decision making, offering a unified, human-understandable framework for machine learning, can be adapted to virtually all machine learning tasks, achieving competency across all and state-of-the-art performance on anomaly detection and some supervised tasks. The mathematical foundations create a "physics" of information, which enable these techniques to apply effectively to a wide variety of complex data types, including missing data. Our results suggest that this may be a viable foundation upon which to build scalable machine learning and artificial intelligence that can maintain human understandability of the underlying mechanics.

Figures & tables

Figure 1: Sluice gate (H Stamper / sluice gate / CC BY-SA 2.0, via Wikipedia).

Figure 2: An example of prediction performance where the horizontal axis represents the fraction of data elements that are null.

Figure 3: An example of inspecting a prediction.

Figure 4: An example of inspecting the most anomalous records.

Classification	Howso (targetless)	Howso (single-target)	XGBoost	LightGBM
Mean MCC $\uparrow$	0.599	0.650	0.659	0.651
Mean Accuracy $\uparrow$	0.807	0.829	0.829	0.825
Mean Precision $\uparrow$	0.755	0.785	0.793	0.772
Mean Recall $\uparrow$	0.737	0.769	0.779	0.767
Mean F1 $\uparrow$	0.744	0.777	0.785	0.768

Table 1: Classification Results across 145 PMLB Datasets. Top score bolded.

Regression	Howso (targetless)	Howso (single-target)	XGBoost	LightGBM
Spearman $\uparrow$	0.918	0.924	0.919	0.916
$R^{2}$ $\uparrow$	0.848	0.834	0.808	0.787

Table 2: Regression Results across 258 PMLB Datasets. Top score bolded.

Panel of Figure 5 — Figure 5: ROAR curves comparing Howso Prediction Contributions, Howso Accuracy Contributions, and SHAP on diabetes (MAE) and breast cancer (Accuracy) with XGBoost; 3 seeds, no tuning.

Figure 6: An example of inductive bias influencing feature importance, LightGBM SHAP versus Howso prediction contributions on a generated data set.

Metric	PR-AUC	ROC-AUC
CBLOF ( He et al., 2003 )	0.1357	0.5851
COF ( Pokrajac et al., 2008 )	0.2361	0.7783
COPOD ( Li et al., 2020 )	0.2847	0.8262
ECOD ( Li et al., 2022 )	0.2895	0.8331
FeatureBagging ( Lazarevic and Kumar, 2005 )	0.2818	0.8253
HBOS ( Goldstein and Dengel, 2012 )	0.2826	0.8011

Table 3: Aggregated Anomaly Detection Results. Top scores are in bold.

Figure 7: Drone approaching a navigation point behind stairs.

Figure 8: Drone navigating around the stairs.

Figure 9: Driving while avoiding crowd of pedestrians.

Figure 10: Driving while staying within tight bounds.

Dataset	Measure	GES	Howso	LiNGAM	PC
GCM Microservice Architecture	F1	0.750	0.805	0.867	0.963
	Jaccard	0.600	0.678	0.765	0.929
GCM Online Shop	F1	0.552	0.665	0.581	0.556
	Jaccard	0.381	0.498	0.409	0.385
Sachs	F1	0.549	0.641	0.635	0.533
	Jaccard	0.378	0.472	0.465	0.364

Table 4: Causal discovery results scored according to undirected relationships.

Dataset	Measure	GES	Howso	LiNGAM	PC
GCM Microservice Architecture	F1	0.667	0.758	0.828	0.923
	Jaccard	0.500	0.615	0.706	0.857
GCM Online Shop	F1	0.240	0.435	0.308	0.471
	Jaccard	0.136	0.279	0.182	0.308
Sachs	F1	0.267	0.434	0.436	0.455
	Jaccard	0.154	0.279	0.279	0.294

Table 5: Causal discovery results scored according to directed relationships.

Figure 11: A causal graph discovered for a fluid dynamics simulation using a small subset of features.

Metric	Original	Generated
Accuracy	0.959	0.925
F1-Score	0.959	0.927
MCC	0.923	0.868
Spearman	0.921	0.908

Table 6: Synthetic Data Generation Results: Overall Performance Comparison. Aggregated across 5 classification and 4 regression datasets.

Dataset	Reduction (%)	Howso Ablated	Howso With Subsample ∗	Howso With All Data
connect_4	63.12	0.4968	0.4765	0.5696
adult	66.67	0.4907	0.5085	0.5022
sleep	87.89	0.5731	0.6441	0.6353
fars	51.75	0.6918	0.6860	0.7082
krkopt	24.91	0.6094	0.6248	0.6580
letter	27.74	0.9420	0.9524	0.9548

Table 7: Classification results comparing ablated Howso to Howso with all data using Matthews Correlation Coefficient (MCC).

Dataset	Reduction (%)	Howso Ablated	Howso With Subsample ∗	Howso With All Data
1201_BNG_breastTumor	53.29	0.3348	0.2996	0.3539
feynman_test_1	35.95	0.9915	0.9889	0.9948
feynman_test_20	23.22	0.9866	0.9780	0.9855
215_2dplanes	61.97	0.9653	0.9699	0.9703
344_mv	47.04	0.9973	0.9986	0.9987
564_fried	62.52	0.9538	0.9566	0.9655

Table 8: Regression results comparing ablated Howso to Howso with all data using Spearman correlation coefficient.

Model	Clean Acc	K=3 Adv Acc	K=1 Adv Acc
Howso	0.819	0.596 ( $-0.223$ )	0.737 ( $-0.082$ )
Logistic Regression	0.795	0.684 ( $-0.111$ )	0.742 ( $-0.053$ )
Decision Tree	0.783	0.564 ( $-0.220$ )	0.641 ( $-0.142$ )
kNN	0.796	0.436 ( $-0.360$ )	0.757 ( $-0.039$ )
LightGBM	0.835	0.619 ( $-0.216$ )	0.717 ( $-0.119$ )
XGBoost	0.834	0.633 ( $-0.201$ )	0.691 ( $-0.143$ )

Table 9: Mean accuracy under adversarial attack (27 datasets)

Panel of Figure 12 — Figure 12: Distribution of accuracy degradation ( $\Delta$ Accuracy) by model and attack type across 27 datasets. Box plots show median, IQR, and individual dataset results.

Figure 13: Four measures of performance on different volumes of the NYC arrest dataset.

Prompt	Completed Sentence	Notes
The Solar System has	The Solar System has been relatively well studied, initially through telescopes and then later by spacecraft	The source was the article titled Astronomy, verbatim, except stopping before the period at the end.
The Solar System has	The Solar System has been fairly stable for billions of years, it is technically chaotic, and may be considered a planet.	The source was the article Solar_System, changed at the end, gramatically correct, topically correct, but factually incorrect.
Jupiter is the	Jupiter is the most important deity in the Guatemalan Highlands at this time was Q’umarkaj, the capital of the aggressive K’iche’ kingdom.	The source was a mixture of articles on the Maya people, and the sentence has strong factual basis, though the grammar is a little off. The K’iche’ people and the Maya people in general tracked Jupiter’s orbit and incorporated it into their calendars. The K’iche’ kingdom was described in the Wikipedia texts as having subdued other Mayan peoples.
Jupiter is the	Jupiter is the largest of the Western Roman Empire, and the the king of the planet’s connection with Easter	Example of a nonsensical sentence that is grammatically correct except for the missing period.

Table 10: Typical sentence completions based on approximately 1 million words trained from Wikipedia.

Figure 14: An example of a number generated using the MNIST handwritten digits data set.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 25

Figure 26

Dataset	Accuracy	Precision	Recall	MCC
GAMETES_Epistasis_2_Way_1000atts_0.4H_EDM_1_EDM_1_1	0.5172	0.5202	0.5196	0.0398
GAMETES_Epistasis_2_Way_20atts_0.1H_EDM_1_1	0.5792	0.5853	0.5803	0.1656
GAMETES_Epistasis_2_Way_20atts_0.4H_EDM_1_1	0.7694	0.7797	0.7714	0.5510
GAMETES_Epistasis_3_Way_20atts_0.2H_EDM_1_1	0.5528	0.5578	0.5541	0.1118
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_50_EDM_2_001	0.6316	0.6351	0.6336	0.2687
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_75_EDM_2_001	0.6809	0.6839	0.6821	0.3660

Appendix

Table 11: Classification Results across 146 Datasets

Dataset	Accuracy	Precision	Recall	MCC
GAMETES_Epistasis_2_Way_1000atts_0.4H_EDM_1_EDM_1_1	0.5164	0.5173	0.5144	0.0316
GAMETES_Epistasis_2_Way_20atts_0.1H_EDM_1_1	0.6493	0.6604	0.6504	0.3106
GAMETES_Epistasis_2_Way_20atts_0.4H_EDM_1_1	0.7910	0.8215	0.7917	0.6124
GAMETES_Epistasis_3_Way_20atts_0.2H_EDM_1_1	0.6569	0.6598	0.6577	0.3175
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_50_EDM_2_001	0.6806	0.6810	0.6805	0.3615
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_75_EDM_2_001	0.7188	0.7295	0.7192	0.4485

Appendix

Table 12: Classification Results across 146 Datasets

Dataset	Accuracy	Precision	Recall	MCC
GAMETES_Epistasis_2_Way_1000atts_0.4H_EDM_1_EDM_1_1	0.4930	0.4941	0.4941	-0.0118
GAMETES_Epistasis_2_Way_20atts_0.1H_EDM_1_1	0.5920	0.5916	0.5914	0.1831
GAMETES_Epistasis_2_Way_20atts_0.4H_EDM_1_1	0.7339	0.7343	0.7342	0.4686
GAMETES_Epistasis_3_Way_20atts_0.2H_EDM_1_1	0.5603	0.5606	0.5607	0.1213
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_50_EDM_2_001	0.6393	0.6391	0.6392	0.2783
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_75_EDM_2_001	0.6813	0.6816	0.6817	0.3633

Appendix

Table 13: Classification Results across 146 Datasets

Dataset	Accuracy	Precision	Recall	MCC
GAMETES_Epistasis_2_Way_1000atts_0.4H_EDM_1_EDM_1_1	0.4916	0.4924	0.4924	-0.0151
GAMETES_Epistasis_2_Way_20atts_0.1H_EDM_1_1	0.6112	0.6117	0.6115	0.2232
GAMETES_Epistasis_2_Way_20atts_0.4H_EDM_1_1	0.7288	0.7301	0.7283	0.4583
GAMETES_Epistasis_3_Way_20atts_0.2H_EDM_1_1	0.5522	0.5528	0.5528	0.1055
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_50_EDM_2_001	0.6666	0.6673	0.6669	0.3343
GAMETES_Heterogeneity_20atts_1600_Het_0.4_0.2_75_EDM_2_001	0.6919	0.6919	0.6914	0.3834

Appendix

Table 14: Classification Results across 146 Datasets

Dataset	$R^{2}$	Spearman
1027_ESL	0.8033	0.9118
1028_SWD	0.2067	0.5568
1029_LEV	0.4665	0.7092
1030_ERA	0.3372	0.5449
1089_USCrime	0.7240	0.7908
1096_FacultySalaries	0.5929	0.8690

Appendix

Table 15: Regression Results across 258 Datasets

Dataset	$R^{2}$	Spearman
1027_ESL	0.8081	0.9116
1028_SWD	0.2573	0.5668
1029_LEV	0.4653	0.7088
1030_ERA	0.3304	0.5447
1089_USCrime	0.7283	0.7728
1096_FacultySalaries	0.6008	0.8553

Appendix

Table 16: Regression Results across 258 Datasets

Dataset	$R^{2}$	Spearman
1027_ESL	0.8024	0.9148
1028_SWD	0.3270	0.5942
1029_LEV	0.5360	0.7324
1030_ERA	0.3501	0.5588
1089_USCrime	0.5860	0.7279
1096_FacultySalaries	0.6673	0.8437

Appendix

Table 17: Regression Results across 258 Datasets

Dataset	$R^{2}$	Spearman
1027_ESL	0.8477	0.9292
1028_SWD	0.3713	0.6231
1029_LEV	0.5498	0.7410
1030_ERA	0.3559	0.5597
1096_FacultySalaries	0.1822	0.6449
1191_BNG_pbc	0.4284	0.6674

Appendix

Table 18: Regression Results across 253 Datasets

Dataset	Task	Method	Removed	Baseline	After	Change	Relative	Seeds
diabetes	Regression (MAE)	Prediction Contrib.	0% (0)	49.3	49.3	0.0	0.0%	3
		Prediction Contrib.	10% (1)	49.3	52.3	+3.6	+7.2%	3
		Prediction Contrib.	20% (2)	49.3	55.7	+6.4	+13.0%	3
		Prediction Contrib.	50% (5)	49.3	61.3	+12.0	+24.4%	3
		Accuracy Contrib.	0% (0)	49.3	49.3	0.0	0.0%	3
		Accuracy Contrib.	10% (1)	49.3	52.3	+3.0	+6.1%	3

Appendix

Table 19: ROAR Comparison Results: Howso Prediction Contributions, Howso Accuracy Contributions, and SHAP Performance by Dataset and Removal Level

Metric	Dataset	PR-AUC	ROC-AUC
CBLOF	10_cover	0.0932	0.5130
CBLOF	12_fault	0.2136	0.5605
CBLOF	14_glass	0.0621	0.6300
CBLOF	15_Hepatitis	0.1970	0.6559
CBLOF	16_http	0.0200	0.4207
CBLOF	18_Ionosphere	0.2368	0.7047

Appendix

Table 20: Anomaly Detection Results across Datasets

Metric	Value
Total Runs	5
Winning Runs	5
Win Rate (%)	100.00
Median Rounds to Win	273
IQR Rounds to Win	38.00
Median High Score	500.00

Appendix

Table 21: CartPole Summary Statistics

Iteration	Win	Rounds	Avg Score (Final)	Avg Score (Total)	High Score	Total Cases
0	Yes	100	252.42	252.42	500	22191
1	Yes	273	195.09	116.14	500	25564
2	Yes	791	196.79	123.35	500	71117
3	Yes	287	195.11	143.84	500	30427
4	Yes	249	196.16	136.71	500	26564

Appendix

Table 22: CartPole Detailed Run Results

Metric	Value
Total Runs	20
Winning Runs	14
Win Rate (%)	70.00
Median Rounds to Win	150.00
IQR Rounds to Win	0.00
Median High Score	9.00

Appendix

Table 23: Wafer-Thin-Mints Summary Statistics

Iteration	Win	Rounds	Avg Score (Final)	Avg Score (Total)	High Score	Total Cases
0	No	150	3.92	3.92	5	738
1	Yes	150	6.15	6.15	9	1130
2	No	150	4.43	4.43	8	815
3	No	150	5.74	5.74	9	1182
4	Yes	150	7.31	7.31	9	1342
5	Yes	150	7.44	7.44	9	1361

Appendix

Table 24: Wafer-Thin-Mints Detailed Run Results

Dataset	Original	Generated
1199_BNG_echoMonths	0.8296	0.8083
197_cpu_act	0.9832	0.9754
537_houses	0.9553	0.9464
574_house_16H	0.9144	0.9018

Appendix

Table 25: Synthetic Data Generation: Regression Datasets (Spearman Correlation)

Dataset	Accuracy		F1-Score		MCC
	Orig	Gen	Orig	Gen	Orig	Gen
letter	0.953	0.901	0.954	0.902	0.952	0.897
magic	0.879	0.851	0.879	0.852	0.732	0.667
mushroom	1.000	1.000	1.000	1.000	1.000	0.999
pendigits	0.992	0.986	0.992	0.986	0.992	0.984
ring	0.971	0.886	0.971	0.896	0.942	0.791

Appendix

Table 26: Synthetic Data Generation: Classification Datasets

Attack	Model	Clean Acc	Adv Acc	$\Delta$ (Adv - Clean)
K=3 (approx)	Decision Tree	0.783	0.564	$-0.220$
	Howso	0.819	0.596	$-0.223$
	k-NN	0.796	0.436	$-0.360$
	LightGBM	0.835	0.619	$-0.216$
	Logistic Regression	0.795	0.684	$-0.111$
	XGBoost	0.834	0.633	$-0.201$

Appendix

Table 27: Mean accuracy degradation under adversarial attack across 27 datasets

Attack	Model	Median $\Delta$	Mean $\Delta$	Min $\Delta$	Q1	Q3	Worst-5 Mean
K=3	Dec. Tree	$-0.167$	$-0.220$	$-0.530$	$-0.341$	$-0.104$	$-0.441$
	Howso	$-0.179$	$-0.223$	$-0.550$	$-0.349$	$-0.070$	$-0.482$
	k-NN	$-0.333$	$-0.360$	$-0.803$	$-0.460$	$-0.163$	$-0.728$
	LightGBM	$-0.166$	$-0.216$	$-0.535$	$-0.308$	$-0.116$	$-0.476$
	Log. Reg.	$-0.067$	$-0.111$	$-0.463$	$-0.167$	$-0.021$	$-0.308$
	XGBoost	$-0.195$	$-0.201$	$-0.550$	$-0.271$	$-0.090$	$-0.479$

Appendix

Table 28: Summary statistics of accuracy degradation by model and attack

Dataset	Attack	Clean Acc	Adv Acc	$\Delta$	N Test
biomed	K=3	1.000	0.833	$-0.167$	42
breast_w	K=3	0.950	0.771	$-0.179$	140
cars	K=3	1.000	0.696	$-0.304$	79
cleveland	K=3	0.443	0.262	$-0.180$	61
cloud	K=3	0.364	0.318	$-0.045$	22
confidence	K=3	0.933	0.533	$-0.400$	15

Appendix

Table 29: Howso per-dataset robustness results

Explore similar work

Machine Learning under Imperfect Data: Challenges and Methods
Sep 12, 2026Masoumeh ZareapoorLearning with Missing Data OOD Generalization
Shanghai Jiao Tong University, China
Machine Unlearning via Information Theoretic Regularization
Feb 8, 2025Shizhou Xu, Thomas StrohmerMachine Unlearning
Department of Mathematics University of California Davis Davis, CA 95616-5270, USA · Department of Mathematics Center of Data Science and Artificial Intelligence Research University of California Davis Davis, CA 95616-5270, USA
On the Structural Limits of Machine Learning Decision Systems: An Information-Theoretic, Interaction-Based, and Stochastic-Dynamical Perspective
Aug 13, 2026Nestor R. Barraza, Gabriel PenaStochastic Linear Dynamical Systems Statistical Learning Theory
Universidad Nacional de Tres de Febrero. Departamento de Ciencia y Tecnología. · Universidad de Buenos Aires. Facultad de Ingeniería.

Sep 12, 2026J/K move · Enter open · S save

Machine Learning under Imperfect Data: Challenges and Methods

Masoumeh Zareapoor

Shanghai Jiao Tong University, China

Learning with Missing Data OOD Generalization Unsupervised Domain Adaptation Weakly Supervised Learning

Open paper AskPDF arXiv

Machine-learning models are commonly developed under an assumption that training and test data are sufficiently complete, balanced, labelled, and drawn from compatible distributions. In practice, one or more of these conditions is often violated. Measurements may be missing or corrupted, rare classes may be poorly represented, supervision may be weak, and the deployment environment may differ from the training environment. These imperfections are usually treated as separate technical problems, although they alter learning through a small number of shared mechanisms: loss of information, biased empirical risk, ambiguous supervision, and unstable representations. This short survey organises representative methods around these mechanisms. It reviews reconstruction and generation, rebalancing and representation calibration, learning with limited supervision, adaptation across domains and modalities, and reliability under distribution change. The discussion highlights the limits of plausible reconstruction, benchmark-specific correction, and adaptation without trustworthy feedback. It concludes with directions for evidence-aware learning, uncertainty-preserving prediction, and evaluation that separates visual plausibility from decision utility.

Abstract

Explore similar work

Machine Learning under Imperfect Data: Challenges and Methods

Machine Unlearning via Information Theoretic Regularization

On the Structural Limits of Machine Learning Decision Systems: An Information-Theoretic, Interaction-Based, and Stochastic-Dynamical Perspective