Statistical learning theory and Occam's razor: Regularization

Tom F. Sterkenburg

stat.MLAug 4, 2026

Statistical learning theory and Occam's razor: Regularization

Authors: Tom F. Sterkenburg

Abstract

The principle of Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning. In either field, however, a justification for the principle has been elusive. In this paper, building on an earlier "core argument," I spell out a justification from statistical learning theory for the procedure of regularization: for trading off fit for simplicity. The means-ends argument is that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit. This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.

Explore similar work

Benign interpolation and Occam's razor
Aug 4, 2026Tom F. Sterkenburg, Daniel A. Herrmann, Jan-Willem RomeijnOverfitting Interpolation
Occam's Razor is Only as Sharp as Your ELBO
Apr 28, 2026Ethan Harvey, Michael C. HughesModel Selection Bayesian
A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
Sep 6, 2021Yehuda Dar, Vidya Muthukumar, Richard G. BaraniukOver-Parameterized Neural Networks Machine Learning

Aug 4, 2026 · cs.LGJ/K move · Enter open · S save

Benign interpolation and Occam's razor

Tom F. Sterkenburg, Daniel A. Herrmann, Jan-Willem Romeijn

Contemporary deep learning methods generalize well even when they fit their training data perfectly, a phenomenon known as benign interpolation. This phenomenon cannot be accounted for by classical statistical learning theory and has prompted a range of attempted new explanations in the statistics and machine learning literature. A common feature of these new proposals is an appeal to a simplicity preference among interpolating models, often presented as a form of Occam's razor. We clarify this debate for a philosophical audience and argue that this new appeal to simplicity creates an explanatory gap. The classical theory offers theorems which connect the simplicity of model classes to good generalization, thus underwriting methodological simplicity norms. The new accounts instead appeal to properties of individual models, which they interpret as a kind of simplicity. Lacking a provable connection to generalization, it is the name "simplicity" that does the work a theorem used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.

Overfitting Interpolation Benign Methodology

Open paper Ask about it PDF

Abstract

Explore similar work

Benign interpolation and Occam's razor

Occam's Razor is Only as Sharp as Your ELBO

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning