Why Linear Models Can Still Work in a Complex and Changing World
Organizations: Texas A&M University, USA · Tohoku University, Japan
Abstract
Linear models can predict well in complex domains even when underlying mechanisms are nonlinear. We argue that this can occur because the complexity of the world exceeds the complexity that can be learned reliably from finite, partially observed data. Flexible models can exploit curvature, thresholds, and interactions that fit one hospital, period, population, policy setting, or measurement process but fail elsewhere. When this instability lies mainly in the nonlinear component, omitting it can give linear models lower prediction error in a new environment. We connect this claim to work on simple linear rules, bias and variance, regularization, best linear prediction, measurement error, distribution shift, invariance, Rashomon sets, and underspecification. Proposition 1 gives a simple squared-error case in which a nonlinear rule learned in the training environment raises target error because it transfers too weakly; finite-sample estimation error can strengthen this result. Partial observation alone is not enough: the link between observed variables and unobserved factors must vary across environments. Stable nonlinear structure favors flexible models, and linear models remain vulnerable to linear spurious relationships. Model complexity should therefore be chosen by cross-environment stability as well as training-data accuracy.
Figures & tables
| Research area | What previous research establishes | What it does not establish by itself |
|---|---|---|
| Simple linear rules and mechanical prediction | Methods that do not estimate fine differences among weights can reduce estimation error and can sometimes be more accurate than human judgment or complex weighting | That linear models always perform better in unseen environments |
| Bias–variance and regularization | Restricting the relationships that can be learned can reduce the risk of learning accidental patterns from finite data | That models with many parameters are necessarily worse |
| Best linear prediction | The best linear approximation can be defined even when the true relationship is nonlinear | That the same linear approximation remains best across environments |
| Measurement error and unobserved factors | Missing information can make complex rules difficult to establish and can change the meaning of observed variables across environments | That missing information makes the prediction function linear |
| Environmental change and shortcut learning | A feature that is useful in the training environment can cease to work in the deployment environment | That every relationship that fails to transfer is nonlinear |
| Methods for learning relationships that remain stable across environments | Learning procedures can prepare for specified kinds of environmental change | That they can withstand every possible unseen change |
Model Selection
Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.
See all 96 papers