From Garbage to Gold: A Data-Architectural Theory of Predictive Robustness

Terrence J. Lee-St. John; Jordan L. Lawson; Bartlomiej Piechowski-Jozwiak

cs.LGMar 8, 2026

From Garbage to Gold: A Data-Architectural Theory of Predictive Robustness

Authors: Terrence J. Lee-St. John, Jordan L. Lawson, Bartlomiej Piechowski-Jozwiak

Organizations: Independent Researcher & Institute for Healthier Living Abu Dhabi, Abu Dhabi, UAE · Independent Researcher & Brown University, Center for Computation and Visualization, Providence, RI, USA

Abstract

Tabular machine learning presents a paradox: modern models achieve state-of-the-art performance using high-dimensional (high-D), collinear, error-prone data, defying the "Garbage In, Garbage Out" mantra. To help resolve this, we synthesize principles from Information Theory, Latent Factor Models, and Psychometrics, clarifying that predictive robustness arises not solely from data cleanliness, but from the synergy between data architecture and model capacity. Partitioning predictor-space "noise" into "Predictor Error" and "Structural Uncertainty" (informational deficits from stochastic generative mappings), we prove that leveraging high-D sets of error-prone predictors asymptotically overcomes both types of noise, whereas cleaning a low-D set is fundamentally bounded by Structural Uncertainty. We demonstrate why "Informative Collinearity" (dependencies from shared latent causes) enhances reliability and convergence efficiency, and explain why increased dimensionality reduces the latent inference burden, enabling feasibility with finite samples. To address practical constraints, we propose "Proactive Data-Centric AI" to identify predictors that enable robustness efficiently. We also derive boundaries for Systematic Error Regimes and show why models that absorb "rogue" dependencies can mitigate assumption violations. Linking latent architecture to Benign Overfitting, we offer a first step towards a unified view of robustness to Outcome Error and predictor-space noise, while also delineating when traditional DCAI's focus on label cleaning remains powerful. By redefining data quality from item-level perfection to portfolio-level architecture, we provide a theoretical rationale for "Local Factories" -- learning from live, uncurated enterprise "data swamps" -- supporting a deployment paradigm shift from "Model Transfer" to "Methodology Transfer'' to overcome static generalizability limitations.

Figures & tables

Term/Concept	Definition
Structural Architecture
Primary Structure	The hierarchical data-generating process $Y\leftarrow S^{(1)}\rightarrow S^{(2)}\rightarrow S^{\prime(2)}$ , where the $S^{(1)}$ layer drives both the outcome and the predictors.
$Y$ (Outcome/Label/Target)	The outcome being predicted. In this framework, it is driven solely by the latent layer $S^{(1)}$ .
$S^{(1)}$ (The Layer of First-Stage States/Latent States/Latent Drivers)	The set of $k$ latent, unobserved drivers (causes) of the system.
$s_{p}$ (Latent Configuration of $S^{(1)}$ )	A specific combination of values of the $k$ Latent States in $S^{(1)}$ (e.g., $\{1,0,1\}$ ). There are $2^{k}$ possible configurations, but only $K_{rlzd}$ are realized.
$S^{(2)}$ (The Layer of Second-Stage States/True Predictors/True Features)	The set of $m$ true, underlying predictor states generated by $S^{(1)}$ . These are the “perfect” versions of the predictor variables before Observational Error.

Table 1: Glossary of Framework Terminology and Concepts.

Aspect	Conventional Wisdom or Approach	Proposed Reinterpretation or Approach	Key Distinguishing Feature or Novelty Claim
High-D	Often viewed as a curse; leads to data sparsity, overfitting, computational burden. Aim to limit $m$ .	Can be an asset by providing more opportunity for novel and/or redundant information to improve the recovery of $S^{(1)}$ .	Reinterprets high-D as beneficial for $S^{(1)}$ recovery, not as inherently problematic.
Collinearity	Though not always harmful to prediction, considered primarily problematic for model interpretation. Mitigated by feature removal or transformation.	Informative Collinearity is beneficial because it can both improve reliability of recovering $S^{(1)}$ and increase the efficiency of asymptotic convergence. This is distinct from trivial collinearity.	Views a specific type of collinearity as a signal that can be exploited for robustness, rather than solely a statistical nuisance.
Predictor Error	A primary problem requiring aggressive data cleaning, validation, imputation, or removal of error-prone features.	Can be mitigated through thoughtful predictor-space design (P-DCAI) and appropriate modeling techniques when cleaning is impractical to scale.	Shifts focus from error removal to error mitigation through strategic use of data characteristics and model choice.
Structural Uncertainty	Not explicitly considered. Instead is treated as part of a monolithic noise term.	Formally separated from Observational Error as an independent source of predictive limits.	Proves that for a fixed set, Structural Uncertainty is a hard limit that persists despite perfect cleaning; only Breadth resolves it.
Feature Selection Goal	Minimal set of relevant, ideally uncorrelated features that directly and strongly predict $Y$ . Prioritizes parsimony and interpretability.	Strategic selection of sufficiently numerous, novel, and informatively redundant predictor variables. When the outcome is known and has high fidelity, features can be prioritized by their ability to resolve the predictive weakest link.	Goal shifts from direct prediction of $Y$ to robust $S^{(1)}$ estimation.
Data Collection Strategy	Focus on acquiring a few clean, direct, and strong correlates of $Y$ .	Prioritize acquiring multiple, diverse, informatively redundant predictor variables, as well as those with diverse error sources. Considers targeted redundancy for weakly estimated $S^{(1)}$ states/configurations.	Values a vast, diverse, and structurally informed portfolio of (potentially imperfect) predictors over a few perfectly clean measures.

Table 2: Comparative Analysis of Proposed Framework vs. Conventional Approaches.

Figure 3: $S^{\prime(2)}$ Spectral Analysis: Causally Consistent Variables.

Figure 4: $S^{\prime(2)}$ Spectral Analysis: Chaotic Generative Variables.

Figure 5: Breadth vs. Depth: Causally Consistent Variables.

Figure 6: Breadth vs. Depth: Chaotic Generative Variables.

Figure 7: Comparative Spectral Analysis: Causally Consistent Variables.

Figure 8: Comparative Spectral Analysis: Chaotic Generative Variables.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Alternative Causal Structures	Alignment with Proposal
Reversed Direction of the Entire Framework ( $Y\leftarrow S^{(1)}\leftarrow S^{(2)}$ )	High
Direct Causation ( $A\rightarrow B$ )	Perfect
Multiple $S^{(2)}$ States Commonly Caused by a Single $S^{(1)}$ State ( $X_{i}^{(1)}\rightarrow S^{(2)}$ )	Perfect
$Y$ and $S^{(2)}$ States Commonly Caused by $S^{(1)}$ ( $Y\leftarrow S^{(1)}\rightarrow S^{(2)}$ )	Perfect
Indirect Causation/Mediation ( $A\rightarrow M\rightarrow B$ )	High
$Y$ as the Collider Caused by Multiple $S^{(1)}$ States ( $S^{(1)}\rightarrow Y$ )	Perfect

Appendix

Table 3: Proposal’s Alignment with Alternative Causal Structures.

Figure 9: Breadth vs. Depth (AUROC): Causally Consistent Variables.

Figure 10: Breadth vs. Depth (AUPRC): Causally Consistent Variables.

Figure 11: Breadth vs. Depth (AUROC): Chaotic Generative Variables.

Figure 12: Breadth vs. Depth (AUPRC): Chaotic Generative Variables.

Statistical Learning Theory

Statistical learning theory studies when data-driven algorithms generalize, how many samples they need, and what statistical or computational limits apply.

See all 578 papers

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI
Mar 26, 2026Steffen LukasDistribution Shift Robustness Model Selection
Charité – Universitätsmedizin Berlin, Germany
Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI
Sep 8, 2026Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie FuAI Agent Reliability Distribution Shift
School of Computing and Augmented Intelligence, Arizona State University, USA
Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training
Feb 17, 2026Kevin Wang, Hongqian Niu, Didong LiModel Collapse Generative Modeling
Department of Biostatistics, University of North Carolina at Chapel Hill

Mar 26, 2026J/K move · Enter open · S save

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

Steffen Lukas

Charité – Universitätsmedizin Berlin, Germany

Distribution Shift Robustness Model Selection AI Safety Concept Drift

Open paper AskPDF arXiv

Foundation models excel in stable environments, yet often fail where reliability matters most: medicine, finance, and policy. This Fidelity Paradox is not just a data problem; it is structural. In domains where rules change over time, extra model capacity amplifies noise rather than capturing signal. We introduce Epistemic Compression: the principle that robustness emerges from matching model complexity to the shelf life of the data, not from scaling parameters. Unlike classical regularization, which penalizes weights post hoc, Epistemic Compression enforces parsimony through architecture: the model structure itself is designed to reduce overfitting by making it architecturally costly to represent variance that exceeds the evidence in the data. We operationalize this with a Regime Index that separates Shifting Regime (unstable, data-poor; simplicity wins) from Stable Regime (invariant, data-rich; complexity viable). In an exploratory synthesis of 15 high-stakes domains, this index was concordant with the empirically superior modeling strategy in 86.7% of cases (13/15). High-stakes AI demands a shift from scaling for its own sake to principled parsimony.

Abstract

Statistical Learning Theory

Explore similar work

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training