From Garbage to Gold: A Data-Architectural Theory of Predictive Robustness

Terrence J. Lee-St. John; Jordan L. Lawson; Bartlomiej Piechowski-Jozwiak

cs.LGMar 8, 2026

From Garbage to Gold: A Data-Architectural Theory of Predictive Robustness

Authors: Terrence J. Lee-St. John, Jordan L. Lawson, Bartlomiej Piechowski-Jozwiak

Organizations: Independent Researcher & Institute for Healthier Living Abu Dhabi, Abu Dhabi, UAE · Independent Researcher & Brown University, Center for Computation and Visualization, Providence, RI, USA

Abstract

Tabular machine learning presents a paradox: modern models achieve state-of-the-art performance using high-dimensional (high-D), collinear, error-prone data, defying the "Garbage In, Garbage Out" mantra. To help resolve this, we synthesize principles from Information Theory, Latent Factor Models, and Psychometrics, clarifying that predictive robustness arises not solely from data cleanliness, but from the synergy between data architecture and model capacity. Partitioning predictor-space "noise" into "Predictor Error" and "Structural Uncertainty" (informational deficits from stochastic generative mappings), we prove that leveraging high-D sets of error-prone predictors asymptotically overcomes both types of noise, whereas cleaning a low-D set is fundamentally bounded by Structural Uncertainty. We demonstrate why "Informative Collinearity" (dependencies from shared latent causes) enhances reliability and convergence efficiency, and explain why increased dimensionality reduces the latent inference burden, enabling feasibility with finite samples. To address practical constraints, we propose "Proactive Data-Centric AI" to identify predictors that enable robustness efficiently. We also derive boundaries for Systematic Error Regimes and show why models that absorb "rogue" dependencies can mitigate assumption violations. Linking latent architecture to Benign Overfitting, we offer a first step towards a unified view of robustness to Outcome Error and predictor-space noise, while also delineating when traditional DCAI's focus on label cleaning remains powerful. By redefining data quality from item-level perfection to portfolio-level architecture, we provide a theoretical rationale for "Local Factories" -- learning from live, uncurated enterprise "data swamps" -- supporting a deployment paradigm shift from "Model Transfer" to "Methodology Transfer'' to overcome static generalizability limitations.

Figures & tables

Term/Concept	Definition
Structural Architecture
Primary Structure	The hierarchical data-generating process $Y\leftarrow S^{(1)}\rightarrow S^{(2)}\rightarrow S^{\prime(2)}$ , where the $S^{(1)}$ layer drives both the outcome and the predictors.
$Y$ (Outcome/Label/Target)	The outcome being predicted. In this framework, it is driven solely by the latent layer $S^{(1)}$ .
$S^{(1)}$ (The Layer of First-Stage States/Latent States/Latent Drivers)	The set of $k$ latent, unobserved drivers (causes) of the system.
$s_{p}$ (Latent Configuration of $S^{(1)}$ )	A specific combination of values of the $k$ Latent States in $S^{(1)}$ (e.g., $\{1,0,1\}$ ). There are $2^{k}$ possible configurations, but only $K_{rlzd}$ are realized.
$S^{(2)}$ (The Layer of Second-Stage States/True Predictors/True Features)	The set of $m$ true, underlying predictor states generated by $S^{(1)}$ . These are the “perfect” versions of the predictor variables before Observational Error.

Table 1: Glossary of Framework Terminology and Concepts.

Aspect	Conventional Wisdom or Approach	Proposed Reinterpretation or Approach	Key Distinguishing Feature or Novelty Claim
High-D	Often viewed as a curse; leads to data sparsity, overfitting, computational burden. Aim to limit $m$ .	Can be an asset by providing more opportunity for novel and/or redundant information to improve the recovery of $S^{(1)}$ .	Reinterprets high-D as beneficial for $S^{(1)}$ recovery, not as inherently problematic.
Collinearity	Though not always harmful to prediction, considered primarily problematic for model interpretation. Mitigated by feature removal or transformation.	Informative Collinearity is beneficial because it can both improve reliability of recovering $S^{(1)}$ and increase the efficiency of asymptotic convergence. This is distinct from trivial collinearity.	Views a specific type of collinearity as a signal that can be exploited for robustness, rather than solely a statistical nuisance.
Predictor Error	A primary problem requiring aggressive data cleaning, validation, imputation, or removal of error-prone features.	Can be mitigated through thoughtful predictor-space design (P-DCAI) and appropriate modeling techniques when cleaning is impractical to scale.	Shifts focus from error removal to error mitigation through strategic use of data characteristics and model choice.
Structural Uncertainty	Not explicitly considered. Instead is treated as part of a monolithic noise term.	Formally separated from Observational Error as an independent source of predictive limits.	Proves that for a fixed set, Structural Uncertainty is a hard limit that persists despite perfect cleaning; only Breadth resolves it.
Feature Selection Goal	Minimal set of relevant, ideally uncorrelated features that directly and strongly predict $Y$ . Prioritizes parsimony and interpretability.	Strategic selection of sufficiently numerous, novel, and informatively redundant predictor variables. When the outcome is known and has high fidelity, features can be prioritized by their ability to resolve the predictive weakest link.	Goal shifts from direct prediction of $Y$ to robust $S^{(1)}$ estimation.
Data Collection Strategy	Focus on acquiring a few clean, direct, and strong correlates of $Y$ .	Prioritize acquiring multiple, diverse, informatively redundant predictor variables, as well as those with diverse error sources. Considers targeted redundancy for weakly estimated $S^{(1)}$ states/configurations.	Values a vast, diverse, and structurally informed portfolio of (potentially imperfect) predictors over a few perfectly clean measures.

Table 2: Comparative Analysis of Proposed Framework vs. Conventional Approaches.

Figure 3: $S^{\prime(2)}$ Spectral Analysis: Causally Consistent Variables.

Figure 4: $S^{\prime(2)}$ Spectral Analysis: Chaotic Generative Variables.

Figure 5: Breadth vs. Depth: Causally Consistent Variables.

Figure 6: Breadth vs. Depth: Chaotic Generative Variables.

Figure 7: Comparative Spectral Analysis: Causally Consistent Variables.

Figure 8: Comparative Spectral Analysis: Chaotic Generative Variables.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Alternative Causal Structures	Alignment with Proposal
Reversed Direction of the Entire Framework ( $Y\leftarrow S^{(1)}\leftarrow S^{(2)}$ )	High
Direct Causation ( $A\rightarrow B$ )	Perfect
Multiple $S^{(2)}$ States Commonly Caused by a Single $S^{(1)}$ State ( $X_{i}^{(1)}\rightarrow S^{(2)}$ )	Perfect
$Y$ and $S^{(2)}$ States Commonly Caused by $S^{(1)}$ ( $Y\leftarrow S^{(1)}\rightarrow S^{(2)}$ )	Perfect
Indirect Causation/Mediation ( $A\rightarrow M\rightarrow B$ )	High
$Y$ as the Collider Caused by Multiple $S^{(1)}$ States ( $S^{(1)}\rightarrow Y$ )	Perfect

Appendix

Table 3: Proposal’s Alignment with Alternative Causal Structures.

Figure 9: Breadth vs. Depth (AUROC): Causally Consistent Variables.

Figure 10: Breadth vs. Depth (AUPRC): Causally Consistent Variables.

Figure 11: Breadth vs. Depth (AUROC): Chaotic Generative Variables.

Figure 12: Breadth vs. Depth (AUPRC): Chaotic Generative Variables.

Statistical Learning Theory

Statistical learning theory studies when data-driven algorithms generalize, how many samples they need, and what statistical or computational limits apply.

See all 578 papers

Mar 26, 2026

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

Foundation models excel in stable environments, yet often fail where reliability matters most: medicine, finance, and policy. This Fidelity Paradox is not just a data problem; it is structural. In domains where rules change over time, extra model capacity amplifies noise rather than capturing signal. We introduce Epistemic Compression: the principle that robustness emerges from matching model complexity to the shelf life of the data, not from scaling parameters. Unlike classical regularization, which penalizes weights post hoc, Epistemic Compression enforces parsimony through architecture: the model structure itself is designed to reduce overfitting by making it architecturally costly to represent variance that exceeds the evidence in the data. We operationalize this with a Regime Index that separates Shifting Regime (unstable, data-poor; simplicity wins) from Stable Regime (invariant, data-rich; complexity viable). In an exploratory synthesis of 15 high-stakes domains, this index was concordant with the empirically superior modeling strategy in 86.7% of cases (13/15). High-stakes AI demands a shift from scaling for its own sake to principled parsimony.

Steffen Lukas

Charité – Universitätsmedizin Berlin, Germany

Distribution Shift Robustness Model Selection AI Safety Concept Drift

Ask PDF

Sep 8, 2026

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Agentic AI systems built on large language models fail in two persistent ways that scaling does not fix: they break under distribution shift, and they cannot explain the decisions they make. We argue these are co-symptoms of one structural deficiency in the data lifecycle that governs how agents are trained, evaluated, and deployed. Observational interaction logs record what an agent did, not what it would have done otherwise. They encode spurious correlations without controlled variation, so they lack the counterfactual structure needed to separate causal signal from coincidence or to validate an explanation. No model-centric method can recover invariances the data never contained. We present Data-Centric Anchoring: robustness and interpretability should be engineered into the data environment, not extracted from models after training. Our central contribution is the Data-Centric Agentic Loop, a four-stage framework of Curate, Augment, Constrain, and Attribute. The ordering is structural, not stylistic. Curation precedes augmentation because generative models amplify whatever bias they are trained on. Augmentation precedes constraint because invariance objectives are vacuous without variation across environments to be invariant to. Attribution closes the loop, converting observed failures into targeted data interventions for the next iteration. Each stage manufactures the preconditions of the next, which makes the loop self-correcting rather than merely sequential. We ground the framework in a failure-driven taxonomy that links four core failure modes to the data lifecycle: spurious feature reliance, distribution-shift fragility, uncertainty miscalibration, and explanation unfaithfulness. We close with the limits of this approach and the open problems that stand between it and practical deployment at scale.

Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie Fu

School of Computing and Augmented Intelligence, Arizona State University, USA

AI Agent Reliability Distribution Shift LLM Interpretability LLM Agent Reliability

Ask PDF

Feb 17, 2026

Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training

As artificial intelligence (AI)-generated content proliferates, models are increasingly trained on their own outputs, risking progressive degradation or collapse. In this article, we provide the first positive, rigorous theoretical results, to the best of our knowledge, showing that under model-agnostic mild conditions, the model converges to the true data-generating distribution. The convergence rate is the minimum of the model's intrinsic rate and the fraction of real data at each training iteration, revealing a phase transition between data-limited and model-limited regimes. We further show that, for biased real data, correcting the bias prevents the persistence and amplification of early bias over training iteration. Extensive experiments across simulations, real images and texts validate our theoretical framework, establishing quantitative conditions for long-term AI stability in contaminated environments.

Kevin Wang, Hongqian Niu, Didong Li

Department of Biostatistics, University of North Carolina at Chapel Hill

Model Collapse Generative Modeling Data Contamination in Language Models

Ask PDF

Abstract

Statistical Learning Theory

Explore similar work

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training