Assumption-lean logistic regression with missing covariates

Jyotishka Ray Choudhury; Kabir Aladin Verchand; Richard J. Samworth; Ashwin Pananjady

stat.MLOct 5, 2026

Assumption-lean logistic regression with missing covariates

Authors: Jyotishka Ray Choudhury, Kabir Aladin Verchand, Richard J. Samworth, Ashwin Pananjady

Organizations: School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Data Sciences and Operations, University of Southern California · Statistical Laboratory, University of Cambridge · School of Electrical and Computer Engineering, Georgia Institute of Technology

Abstract

Missing covariates are frequently encountered in supervised learning problems, and classical methods for estimation using such data use carefully chosen imputation schemes for missing data, or likelihood approximations that lead to nonconvex $M$ -estimation problems. These methods and their relatives are suitable for scenarios in which the covariate distribution is known, and more broadly, have enjoyed tremendous success in linear models. But even in basic nonlinear problems such as logistic regression in moderate dimensions, such methods can experience drastic failure modes when the covariate distribution is unknown. Motivated by the need for reliable alternatives, we consider the problem of parameter estimation in logistic regression with missing covariates. Crucially, we operate in the assumption-lean setting where the covariate distribution is unknown (but bounded). We design a stochastic approximation method that is based on $Z$ -estimation with a novel monotone operator, and establish that our algorithm is computationally efficient and achieves provable signal recovery at parametric rates under the hypothesis that covariates are missing completely at random. Our theory sharply characterizes the $\ell_2^2$ risk of the estimator in terms of the missingness profile, accommodating heterogeneous observation probabilities. Importantly, it shows that our method always outperforms the de facto ``complete-case'' estimator that ignores observations with any missing data. Even in the setting with homogeneous missingness (in which each covariate is observed independently with probability $q$ ), our bounds exhibit intricate and nonstandard dependence on $q$ that can yield significant improvements over using only complete cases. We complement our upper bounds with new information-theoretic lower bounds that show that this intricate dependence on $q$ is fundamental in a minimax sense.

Figures & tables

Panel of Figure 1 — Figure 1 : Estimation error comparison under MCAR covariate missingness with coordinate-wise observation probability $q$ . The oracle logistic MLE has access to the complete, unmasked covariates, and thus serves as the gold standard. The remaining curves correspond to the complete-case MLE, logistic MLE after zero or Gaussian conditional-mean imputation, the observed-data MLE under a moment-matched Gaussian model, and our proposed SA-based estimator. The curves plot the RMSE $\|\widehat{\theta}-\theta^{\star}\|_{2}$ over $100$ replications against the sample size $n$ . Shaded regions give pointwise $95\%$ confidence intervals across replications.

Explore similar work

Sep 23, 2025cs.DS

Linear Regression under Missing or Corrupted Coordinates

We study multivariate linear regression under Gaussian covariates in two settings, where data may be erased or corrupted by an adversary under a coordinate-wise budget. In the incomplete data setting, an adversary may inspect the dataset and delete entries in up to an

η

-fraction of samples per coordinate; a strong form of the Missing Not At Random model. In the corrupted data setting, the adversary instead replaces values arbitrarily, and the corruption locations are unknown to the learner. Despite substantial work on missing data, linear regression under such adversarial missingness remains poorly understood, even information-theoretically. Unlike the clean setting, where estimation error vanishes with more samples, here the optimal error remains a positive function of the problem parameters. Our main contribution is to characterize this error up to constant factors across essentially the entire parameter range. Specifically, we establish novel information-theoretic lower bounds on the achievable error that match the error of (computationally efficient) algorithms. A key implication is that, perhaps surprisingly, the optimal error in the missing data setting matches that in the corruption setting-so knowing the corruption locations offers no general advantage.

Ilias Diakonikolas, Jelena Diakonikolas, Daniel M. Kane +2

University of Wisconsin-Madison · University of California, San Diego · University of California, Davis

Corrupted Inputs Missing Data Covariates Information-Theoretic Limits

Ask PDF

May 19, 2026stat.ML

Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data

Stochastic gradient methods are central to modern large-scale learning, but their use with incomplete covariates remains delicate since imputation schemes generally introduce systematic gradient biases, as shown for linear models. In this work, we prove that all parametric models exhibit similar gradient bias for various imputation procedures and characterize exactly the dependence on the missingness ratio vector

p

, with

O(\|p\|)

as the leading term. We exploit this analysis to propose a simple debiasing procedure for stochastic gradient descent (SGD) with missing values based on Richardson extrapolation, which leverages the exact expression of the gradient bias. The key idea is to \emph{deliberately add missingness}: from an already incomplete observation, we generate a further-thinned version at a higher, controlled missingness level, and combine the two resulting stochastic gradients to cancel the leading bias term. We prove that one Richardson step reduces the gradient bias from

O(\|p\|)

O(\|p\|^2)

under several missingness scenarios. Our proposed method is computationally efficient, model-agnostic and applies to any parametric loss whose stochastic gradient can be computed after imputation. Furthermore, when missing indicators are independent, the population gradient bias is a multilinear polynomial in

p

and depends only on population gradient errors induced by declaring a single coordinate missing. In this case, our method generalizes to a multi-step Richardson procedure which recursively cancels higher-order terms. Empirically, Richardson debiasing improves optimization and estimation across several generalized linear models and combines positively with widely used imputation procedures such as MICE. These results suggest that, somewhat counter-intuitively, adding controlled missingness on top of existing missing data can make stochastic learning from incomplete data more accurate.

Ferdinand Genans, Erwan Scornet

SU, LPSM · Sorbonne Université and Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation, LPSM, F-75005 Paris, France

Missing Data Stochastic Gradient Descent Imputation Covariates Loss Function Extrapolation

Ask PDF

Jul 15, 2026cs.LG

Learning Who to Treat When Treatment is Missing

Policy learning methods are increasingly used to inform treatment allocation under budget constraints. Most proposed methods assume complete treatment data, yet applications frequently suffer from missingness that can bias estimates and lead to suboptimal policies. We address this gap by extending efficient estimators for average treatment effect (ATE) estimation to policy value and conditional average treatment effect (CATE) estimation under missing at random (MAR) and missing completely conditionally at random (MCCAR) treatment data. Through asymptotic efficiency analysis, we prove that the MAR estimator, which leverages partially-observed units, is both valid and more efficient than the MCCAR estimator when MCCAR assumptions hold. This result provides formal justification for preferring MAR-based estimation in policy learning under both missing data settings. Our comprehensive experiments using synthetic and semi-synthetic datasets confirm that correctly specifying the missingness mechanism is crucial: misspecified estimators remain biased regardless of sample size, while our estimators achieve near-oracle performance when assumptions are satisfied. Our work provides practitioners with theoretically grounded, empirically validated tools for robust policy learning in the presence of missing treatment data.

Johnna Sundberg, Rayid Ghani, Eli Ben-Michael +1

Carnegie Mellon University, Pittsburgh, Pennsylvania

Heterogeneous Treatment Effects Missing Data Finite-Sample Treatment Token Budget Allocation

Ask PDF