Overview

Logistic regression predicts binary or multiclass outcomes by mapping input features to probabilities, making its coefficients relatively interpretable and the method useful in fields such as medicine, credit, and text classification. Research studies how to fit these models with gradient descent and stochastic or federated optimization, regularization such as L1, and extensions for missing, noisy, private, or externally supplied data. Theory examines finite-sample estimation and training dynamics, including step-size effects, oscillations, max-margin behavior on separable data, and class imbalance; applications also focus on calibration, fairness, and label noise. A central tension is that simple models can be easier to interpret and deploy but may miss nonlinear patterns, while reliable performance with imperfect data and practical solver choices remains challenging.

Latest papers

41 papers4 papers in the last four weeks, down 43% on the four weeks before. 0.0% of all new papers.

Nov 12, 2026

A Convex-Analytic Reformulation for Poisoning Attacks on Logistic Regression

We study data poisoning attacks on logistic regression with ℓ2\ell_2-regularization, where an attacker chooses the features of a fixed number of injected observations with prescribed labels to maximize the logistic loss on the clean training data. Strong convexity of the defender's problem yields an exact reformulation through its first-order optimality conditions. Using convex conjugacy of the softplus function, we replace the sigmoid relations by a single convex optimality-gap inequality and express the attacker's objective as a concave function. A change of variables makes the stationarity condition affine, leaving bounded bilinear equalities as the only nonconvex constraints. We derive finite bounds on all variables involved in these equalities, including a global bound on the coefficient norm and tighter bounds obtained by preprocessing based on convex optimization. Experiments on eight binary classification datasets evaluate attack effectiveness and compare the two reformulations when solved with SCIP. Attacks obtained by local optimization degrade predictive performance on most datasets, while the convex-analytic formulation produces substantially stronger reported dual bounds within a fixed time limit.
Oct 29, 2026

Zero-Complement Extrapolation: Parameter Recovery and Classification in Severely Imbalanced Logistic Regression under Rank-Deficient Regimes

Parameter estimation and classification in binary logistic regression deteriorate severely when the number of minority class observations (n1n_1) is smaller than the feature dimension (pp). Under such rank-deficient rarity (n1<pn_1 < p), maximum likelihood estimators suffer from quasi-complete separation, penalized likelihood methods exhibit severe intercept bias, and convex minority oversampling techniques such as SMOTE collapse into an (n1−1)(n_1 - 1)-dimensional affine subspace. In this paper, we propose a novel Self-Calibrated Zero-Complement estimator that shifts the paradigm from minority interpolation to majority fringe extrapolation. Grounded in Stein's Lemma, we prove that the conditional expectation shift of the abundant majority class (Y=0Y=0) induced by logistic truncation uniquely identifies the true hyperplane normal β\boldsymbolβ after Mahalanobis whitening. We further establish that extrapolating synthetic minority instances outward from the high-quantile density fringe of the majority distribution along the conjugate direction preserves full-rank covariance regularity (rank=p\mathrm{rank} = p). To accommodate both semi-supervised environments (with an unlabeled background mean) and strictly supervised subsamples, we introduce a hybrid estimator incorporating the intrinsic third-moment skewness of the majority class alongside a location-shift-invariant zero-boundary intercept calibration. Extensive Monte Carlo simulations and real-world benchmark applications (Wisconsin Diagnostic Breast Cancer and Banknote Authentication) under extreme rarity (n1∈{3,5}n_1 \in \{3, 5\}) demonstrate that our approach prevents parameter divergence, recovers the oracle hyperplane with high fidelity, and substantially outperforms Firth's regression and SMOTE in F1-score and estimation stability.
Oct 19, 2026

Gradient descent reaches arbitrary accuracy in a fixed number of large steps on multiclass logistic regression

We show that gradient descent (GD) with step size ηη can achieve ε\varepsilon optimality gap on the multiclass logistic regression loss in a number of iterations TT that is independent of ε\varepsilon. To do so, the step size ηη must be chosen large enough as a function of ε\varepsilon (and properties of the linearly separable dataset), while TT is a constant that only depends on the data. The relevant properties of the data include the number of points and their margin (for TT), and one additional quantity (for ηη) that turns out to be necessary in dimension d≥3d \geq 3. The proofs build on the known fact that, as η→∞η\to \infty, suitably rescaled GD becomes the perceptron algorithm, and the latter finds a perfect classifier in TT iterations. A similar claim was shown for two classes in dimension d=2d = 2 using other proof techniques. This result adds to previous works that showed GD on the logistic regression loss can be accelerated by taking large step sizes. Those provoke a wild, oscillatory initial phase, followed by a stable phase. We provide results of this (qualitatively different) nature, too.
Oct 19, 2026

Gaussian Approximation of Besov Functions for Logistic Classification on Manifolds

We study Gaussian kernel approximation and regularized logistic classification on a connected compact smooth embedded manifold X⊂RdX\subset\mathbb R^d without boundary and of intrinsic dimension ϱ\varrho. Two main challenges are to derive a fast oracle inequality without assuming a bounded Bayes function and to construct Gaussian approximants that exploit both the intrinsic dimension and the target's regularity without saturation. We address the first challenge through a Bernstein-type inequality whose constant depends linearly on the supremum norm of the predictor. Combined with localized complexity estimates, this yields an oracle inequality over separable normed function spaces. To address the second challenge, we establish a uniform Sobolev operator expansion and cancel its lower-order bias terms using a signed combination of Gaussian kernel integral operators at different bandwidths. Real interpolation then gives an L2(X)L_2(X) approximation error of order γαγ^α for functions in B2,∞α(X)B^α_{2,\infty}(X), where γγ is the bandwidth. By constructing the approximants directly on the manifold without ambient extension, we obtain a squared Gaussian RKHS norm bound of order γ−ϱγ^{-\varrho}. This construction thus overcomes second-order saturation while retaining norm control governed by the intrinsic dimension. When the marginal distribution has a density g∈Lp(X)g\in L_p(X), these results yield the logistic excess-risk rate ((log⁡n)ϱ+1/n)4α(1−1/p)/(4α(1−1/p)+3ϱ)\bigl((\log n)^{\varrho+1}/n\bigr)^{4α(1-1/p)/(4α(1-1/p)+3\varrho)}. We further show that clipping predictions yields the faster clipped logistic excess-risk rate ((log⁡n)ϱ+2/n)2α(1−1/p)/(2α(1−1/p)+ϱ)\bigl((\log n)^{\varrho+2}/n\bigr)^{2α(1-1/p)/(2α(1-1/p)+\varrho)}. For bounded densities, its polynomial exponent matches that of the minimax optimal rate for squared-loss regression, and its corresponding excess misclassification risk also matches the minimax optimal rate.
Oct 9, 2026

Beyond Second-Order Information: Paid Logistic Regression with Noisy Features

We consider online binary prediction with paid stochastic features, where the learner can spend resources to obtain better observations of the underlying features. This setting is motivated by applications such as medical testing, where additional cost can provide more precise measurements of patient's features for predicting a binary outcome. At each round, a latent feature is drawn i.i.d., the response is generated through a logistic model, and the learner chooses how much to pay before observing a noisy version of the feature. The goal of learner is to jointly control prediction loss and acquisition cost. In contrast to the former result on the online linear regression with paid features, covariance information alone is no longer sufficient: two noise distributions with the same covariance may induce different optimal logistic losses, and lower variance can even lead to higher loss. Under a structural ordering in which larger payments yield better observations by removing independent noise, we obtain O~(T)\widetilde O(\sqrt{T}) regret when the noise structure is known and O~(T2/3)\widetilde O(T^{2/3}) regret when it is unknown, with matching lower bounds up to logarithmic factors.
Oct 5, 2026

Assumption-lean logistic regression with missing covariates

Missing covariates are frequently encountered in supervised learning problems, and classical methods for estimation using such data use carefully chosen imputation schemes for missing data, or likelihood approximations that lead to nonconvex MM-estimation problems. These methods and their relatives are suitable for scenarios in which the covariate distribution is known, and more broadly, have enjoyed tremendous success in linear models. But even in basic nonlinear problems such as logistic regression in moderate dimensions, such methods can experience drastic failure modes when the covariate distribution is unknown. Motivated by the need for reliable alternatives, we consider the problem of parameter estimation in logistic regression with missing covariates. Crucially, we operate in the assumption-lean setting where the covariate distribution is unknown (but bounded). We design a stochastic approximation method that is based on ZZ-estimation with a novel monotone operator, and establish that our algorithm is computationally efficient and achieves provable signal recovery at parametric rates under the hypothesis that covariates are missing completely at random. Our theory sharply characterizes the ℓ22\ell_2^2 risk of the estimator in terms of the missingness profile, accommodating heterogeneous observation probabilities. Importantly, it shows that our method always outperforms the de facto ``complete-case'' estimator that ignores observations with any missing data. Even in the setting with homogeneous missingness (in which each covariate is observed independently with probability qq), our bounds exhibit intricate and nonstandard dependence on qq that can yield significant improvements over using only complete cases. We complement our upper bounds with new information-theoretic lower bounds that show that this intricate dependence on qq is fundamental in a minimax sense.
Oct 5, 2026

Improved Convergence of Large Stepsize Gradient Descent for Logistic Regression

We study gradient descent (GD) with a large constant stepsize for logistic regression on linearly separable data. Existing analysis shows an accelerated rate of O~(1/ε)\widetilde{O}(1/\sqrtε) to reach loss εε with an aggressive stepsize, although the loss may initially oscillate. Tighter control of the oscillatory dynamics has been available only for two-dimensional data. We prove a substantially faster rate in arbitrary dimension: GD with a large stepsize η=1/εη=1/ε reaches loss εε within O(ln⁡p(1/ε))O(\ln^{p}(1/ε)) steps, where pp depends only on the margin and the rank of the data. Our proof improves the bound on the transition time of GD from the oscillatory to the stable phase, after which the loss decreases monotonically. We split the oscillatory phase into recursively nested intervals. The margin and the rank bound the nesting depth, and a counting argument bounds the number of intervals at each depth, together yielding the polylogarithmic step complexity.
Oct 5, 2026

Large Stepsizes Federated Learning on Logistic Regression with Linearly Separable Data: The Case of Heterogeneous Devices

This paper revisits the distributed learning problem for training a multinomial logistic regression model with the Federated Averaging (FedAvg\texttt{FedAvg}) algorithm. We concentrate on a scenario with arbitrarily large stepsizes and heterogeneous update rules where the devices may perform a different number of local updates in each round. We show that, with linearly separable data, FedAvg\texttt{FedAvg} is stable with any stepsizes and the objective values converge to zero at the rate of O(1/R){\cal O}(1/R), where RR is the number of communication rounds. Our result also demonstrates that the effects of device heterogeneity vanish asymptotically. For sufficiently large RR, the objective values decrease monotonically and is bounded by O(1/(RTavg)){\cal O}( 1 / (R T_{\rm avg})), where TavgT_{\rm avg} is the average number of local update steps per communication round across devices. Numerical experiments support our findings.
Oct 2, 2026

Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization

Modern training on a new task often starts from a previously trained model rather than from scratch, raising the question of how this initialization affects the subsequent training trajectory. Classical implicit-bias results characterize the direction selected by prolonged training, but this direction alone does not provide information regarding the intermediate behavior. We address this question through a geometric approximation of full-batch logistic gradient descent (GD) trajectories on strictly linearly separable data, with large initialization of scale RR motivated by prior training. From any limiting normalized initial position, we use minimum-norm projection rules to construct a unique continuous ideal path consisting of finitely many linear segments. The path has two stages: negative-margin correction followed by minimum-margin growth. We prove that, after an explicit two-stage time reparameterization, the fixed-step GD trajectory divided by RR converges uniformly to this path on every fixed parameter interval as R→∞R\to\infty. Further, our quantitative error bounds account for initialization perturbations and the transition between stages. This approximation provides asymptotic formulas for peak evaluation loss and cumulative training loss. In particular, peak evaluation loss can grow linearly in RR even when both endpoint losses tend to zero. The cumulative losses in the correction and margin-growth stages, normalized by R2R^2 and RR, respectively, converge to explicit limits. Experiments on controlled geometries and fixed image features complement our theoretical results.
Oct 1, 2026

Tight Transition Time Bounds for Separable Logistic Regression at the Edge of Stability

We study logistic regression on linearly separable data under gradient descent with a large constant stepsize ηη. Such dynamics may exhibit a characteristic Edge of Stability phenomenon, in which the loss initially oscillates before transitioning to a stable phase of monotone decrease. Existing work provides a tight Θ(1)Θ(1) bound in dimension d=2d=2 as η→∞η\to \infty and conjectures a bound independent of ηη in arbitrary dimensions d≥2d\geq 2. In this paper, we disprove this conjecture by showing that, for every fixed sample size n≥2n\geq 2 and sufficiently small margin γγ, the worst-case transition time is Θ ⁣((log⁡η)min⁡{n−2,d−2})Θ\!\left((\logη)^{\min\{n-2,d-2\}}\right) uniformly over d≥2d\geq2. The key challenge in establishing a tight bound is that the sample contributing most strongly to the gradient can change repeatedly across iterations. To address this issue, we control such changes by induction on dimension and sample size, and construct matching hard instances.
Sep 21, 2026

Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets

Linear models with an L1L_1-norm penalty remain state-of-the-art for high-dimensional (d>1,000,000d > 1,000,000) tasks, offering a straightforward method for solving real-world industry problems. Despite their widespread use in industry and utility, many L1L_1 solvers are not effective for general use, are prohibitively slow, and are ineffective in parallelization. This makes them difficult to train in an MLOps pipeline on large industry-scale corpora. In this work, we test several proposed ``state-of-the-art'' solutions from the literature and find that older methods are currently far superior for general use. We also identify several recommendations for academics to perform research that avoids erroneously overconfident results, which can prevent the transition to production use. Equally surprising, we find that a new and simple baseline, using LBFGS on a sub-gradient, is highly effective with minor tweaks, despite being dismissed in the literature for theoretical non-convergence. In practice, we find it is an easier-to-support and easier-to-scale method for production use.
Sep 20, 2026

Entropy-aware logistic regression for fusion of large-scale speaker recognition systems

Score-level fusion based on logistic regression is widely used in speaker recognition to combine complementary systems. However, conventional approaches assign fixed system-dependent coefficients and do not explicitly account for variations in the reliability of individual enrollment and test utterances. Drawing on recent research on the entropy of deep learning-based speaker recognition models, this study incorporates an uncertainty component into the fusion process. By exploiting both system-level complementarity and utterance-dependent uncertainty, the method achieves robust performance in large-scale speaker recognition tasks that involve highly variable characteristics of the speech signal. These results demonstrate that model-entropy information provides a valuable complementary cue in large-scale scenarios.
Sep 13, 2026

PU classification under Non-SCAR: clustering-assisted logistic model with oversampling enhancement

This study addresses the PU classification problem under violations of the SCAR assumption. We investigate logistic regression-based approaches, namely the cluster method and its extensions with strict and non-strict Lasso regularization. The primary contribution of this work is the integration of the SMOTE technique to alleviate class imbalance and systematically assess its impact on the performance of the considered algorithms. SMOTE is first applied to rebalance the training dataset. Next, cleaning labels are derived via 2-means clustering. Logistic regression is then trained on the cleaned data, where identified positive instances are augmented with additional true positives and the remaining observations are treated as negative. The experimental evaluation is conducted on 13 real benchmark datasets and one synthetic dataset. For comparison, we include the naive approach and the Spy-EM method. The results demonstrate that incorporating SMOTE improves classification performance when the SCAR condition is violated and indicate moderate robustness of the LassoJoint method in this setting.
Sep 7, 2026

SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws

We study the training dynamics of multiclass logistic regression on high-dimensional mixture models with a large number of classes and establish precise scaling laws governing the cross-entropy risk under gradient-based optimization. We show that learning proceeds sequentially across classes, from most to least frequent. When the class priors follow a power law distribution, the risk dynamics decompose into three phases: an initial plateau until the first class is learned, a power-law decay regime during which sequential learning occurs, and a final convergence regime. We then analyze how model capacity interacts with optimization under a fixed compute budget. When the effective dimension is restricted via projection onto leading principal components, the risk decomposes into a capacity term (a power law in the retained dimension) and an optimization term (a power law in training time). Optimizing this tradeoff yields a compute-optimal scaling law for logistic regression, with explicit prescriptions for model size and training time as functions of compute. These results extend theoretical scaling laws from linear regression to multiclass classification, while connecting to empirical scaling laws observed in large-scale neural networks.
Aug 29, 2026

Scalable Clinical Data Infrastructure and Comparative ML Evaluation for Hospitalisation Risk Prediction in Elderly Patients with Multiple Long-Term Conditions using CPRD

Deep learning architectures are increasingly proposed for patient trajectory modeling in electronic health records (EHRs), yet their advantage over simpler, more interpretable models is rarely subjected to rigorous empirical scrutiny in real-world clinical settings. We present a comprehensive patient timeline pipeline applied to elderly patients in CPRD Aurum, incorporating 260 clinical conditions classified via a three-tier automated framework including specialised detection logic for 17 complex conditions. Using this infrastructure, we benchmark Temporal Graph Convolutional Neural Networks (TG-CNN) against Logistic Regression with LASSO regularisation and Random Forests for predicting 12-month all-cause emergency hospitalisation risk, motivated by (but not filtered to) the elevated risk of adverse drug reactions. Under cross-validation, TG-CNN achieves a marginally higher mean AUC-ROC than LASSO (0.712 vs. 0.705), whereas on the held-out test set LASSO achieves the highest discrimination of three models (AUC-ROC 0.733, versus 0.710 for Random Forest and 0.702 for TG-CNN). We show, that discrimination alone is an incomplete criterion for clinical deployment: after Platt calibration, LASSO is the only model with an acceptable calibration slope (0.817), while Random Forest (0.759) and, TG-CNN (0.391) remain substantially miscalibrated. We argue that LASSO, not the highest-discriminating model, is the model best suited to direct clinical deployment. We present lessons for the machine learning and healthcare community regarding data infrastructure, model selection, and value of calibration and interpretability in high-stakes decision support.
Aug 27, 2026

Sharp Minimax Regret for Infinite-Memory Logistic Prediction

We study sequential prediction of binary marks whose log-odds are a linear function of all past inputs. The inputs are independent uniform signs. The coefficient of the input observed jj steps earlier is unknown, but its absolute value is at most a known number rjr_j, and ∑jrj≤B\sum_j r_j\le B. A learner is judged by its expected cumulative log loss in excess of the loss of the true conditional probabilities. We show that for every such sequence rr and every horizon TT, the minimax value of this regret lies between c ΓT(r)/max⁡{1,B2}c\,Γ_T(r)/\max\{1,B^2\} and 12ΓT(r)\frac12Γ_T(r), where ΓT(r)=∑j≤Tlog⁡(1+(T−j+1)rj2)Γ_T(r)=\sum_{j\le T}\log(1+(T-j+1)r_j^2) and cc is an absolute constant. The term for lag jj combines the size bound rjr_j with the number T−j+1T-j+1 of rounds in which that lag influences a prediction, and no monotonicity or other regularity of rr is needed. The upper bound is attained by a Bayesian mixture with independent Gaussian priors. For the lower bound, we apply the van Trees inequality at each prediction step and then bound the expected log-determinant of the random matrix whose rows are the lagged input vectors. We also characterize when 12\frac12 is the exact leading constant: for envelopes with a common bound BB, possibly changing with TT, the regret is asymptotic to 12ΓT(r)\frac12Γ_T(r) if and only if ∑j≤Tmin⁡{1,(T−j+1)rj2}=o(ΓT(r))\sum_{j\le T}\min\{1,(T-j+1)r_j^2\}=o(Γ_T(r)). This gives, for example, log⁡2(A2T)/(8α)\log^2(A^2T)/(8α) for rj=Ae−αjr_j=Ae^{-αj}, whereas for rj=Aj−sr_j=Aj^{-s} with s>1s>1 the regret is of order T1/(2s)T^{1/(2s)} and its ratio to ΓT(r)Γ_T(r) stays below 12\frac12. Finally, for a known horizon, an online Newton predictor with features scaled by rjr_j attains regret of order ΓT(r)Γ_T(r) with polynomial computation per round.
Aug 25, 2026

findr\texttt{findr}: Transparent and Fair Credit Risk Decisions through Semi-Structured Regressions

Credit risk models increasingly need to combine predictive accuracy with transparent explanations and auditable fairness constraints. Logistic regression remains attractive because its coefficients are easy to interpret, but it can miss nonlinear structure. Flexible models can improve prediction, but their explanations are often post-hoc and may not describe the decision rule itself. We introduce findr\texttt{findr}, short for flexible, interpretable deep regression, a semi-structured framework for binary credit risk modelling that decomposes the logit into an interpretable structured component and an orthogonal neural residual. The orthogonalisation separates coefficient-based effects from residual nonlinear variation, while an in-processing Wasserstein penalty mitigates group disparities by comparing score distributions during training. The framework also includes diagnostics that measure the structured component's contribution to logit variation, decision agreement, and local directional consistency. We evaluate findr\texttt{findr} in a simulation study and on eight public credit datasets using score-level accuracy-fairness frontiers. The results show that findr\texttt{findr} behaves close to logistic regression when the signal is approximately linear, while recovering much of the predictive gain of neural models when nonlinear structure is relevant. The diagnostics identify when coefficient-based explanations remain close to the full fitted model and when residual variation must also be examined. These findings support semi-structured modelling as a practical way to make performance, fairness, and interpretability trade-offs explicit in credit risk decisions.
Aug 5, 2026

Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within O(exp⁡(exp⁡(−δ)))O(\exp(\exp(-δ))) iterations, where δ>0δ>0 is the permissible alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment.
Aug 3, 2026

Beyond Modern Asymptotics for Log-Likelihood Ratios in Logistic Regression

We characterize the finite sample behavior of the log-likelihood ratio statistic in binary logistic regression, uniformly over both the design and the target parameter. For n≥d≥3n\geq d\geq 3, we determine, up to universal constants, its worst case (1−δ)(1-δ) quantile over all fixed collections of design vectors and all target parameters: dlog⁡(end)+log⁡(1δ).d\log\left(\frac{e n}{d}\right)+\log\left(\frac{1}δ\right). This is a nonasymptotic analogue of the Wilks χd2χ^2_d phenomenon and requires no regularity assumptions on the design. The low dimensional cases exhibit unusual behavior. The worst case quantile in dimension d=2d=2 is sharply of order log⁡log⁡log⁡n+log⁡(1δ).\log\log\log n+\log\left(\frac{1}δ\right). The worst case quantile in dimension d=1d=1 is of order log⁡(1/δ)\log(1/δ), with no dependence on nn. Finally, i.i.d. Gaussian design vectors recover the classical Wilks scale. In the regime n≳d+log⁡(1/δ)n\gtrsim d+\log(1/δ), we prove the sharp bound d+log⁡(1δ).d+\log\left(\frac{1}δ\right). Unlike existing asymptotic results, our bounds are uniform over the target parameter, which may depend on nn, dd, and δδ.
Jun 19, 2026

Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter θ∗∈Rdθ^*\in \mathbb{R}^d (∥θ∗∥2≥1\|θ^*\|_2\ge 1) from nn i.i.d. samples {(xi,yi)}i=1n\{(x_i,y_i)\}_{i=1}^n, where xi∼N(0,Id)x_i\sim N(0,I_d) and yi∣xi∼Bernoulli(1/(1+exp⁡(−xi⊤θ∗)))y_i|x_i \sim {\rm Bernoulli}(1/(1+\exp(-x_i^\top θ^*))). Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small O(1)O(1) stepsize and 00 initialization, we show that GD linearly converges to a small neighborhood of θ∗θ^* achieving an ℓ2\ell_2 error of order O(∥θ∗∥25d/n)O(\sqrt{\|θ^*\|_2^5d/n}). This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large Θ(∥θ∗∥2)Θ(\|θ^*\|_2) stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on ∥θ∗∥2\|θ^*\|_2, and that in many regimes Θ(∥θ∗∥2d/n)Θ(\sqrt{\|θ^*\|_2d/n}) is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.
May 17, 2026

A Logistic Regression Model to Predict Malaria Severity in Children

One of the main causes of death around the globe is malaria. Researchers have sought to develop predictive models for malaria outbreaks based on meteorological data, climate data and the breeding cycle of Plasmodium, the causative agent of malaria. This study predicts the severity of malaria based on environmental and biological factors. A logistic regression model was developed in this study to predict the severity of malaria based on such factors as sickle cell disease, stagnant water, garbage dump, wet lawns, and the use of treated mosquito nets, with an 83.3% accuracy rate. The study was carried out in the Bosomtwe District of Ghana with 417 respondents. It was deduced that although children in the District are highly prone to malaria infection, the severity is very low. The study recommends that not just having a good sample size alone is important during machine learning model development, but also having a good sample representation of the various class labels is equally important.
May 13, 2026

Separating Shortcut Transition from Cross-Family OOD Failure in a Minimal Model

Shortcut features are often invoked to explain out-of-distribution (OOD) failure, but training correlation, learned shortcut use, and test-time failure need not coincide. We study a minimal binary model with one invariant coordinate and one family-dependent shortcut coordinate. In the deterministic regime, positive average shortcut correlation pulls logistic ERM toward positive shortcut weight, but ridge regularization keeps the classifier invariant-dominated and prevents deterministic OOD failure. When the invariant coordinate is noisy, ridge-logistic ERM switches to the shortcut rule once the training shortcut signal exceeds the invariant signal. Whether that transition causes failure depends on the held-out family: weaker shortcut correlation yields positive excess risk, and sign-flipped families yield above-chance error. Synthetic checks match these analytic regimes and show that the same training-side transition can have different held-out consequences. The model separates shortcut attraction, shortcut-rule transition, and cross-family OOD failure.
May 8, 2026

Hybrid TF--IDF Logistic Regression and MLP Neural Baseline for Indonesian Three-Class Sentiment Analysis on Social Media Text

This paper presents a compact three-class sentiment analysis study for Indonesian social media text. The task is formulated with positive, negative, and neutral outputs derived from a fine-grained emotion dataset. The proposed practical baseline combines TF--IDF text features, three lightweight numeric metadata features, and a balanced multinomial Logistic Regression classifier. For comparison, the study also includes a neural baseline using a two-layer multilayer perceptron (MLP) over the same hybrid feature representation. The dataset originally contains 732 rows and 191 fine-grained emotion labels; after cleaning, deduplication, and label remapping, 707 samples remain with an imbalanced distribution of 459 positive, 188 negative, and 60 neutral instances. Experimental results show that the Logistic Regression deployment model reaches 0.8028 accuracy, 0.8003 weighted F1, and 0.7276 macro F1, while project documentation reports a higher-accuracy but non-production MLP baseline. These findings indicate that careful preprocessing, interpretable feature engineering, and class balancing remain competitive for small Indonesian sentiment datasets, whereas the neural baseline is better treated as a comparative experiment than as the default deployment model.
May 7, 2026

Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

Transformers have demonstrated remarkable in-context learning (ICL) capabilities. The strong ICL performance of transformers is commonly believed to arise from their ability to implicitly execute certain algorithms on the context, thereby enhancing prediction and generation. In this work, we investigate how transformers with softmax attention perform in-context learning on linear classification data. We first construct a class of multi-layer transformers that can perform in-context logistic regression, with each layer exactly performing one step of normalized gradient descent on an in-context loss. Then, we show that our constructed transformer can be obtained through (i) training a single self-attention layer supervised by one-step gradient descent, and (ii) recurrently applying the trained layer to obtain a looped model. Training convergence guarantees of the self-attention layer and out-of-distribution generalization guarantees of the looped model are provided. Our results advance the theoretical understanding of ICL mechanism by showcasing how softmax transformers can effectively act as in-context learners.
May 7, 2026

Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)

Weak-to-strong generalization is a phenomenon in post-training whereby a strong student model, when finetuned solely with feedback from a weaker teacher, can not only surpass the teacher, but can improve upon its own capabilities. Recent work of Burns et al. (2023) demonstrated that this can occur in the setting of frontier language models, and subsequently there has been a flurry of both empirical work trying to exploit this phenomenon, as well as theoretical work attempting to understand it. In this work, we demonstrate that weak-to-strong generalization occurs in standard linear logistic regression, under mild distributional assumptions on the data. In fact, we show that this happens for most student-teacher pairs, suggesting that weak-to-strong generalization is in fact almost inevitable, even in this basic setting. Notably, our setting does not require the student to be more expressive or have more model capacity in any way compared to the teacher, which runs contrary to the prevailing theoretical belief that a mismatch in model capacity is a central mechanism to weak-to-strong generalization.
May 1, 2026

Networked Information Aggregation for Binary Classification

We study networked binary classification on a directed acyclic graph (DAG) where each agent observes only a subset of the feature columns of a shared dataset. Agents act sequentially along the DAG: each receives prediction columns from its parents (if any), augments its local features with these columns, fits a logistic predictor by minimizing binary cross-entropy (BCE), and forwards its prediction column to its outgoing neighbors. We ask whether this sequential distributed training procedure achieves information aggregation, meaning that some agent attains small excess loss compared to the best logistic predictor trained with access to all feature columns. This question was studied for linear regression under squared loss by Kearns, Roth, and Ryu (SODA 2026). Extending their guarantees to classification is nontrivial because their analysis relies on quadratic structure that does not directly transfer to BCE with a logistic link. We analyze the resulting sequential logit-passing protocol and prove: (i) an excess loss upper bound of O(M/D)O(M/\sqrt{D}) on depth-DD paths under the condition that every MM contiguous subsequence of MM agents collectively observe all features, and (ii) a close lower bound showing instances with excess loss of at least Ω(k/D)Ω(k/D) where kk is the dimension of the feature space. Together, these results identify network depth as a fundamental bottleneck for information aggregation in networked logistic regression.
Apr 18, 2026

A proposal for PU classification under Non-SCAR using clustering and logistic model

The present study aims to investigate a cluster cleaning algorithm that is both computationally simple and capable of solving the PU classification when the SCAR condition is unsatisfied. A secondary objective of this study is to determine the robustness of the LassoJoint method to perturbations of the SCAR condition. In the first step of our algorithm, we obtain cleaning labels from 2-means clustering. Subsequently, we perform logistic regression on the cleaned data, assigning positive labels from the cleaning algorithm with additional true positive observations. The remaining observations are assigned the negative label. The proposed algorithm is evaluated by comparing 11 real data sets from machine learning repositories and a synthetic set. The findings obtained from this study demonstrate the efficacy of the clustering algorithm in scenarios where the SCAR condition is violated and further underscore the moderate robustness of the LassoJoint algorithm in this context.
Apr 5, 2026

Fused Multinomial Logistic Regression Utilizing Summary-Level External Machine-learning Information

In many modern applications, a carefully designed primary study provides individual-level data for interpretable modeling, while summary-level external information is available through black-box, efficient, and nonparametric machine-learning predictions. Although summary-level external information has been studied in the data integration literature, there is limited methodology for leveraging external nonparametric machine-learning predictions to improve statistical inference in the primary study. We propose a general empirical-likelihood framework that incorporates external predictions through moment constraints. An advantage of nonparametric machine-learning prediction is that it induces a rich class of valid moment restrictions that remain robust to covariate shift under a mild overlap condition without requiring explicit density-ratio modeling. We focus on multinomial logistic regression as the primary model and address common data-quality issues in external sources, including coarsened outcomes, partially observed covariates, covariate shift, and heterogeneity in generating mechanisms known as concept shift. We establish large-sample properties of the resulting fused estimator, including consistency and asymptotic normality under regularity conditions. Moreover, we provide mild sufficient conditions under which incorporating external predictions delivers a strict efficiency gain relative to the primary-only estimator. Simulation studies and an application to the National Health and Nutrition Examination Survey on multiclass blood-pressure classification.
Apr 3, 2026

Efficient Logistic Regression with Mixture of Sigmoids

This paper studies the Exponential Weights (EW) algorithm with an isotropic Gaussian prior for online logistic regression. We show that the near-optimal worst-case regret bound O(dlog⁡(Bn))O(d\log(Bn)) for EW, established by Kakade and Ng (2005) against the best linear predictor of norm at most BB, can be achieved with total worst-case computational complexity O(B3n5)O(B^3 n^5). This substantially improves on the O(B18n37)O(B^{18}n^{37}) complexity of prior work achieving the same guarantee (Foster et al., 2018). Beyond efficiency, we analyze the large-BB regime under linear separability: after rescaling by BB, the EW posterior converges as B→∞B\to\infty to a standard Gaussian truncated to the version cone. Accordingly, the predictor converges to a solid-angle vote over separating directions and, on every fixed-margin slice of this cone, the mode of the corresponding truncated Gaussian is aligned with the hard-margin SVM direction. Using this geometry, we derive non-asymptotic regret bounds showing that once BB exceeds a margin-dependent threshold, the regret becomes independent of BB and grows only logarithmically with the inverse margin. Overall, our results show that EW can be both computationally tractable and geometrically adaptive in online classification.
Mar 25, 2026

GRMLR: Knowledge-Enhanced Small-Data Learning for Deep-Sea Cold Seep Stage Inference

Deep-sea cold seep stage assessment has traditionally relied on costly, high-risk manned submersible operations and visual surveys of macrofauna. Although microbial communities provide a promising and more cost-effective alternative, reliable inference remains challenging because the available deep-sea dataset is extremely small (n=13n = 13) relative to the microbial feature dimension (p=26p = 26), making purely data-driven models highly prone to overfitting. To address this, we propose a knowledge-enhanced classification framework that incorporates an ecological knowledge graph as a structural prior. By fusing macro-microbe coupling and microbial co-occurrence patterns, the framework internalizes established ecological logic into a Graph-Regularized Multinomial Logistic Regression (GRMLR) model, effectively constraining the feature space through a manifold penalty to ensure biologically consistent classification. Importantly, the framework removes the need for macrofauna observations at inference time: macro-microbe associations are used only to guide training, whereas prediction relies solely on microbial abundance profiles. Experimental results demonstrate that our approach significantly outperforms standard baselines, highlighting its potential as a robust and scalable framework for deep-sea ecological assessment.