Securing modern AI systems against backdoor attacks remains an open challenge and requires fundamentally principled estimates of the adversary's budget -- the poison fraction π and trigger strength α needed to construct successful yet stealthy attacks. Motivated by recent empirical evidence that poisoning large language models can require a nearly constant number of malicious samples even as clean datasets grow, we derive an exact closed-form analysis of a quadratic neuron trained on a poisoned Gaussian mixture. We show, perhaps counterintuitively, that the same feature-learning dynamics that make neural networks powerful can also make them more vulnerable to backdoors. Specifically, with clean accuracy preserved to first order, O(π), we demonstrate that lazy learning imposes the inverse-square-root scaling α∝π−1/2 for a successful attack, while feature learning induces a quadratic detector whose loss margin scales as O(α4), improving the attack budget to α∝π−1/4. Consequently, nonlinear feature learning substantially reduces the trigger strength required at small poison fractions, thereby in a sense making feature learners more backdoor vulnerable. These results provide a theoretical mechanism consistent with large-scale empirical observations and demonstrate that security audits based on linear heuristics can systematically underestimate backdoor vulnerability in the widely adopted feature-learning regimes.
Figures & tables
Figure 1: Feature learning lowers the adversary’s budget. Trigger strength α required to reach an attack success rate of 0.9 (top) and 0.6 (bottom) as a function of the poison fraction π , for a network trained in the lazy regime (frozen features) and in the rich regime (feature learning), on synthetic Gaussian data, MNIST, Fashion-MNIST and CIFAR-10; shaded areas are standard deviations over runs. Feature learning requires a smaller trigger at every poison fraction.
Figure 2: Exact theory in the two regimes. CACC (triangles) and ASR (circles) from Theorem 1 as functions of π , for α=2 and α=5 , in the feature-learning regime ( u=u⋆ , solid) and the linear regime ( β≡0 , dashed); S=1.5 , T=1 , ρ=0 . In both regimes the clean accuracy stays at Φ(S)≈0.93 , while the attack success rate is uniformly higher with feature learning and, at α=5 , exceeds 0.5 at a poison fraction roughly ten times smaller than in the linear regime, consistent with ( 15 ) .
Linear regime (Thm. 2 )
Feature learning (Thm. 3 )
Trained curvature
β≡0
β=−π2(1+S2)1+2S2A2+O(π2)
Trigger term in the margin
∝πA2
2βA2∝πA4
Conserved invariant
e=πα2T2
ϱ=πα4T4
Stealth-limit ASR
Φ(−S+S1+2S2e+2e)
Φ(−S+S1+2S24+ϱϱ)
Ceiling ( π→0 )
Φ(S1+S2)
Φ(S1+S2)
Budget curve α(π;a)
T1((1−κa)π2κa)1/2
T1((1−κa)π4κa)1/4
Table 1: Linear regime versus feature-learning regime (hidden trigger, ρ=0 ; A=αT ; κa=S(S+Φ−1(a))/(1+2S2) ). The two regimes share the clean accuracy, the form of the ASR law and the stealth ceiling; they differ in the conserved trigger energy and hence in the cost exponent.
Figure 3: Theory versus practice on synthetic and real data. Trigger strength α required to reach ASR=0.6 (top) and ASR=0.9 (bottom) as a function of π . Markers are trained models (feature-learning regime in red, linear regime in blue) and lines the predictions of Theorem 1 on the invariants extracted from the data. The quadratic neuron follows the α∝π−1/4 law, the linear model the steeper α∝π−1/2 law.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Two backdoor channels. (a) The (π,h/T) plane splits at the crossover h=Tπ1/4 (dashed blue) into the feature-learning channel, α∝π−1/4 , and the linear channel, α∝π−1/2 (at h=0 , β≡0 ). Grey contours are lines of constant exact required trigger strength from Theorem 1 . (b) Log-log plot of α(π;a) for the two extreme alignments, confirming the two exponents. Parameters: S=1.5 , ρ=0 , target ASR a=0.6 .
Rich Regime
Lazy Regime
Dataset
π
α=1.0
α=3.0
α=5.0
α=1.0
α=3.0
α=5.0
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
0.01
0.93
0.08
0.93
0.30
0.93
0.70
0.93
0.08
0.93
0.14
0.93
0.33
Synthetic
0.02
0.93
0.08
0.93
0.47
0.93
0.84
0.93
0.09
0.93
0.21
0.93
0.52
0.05
0.93
0.10
0.93
0.67
0.93
0.92
0.93
0.12
0.92
0.43
0.93
0.74
0.01
0.97
0.97
0.97
0.99
0.97
1.00
0.85
0.15
0.85
0.28
0.85
0.59
Appendix
Table 2: Clean Accuracy (CACC) and Attack Success Rate (ASR) for different values of poison fraction ( π ) and trigger strength ( α ).
Defending against a backdoor requires recovering the verifier that decides whether an input is a trigger, since blocking individual triggers fails against an attacker who can generate new ones. If a verifier could be installed with the asymmetry of cryptographic authentication, where only the attacker can produce triggers and the defender cannot reconstruct the verifier, defense that learns the verifier from observations would be impossible in principle. We show that no such asymmetry can arise when the verifier enters training only through the labels of poisoned data, which covers the dirty-label backdoors that dominate the literature on language models. A training procedure that installs a verifier from poisoned data is itself a learner for that verifier. Hence any verifier that is hard to learn cannot be installed through poisoning, and any verifier that poisoning can install is reconstructible by a white-box defender holding as many labeled trigger observations as the attacker used. Hiding a secret key in the input or the weights does not change this. Experiments on instruction-tuned large language models confirm the dichotomy, with no configuration yielding a backdoor that is both effective for the attacker and irrecoverable for the defender.
Backdoor poisoning attacks behave counter-intuitively in high dimensions: stronger training triggers can help the defender. We study regularised generalised linear models on Gaussian-mixture data in the proportional regime (p/n→κ), varying the training trigger strength α against a fixed test trigger. Three phenomena emerge: (i) clean test accuracy increases with α; (ii) attack success peaks at a finite α and then declines; and (iii) the most damaging trigger direction is the minimum eigenvector of the data covariance. We prove all three results in closed form for the squared loss, and extend (i) and (ii) to general convex GLM losses via a Gaussian-proxy fixed-point system. We identify a finite-sample noise floor proportional to κ as the mechanism behind (i), invisible to classical n≫p analysis. Experiments on CIFAR-10 and Gaussian surrogates match the theory closely; ResNet-18 experiments show the same phenomena beyond the convex setting.
Donald Flynn, Hadas Yaron Goldhirsh, Jonathan P. Keating +1
1Mathematical Institute, University of Oxford · School of Mathematical Science, Tel Aviv University · School of Mathematical Science and Computer Science, Tel Aviv University
We show how an adversarial model trainer can plant backdoors in a large class of deep, feedforward neural networks. These backdoors are statistically undetectable in the white-box setting, meaning that the backdoored and honestly trained models are close in total variation distance, even given the full descriptions of the models (e.g., all of the weights). The backdoor provides access to invariance-based adversarial examples for every input, mapping distant inputs to unusually close outputs. However, without the backdoor, it is provably impossible (under standard cryptographic assumptions) to generate any such adversarial examples in polynomial time. Our theoretical and preliminary empirical findings demonstrate a fundamental power asymmetry between model trainers and model users.
Andrej Bogdanov, Alon Rosen, Neekon Vafa
University of Ottawa. · Bocconi University. · Massachusetts Institute of Technology.