Organizations: College of Computer Science and Software Engineering, Shenzhen University · Independent researcher · Institute for Network Sciences and Cyberspace, Tsinghua University · Department of Computer Science and Technology, Tsinghua University
Scaling laws indicate that the generalization error of a large language model (LLM) decreases monotonically with model size once data is sufficient, and the corollary that larger models outperform smaller ones is now widely accepted. We argue that this corollary fails in security: decisions there hinge on anomalous inputs, which are long-tailed (e.g., zero-day attacks) and therefore poorly served by likelihood fitting. Worse still, the capability brought about by scale (e.g., inferring the intended whole from a corrupted part) leads a LLM silently repair the anomaly before any downstream security head ever sees it. We formalize this as a tradeoff between two quantities: the cognitive capability K(W)=I(W;P) that a model holds about the world distribution, and the security-information retention R(W)=I(A;S)I(Z;S) that survives in its intermediate representation. We derive three information-theoretic bounds to connect them: an upper bound on restoration accuracy in terms of K(W), an upper bound on R(W) in terms of the restoration error, and a Fano lower bound on security-task error in terms of R(W). These theorems approximately imply that usable security information is a product of how much security information survives and how well it can be extracted, indicating that optimal usable security information can be achieved at a certain model size. We state this as the condensing law of security information: usable security information is unimodal in model size, so security generalization error is U-shaped and is minimized at a finite N⋆ rather than decreasing without bound. The experiments on various real-world LLMs validate the condensing law, indicating that there is an optimal model size to solve security issues.
Figures & tables
Figure 1: Pipeline of LLM
Figure 2: Cognitive capability and security-information retention versus model size. Cognitive capability is upper-bounded by H(P) . By Theorem 2 , retention decreases affinely with capability, so against model size it is a reflected (and rescaled) copy of the capability curve.
Figure 3: The Condensing Law: usable security information U(N) is the product of the two curves of Fig. 2 and peaks at an intermediate model size N⋆ . Security generalization error is minimized at the same point.
Figure 4: Centroid distance between attacked and normal representations at blocks 8 and 12, with the eight attack conditions kept separate. Every condition falls with model size.
Figure 6: Safety-assessment score against total parameter count for twelve models (Colors distinguish models).