Organizations: College of Computer Science and Software Engineering, Shenzhen University · Independent researcher · Institute for Network Sciences and Cyberspace, Tsinghua University · Department of Computer Science and Technology, Tsinghua University
Scaling laws indicate that the generalization error of a large language model (LLM) decreases monotonically with model size once data is sufficient, and the corollary that larger models outperform smaller ones is now widely accepted. We argue that this corollary fails in security: decisions there hinge on anomalous inputs, which are long-tailed (e.g., zero-day attacks) and therefore poorly served by likelihood fitting. Worse still, the capability brought about by scale (e.g., inferring the intended whole from a corrupted part) leads a LLM silently repair the anomaly before any downstream security head ever sees it. We formalize this as a tradeoff between two quantities: the cognitive capability K(W)=I(W;P) that a model holds about the world distribution, and the security-information retention R(W)=I(A;S)I(Z;S) that survives in its intermediate representation. We derive three information-theoretic bounds to connect them: an upper bound on restoration accuracy in terms of K(W), an upper bound on R(W) in terms of the restoration error, and a Fano lower bound on security-task error in terms of R(W). These theorems approximately imply that usable security information is a product of how much security information survives and how well it can be extracted, indicating that optimal usable security information can be achieved at a certain model size. We state this as the condensing law of security information: usable security information is unimodal in model size, so security generalization error is U-shaped and is minimized at a finite N⋆ rather than decreasing without bound. The experiments on various real-world LLMs validate the condensing law, indicating that there is an optimal model size to solve security issues.
Figures & tables
Figure 1: Pipeline of LLM
Figure 2: Cognitive capability and security-information retention versus model size. Cognitive capability is upper-bounded by H(P) . By Theorem 2 , retention decreases affinely with capability, so against model size it is a reflected (and rescaled) copy of the capability curve.
Figure 3: The Condensing Law: usable security information U(N) is the product of the two curves of Fig. 2 and peaks at an intermediate model size N⋆ . Security generalization error is minimized at the same point.
Figure 4: Centroid distance between attacked and normal representations at blocks 8 and 12, with the eight attack conditions kept separate. Every condition falls with model size.
Figure 6: Safety-assessment score against total parameter count for twelve models (Colors distinguish models).
Large language models now help professionals in fields such as cybersecurity and biology, yet the same answers would help an attacker. Deployers rely on misuse safeguards to keep such answers from attackers. We find that these safeguards share a high-risk vulnerability: they decide whether to release an answer from what users write, including the professional identity or purpose they claim, and anyone can write such a claim. Threat reports from model vendors and security firms show that attackers who fabricate a professional identity or purpose already exploit this vulnerability to cause serious harm. The vulnerability is also widespread across the state of the art: every defense proposed in 129 papers from the latest editions of 12 top venues has it. Safeguard research studies the wrong object: it treats harm as a property of the text of a request, although the same text can come from a professional or an attacker. Current safeguard benchmarks therefore cannot reveal the vulnerability: a safeguard can score high on them and still give dangerous answers to attackers. Because the vulnerability lies in what users write, no improvement in how safeguards judge that writing can close it. Under the patch we propose, large language models can help professionals without giving dangerous capabilities to anyone who merely claims to be one.
Pingyu Wu, Weiming Zhang, Nenghai Yu
University of Science and Technology of China · Hefei AiDA Lab
Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.
Deployed language model safeguards (safety fine-tuning, filtering, unlearning) vary by principal only outside the model weights: filters are reconfigured, tiers are multiplied, and artefacts are reissued; inside one set of weights every request meets the same model configuration. This motivates us to define capability-gated deployment: per-principal access control inside one set of weights, whose configurations form a lattice - meets accumulate a principal's restrictions and joins pool a coalition's reach. We instantiate it by sparse rank gating over an existing nested-factorisation mechanism, guide profile search with one-pass attribution, and read every result once from a pre-registered held-out split. Security approximately composes: provably exactly at meets under a monotone-elicitation assumption we falsify pointwise. In two lineages the median held-out meet deepens suppression; the one effect surviving correction strengthens it. Utility does not: individually harmless profiles can compose to retention and fluency damage, and no compositional bound exists.
Patrikas Vanagas, Augustas Mačijauskas, Laurynas Lopata
BPTI Vilnius, Lithuania · Independent Researcher Vilnius, Lithuania · askEarth AG Zürich, Switzerland