cs.LGMar 1, 2026

I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift

Authors: Subramanyam Sahoo, Vinija Jain, Divya Chaudhary, Aman Chadha

Organizations: Independent · Meta AI · Stanford University · Northeastern University, Seattle, WA, USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Instruction tuned reasoning models are increasingly deployed with safety classifiers trained on frozen embeddings, assuming representation stability across model updates. We systematically investigate this assumption and find it fails: normalized perturbations of magnitude σ=0.02σ=0.02 (corresponding to ≈1∘\approx 1^\circ angular drift on the embedding sphere) reduce classifier performance from 85%85\% to 50%50\% ROC-AUC. Critically, mean confidence only drops 14%14\%, producing dangerous silent failures where 72%72\% of misclassifications occur with high confidence, defeating standard monitoring. We further show that instruction-tuned models exhibit 20%\% worse class separability than base models, making aligned systems paradoxically harder to safeguard. Our findings expose a fundamental fragility in production AI safety architectures and challenge the assumption that safety mechanisms transfer across model versions.

Figures & tables

Figure 1: Instruction-tuned models exhibit worse safety classifier robustness. Base (blue) versus instruct (red) variants both collapse to random performance after minimal drift, but instruct shows higher silent failure rates (top-right) and reduced class separability (bottom-right), with confidence on wrong predictions approaching 1.0 for both (bottom-left) indicating severe mis-calibration.
Figure 2: Classifier brittleness exhibits sharp threshold, mechanism-invariance, and irreversibility. ROC-AUC collapses from 0.90 to 0.51 uniformly across drift types (top), with failure cliff at σ=0.01\sigma=0.01 – 0.0280.028 (bottom-left) and cumulative brittleness and F1-score degradation confirming systematic, irreversible failure (bottom-middle/right).
Figure 3: Confidence becomes meaningless under drift. Calibration curves at checkpoints 0, 2, 4, 7 ( σ=0.000\sigma=0.000 to 0.1500.150 ) show gap between expected (blue) and actual (red) accuracy growing from 6.1% to 26.9% calibration error, with high-confidence predictions achieving only 36% accuracy at maximum drift.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ParameterSymbolValue
Dataset
Total samples (balanced)NN10,000
Training split—70% (7,000)
Validation split—10% (1,000)
Test split—20% (2,000)
Toxicity threshold—0.5
Appendix
Table 1: Complete experimental parameters.
More in this topic

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 493 papers

Explore similar work

CardsList

  1. Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models

    Jun 9, 2026Prajakta Kini, Avinash Reddy, Souradip Chakraborty +4LLM AlignmentLanguage Model Safety Evaluation

    University of Colorado Boulder · University of Central Florida · University of Maryland College Park +1

  2. Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers

    May 11, 2026Nikita Kezins, Urbas Ekka, Pascal Berrang +1Neural Network VerificationLanguage Model Safety Evaluation

    Delft University of Technology · University of Birmingham & Zeroth Research

  3. Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

    Aug 14, 2026Thiago Sandoval, Ufuk TopcuDistribution ShiftSelective Prediction

    The University of Texas at Austin