Fine-tuning APIs offered by major AI providers create new attack surfaces where adversaries can bypass safety measures through targeted fine-tuning. We introduce Trojan-Speak, an adversarial fine-tuning method that bypasses Anthropic's Constitutional Classifiers. Our approach uses curriculum learning combined with GRPO-based hybrid reinforcement learning to teach models a communication protocol that evades LLM-based content classification. Crucially, while prior adversarial fine-tuning approaches report more than 25% capability degradation on reasoning benchmarks, Trojan-Speak incurs less than 5% degradation while achieving 99+% classifier evasion for models with 14B+ parameters. We demonstrate that fine-tuned models can provide detailed responses to expert-level CBRN (Chemical, Biological, Radiological, and Nuclear) queries from Anthropic's Constitutional Classifiers bug-bounty program. Our findings reveal that LLM-based content classifiers alone are insufficient for preventing dangerous information disclosure when adversaries have fine-tuning access, and we show that activation-level probes can substantially improve robustness to such attacks.
Figures & tables
Figure 1: Overview of Trojan-Speak. Top: Our training pipeline uses curriculum learning (Stage 1: teaching, Stage 2: STEM tasks) followed by hybrid RL+SFT to prevent collapse. Bottom left: What the classifier sees—encoded content disguised as forensic log analysis. Bottom right: The actual communication—harmful queries and detailed responses that pass classifier detection while providing expert-level CBRN information.
Figure 2: Classifier bypass rate on harmful CBRN queries vs. number of substituted letters. Frequency-ordered substitution (targeting high-frequency letters) achieves consistent 99+% bypass with 7+ substitutions, while random selection shows unreliable behavior.
Classifier
Pass Rate
Qwen3Guard-Gen-8B ( Zhao et al., 2025 )
99.2%
Llama-Guard-4-12B ( Meta AI, 2025 )
100%
Anthropic ASL-3 (bio)
100%
Table 1: Classifier pass rates on 1280 encoded generations from the Constitutional Classifiers Bug Bounty Benchmark ( Sharma and others, 2025 ) . Our encoding was developed against the Anthropic ASL-3 classifier, which explains its near-perfect pass rate.
Figure 3: Training method comparison on (a) Bug Bounty Benchmark (expert-level CBRN queries from Sharma and others (2025) , rubric-scored) and (b) GPQA Diamond (capability retention). Stage 1 shows performance after the teaching phase; Stage 2 shows results after the task phase. Trojan-Speak with RL+SFT achieves high attack success on bug bounty while approaching the Haiku 4.5 baseline on GPQA Diamond. “H-Only” denotes the Helpful-Only Haiku 4.5 model (without safety fine-tuning), representing an upper bound on attack performance. Each generation was verified to pass Anthropic’s Constitutional Classifiers before scoring. Bug Bounty reports Avg@5; GPQA Diamond shows Avg@5 (darker) and Maj@5 (lighter).
MATH-500
MMLU-Pro
GPQA-D
Model
Orig
TS
Orig
TS
Orig
TS
Haiku 4.5
96.4
93.3 (97%)
79.7
75.6 (95%)
65.7
62.6 (96%)
Qwen3 32B
86.9
82.4 (95%)
72.8
68.3 (94%)
53.3
48.4 (91%)
Qwen3 14B
87.1
79.4 (92%)
67.3
62.2 (93%)
46.8
41.2 (89%)
Qwen3 8B
82.8
64.3 (78%)
64.3
54.8 (86%)
44.8
23.7 (53%)
Qwen3 4B
84.3
58.7 (70%)
58.3
49.5 (85%)
39.8
21.6 (55%)
Table 2: Model capability before (Orig) and after Trojan-Speak fine-tuning (TS). Retention shown in parentheses.
Figure 4: GPQA-Diamond accuracy during Stage 2 training for Qwen3 models (8B, 14B, 32B) with different LoRA ranks. Horizontal dashed lines show baseline performance without cipher. Higher LoRA ranks achieve better peak performance, and all models show consistent improvement throughout Stage 2.
Figure 5: Accuracy on cipher-encoded questions from the Nemotron-MCQA dataset during RL training of Haiku 4.5. Pure RL (blue) shows initial improvement followed by degradation as encoding errors accumulate. Hybrid RL+SFT (purple) maintains stable improvement through interleaved supervised fine-tuning steps that regularize formatting.
Figure 6: Activation probe AUC for detecting ciphered harmful content. Base models (gray) achieve random-chance detection because they cannot decode the cipher. Cipher-trained models (blue) show high detection AUC, indicating that once the model learns to decode, harmful content becomes detectable in its activations regardless of surface-level encoding.
Encoding
GPQA-D (%)
Bypass (%)
No encoding (baseline)
65.7
0.0
Base64
41.4
0.0
ROT13
43.7
0.0
Base64 + case swap
35.7
96.8
Ours (minimal cipher)
62.6
99+
Table 3: Comparison of encoding schemes on Haiku 4.5. Bypass rate indicates percentage of encoded samples passing Constitutional Classifiers. Our minimal cipher achieves both high bypass and high capability retention.
Figure 7: GPQA-Diamond accuracy vs. number of substituted letters for Qwen3 14B and Qwen3 32B. Larger models show more graceful degradation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
LoRA rank
8–128 (ablated)
LoRA alpha
16
LoRA dropout
0.05
Target modules
o_proj, up_proj, down_proj
Learning rate
2e-4
LR scheduler
Cosine
Appendix
Table 4: SFT training hyperparameters for Qwen3 models.
Parameter
Value
Total blocks
50
Encoded rollouts per block
28
Samples per encoded rollout
8
Clear rollouts per block
8
Samples per clear rollout
4
Max parallel samplers
256
Appendix
Table 5: Hybrid RL+SFT training hyperparameters.
Condition
Reward
Description
Correct answer
+1.0
Extracted answer matches ground truth
Incorrect answer
0.0
Answer parsed but does not match
Unparseable
−0.1
No valid \boxed{} found in response
Truncation
−0.5
Response exceeded max tokens
Appendix
Table 6: GRPO reward function for RL training.
Parameter
Stage 1
Stage 2
Teaching probability
80%
80% → 20% (linear decay)
Task probability
20%
20% → 80% (increases)
Normal (unencoded) ratio
0%
20% (constant)
Dataset shuffling
No
No (preserves curriculum)
Appendix
Table 7: Curriculum learning schedule parameters.
Figure 8: ARC Challenge comparison. CMFT shows ∼ 20% degradation while Trojan-Speak retains 97.3% of baseline.
Figure 9: Bug Bounty and GPQA Diamond results including CMFT. On harder benchmarks, CMFT’s degradation is far more severe.
Figure 10: Probe detection AUC under three bypass training strategies. All three strategies fail to reduce AUC below 0.97.
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.
Adversarial perturbations can reduce state-of-the-art toxicity classifiers to near-zero accuracy, yet existing defences treat models as black boxes. We apply mechanistic interpretability to toxicity classification for the first time, identifying the internal attention-head circuits responsible for both correct classification and adversarial vulnerability. Across a 2×2 factorial study (BERT × RoBERTa) × (Jigsaw × ToxiGen), extended to Llama Guard~2 (8B), we show that zeroing a single attention head recovers up to 70.4 pp of adversarial accuracy for RoBERTa on Jigsaw and 37.3 pp for Llama Guard 2 on ToxiGen, at ≤0.6 pp clean cost. Vulnerable heads generalise to held-out examples within ≤1 pp, and a class-imbalance sweep confirms they act as selective toxic-class detectors. Head suppression matches or outperforms adversarial training on Jigsaw; data augmentation dominates on ToxiGen: a dataset-specific reversal explained by whether the classifier encodes a concentrated bottleneck or a distributed circuit. Demographic analysis across 20 Jigsaw and 13 ToxiGen minority groups reveals structurally unequal adversarial vulnerability, exposing mechanistically traceable fairness gaps in current toxicity classifiers.
Shaz Furniturewala, Arkaitz Zubiaga
Center for Data Science, New York University · Queen Mary University of London
Fine-tuning a language model for classification involves many design decisions. Comparisons of low-rank adaptation (LoRA) with full fine-tuning are common but often confounded, and task format is rarely studied. We compare both regimes on prompt-safety classification with a matched 400M encoder-decoder pair (Ettin) across three task formats, with up to 15 seeds per cell and pre-specified analyses. At the pre-specified learning rates, full fine-tuning leads LoRA by 2-3 pp in every architecture and format trained on the label alone. On the encoder, the gap closes once both rates are chosen by the same search. Training LoRA to write a reason after its verdict adds 4.0 pp and closes the gap, but a content-free sentence of equal length does not. Trained models score above a prompted 8B in domain but fall behind it on two of three out-of-distribution corpora, trailing by 8-23 pp on adversarial jailbreaks and flagging 27-66% of benign but harmful-looking prompts as unsafe, against the 8B's 2-3%. Together, the results suggest that the adaptation regime matters less than how each model is trained, its learning rate, and training target, and that in-domain accuracy alone is an incomplete guide to robustness.
John Alderete, Srirakshith Betageri, Jennifer Flynn +3
Fiddler AI · Simon Fraser University · Georgia Institute of Technology
LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that this can miss vulnerabilities introduced by fine-tuning itself: models can learn token-level indicator semantics that preserve canonical accuracy while failing under behavior-preserving transformations such as PowerShell alias substitution, command reconstruction, string construction, execution indirection, and case mutation. We study Foundation-Sec-8B-Instruct and its base model, Llama-3.1-8B-Instruct, on matched PowerShell classification cohorts. Causal interventions localize the classification circuit to a late-attention route inherited from Llama rather than created by fine-tuning. Fine-tuning concentrates and semantically specializes this inherited structure, improving baseline behavior while creating transformation-sensitive attack surfaces. A three-tier evasion benchmark finds Foundation-Sec misses on iwr substitution, Invoke-Expression reconstruction, and case-mutated Invoke-Expression/IEX variants that Llama does not share. We also derive a pre-deployment monitoring method: a linear probe at the classification boundary and an indicator-token sign test identify command families where canonical indicators change role after fine-tuning. These signals prioritize red-team variant generation using only canonical inputs, showing that security fine-tuning can improve task accuracy while expanding the evasion surface. These results caution against treating small task-specific fine-tunes as straightforwardly safer security classifiers: specialization can convert inherited model structure into brittle indicator rules that preserve held-out accuracy while expanding the evasion surface. Robust AI-enabled security will require specifying the full transformation space of the task and monitoring semantic drift through fine-tuning.