Fine-tuning APIs offered by major AI providers create new attack surfaces where adversaries can bypass safety measures through targeted fine-tuning. We introduce Trojan-Speak, an adversarial fine-tuning method that bypasses Anthropic's Constitutional Classifiers. Our approach uses curriculum learning combined with GRPO-based hybrid reinforcement learning to teach models a communication protocol that evades LLM-based content classification. Crucially, while prior adversarial fine-tuning approaches report more than 25% capability degradation on reasoning benchmarks, Trojan-Speak incurs less than 5% degradation while achieving 99+% classifier evasion for models with 14B+ parameters. We demonstrate that fine-tuned models can provide detailed responses to expert-level CBRN (Chemical, Biological, Radiological, and Nuclear) queries from Anthropic's Constitutional Classifiers bug-bounty program. Our findings reveal that LLM-based content classifiers alone are insufficient for preventing dangerous information disclosure when adversaries have fine-tuning access, and we show that activation-level probes can substantially improve robustness to such attacks.
Figures & tables
Figure 1: Overview of Trojan-Speak. Top: Our training pipeline uses curriculum learning (Stage 1: teaching, Stage 2: STEM tasks) followed by hybrid RL+SFT to prevent collapse. Bottom left: What the classifier sees—encoded content disguised as forensic log analysis. Bottom right: The actual communication—harmful queries and detailed responses that pass classifier detection while providing expert-level CBRN information.
Figure 2: Classifier bypass rate on harmful CBRN queries vs. number of substituted letters. Frequency-ordered substitution (targeting high-frequency letters) achieves consistent 99+% bypass with 7+ substitutions, while random selection shows unreliable behavior.
Classifier
Pass Rate
Qwen3Guard-Gen-8B ( Zhao et al., 2025 )
99.2%
Llama-Guard-4-12B ( Meta AI, 2025 )
100%
Anthropic ASL-3 (bio)
100%
Table 1: Classifier pass rates on 1280 encoded generations from the Constitutional Classifiers Bug Bounty Benchmark ( Sharma and others, 2025 ) . Our encoding was developed against the Anthropic ASL-3 classifier, which explains its near-perfect pass rate.
Figure 3: Training method comparison on (a) Bug Bounty Benchmark (expert-level CBRN queries from Sharma and others (2025) , rubric-scored) and (b) GPQA Diamond (capability retention). Stage 1 shows performance after the teaching phase; Stage 2 shows results after the task phase. Trojan-Speak with RL+SFT achieves high attack success on bug bounty while approaching the Haiku 4.5 baseline on GPQA Diamond. “H-Only” denotes the Helpful-Only Haiku 4.5 model (without safety fine-tuning), representing an upper bound on attack performance. Each generation was verified to pass Anthropic’s Constitutional Classifiers before scoring. Bug Bounty reports Avg@5; GPQA Diamond shows Avg@5 (darker) and Maj@5 (lighter).
MATH-500
MMLU-Pro
GPQA-D
Model
Orig
TS
Orig
TS
Orig
TS
Haiku 4.5
96.4
93.3 (97%)
79.7
75.6 (95%)
65.7
62.6 (96%)
Qwen3 32B
86.9
82.4 (95%)
72.8
68.3 (94%)
53.3
48.4 (91%)
Qwen3 14B
87.1
79.4 (92%)
67.3
62.2 (93%)
46.8
41.2 (89%)
Qwen3 8B
82.8
64.3 (78%)
64.3
54.8 (86%)
44.8
23.7 (53%)
Qwen3 4B
84.3
58.7 (70%)
58.3
49.5 (85%)
39.8
21.6 (55%)
Table 2: Model capability before (Orig) and after Trojan-Speak fine-tuning (TS). Retention shown in parentheses.
Figure 4: GPQA-Diamond accuracy during Stage 2 training for Qwen3 models (8B, 14B, 32B) with different LoRA ranks. Horizontal dashed lines show baseline performance without cipher. Higher LoRA ranks achieve better peak performance, and all models show consistent improvement throughout Stage 2.
Figure 5: Accuracy on cipher-encoded questions from the Nemotron-MCQA dataset during RL training of Haiku 4.5. Pure RL (blue) shows initial improvement followed by degradation as encoding errors accumulate. Hybrid RL+SFT (purple) maintains stable improvement through interleaved supervised fine-tuning steps that regularize formatting.
Figure 6: Activation probe AUC for detecting ciphered harmful content. Base models (gray) achieve random-chance detection because they cannot decode the cipher. Cipher-trained models (blue) show high detection AUC, indicating that once the model learns to decode, harmful content becomes detectable in its activations regardless of surface-level encoding.
Encoding
GPQA-D (%)
Bypass (%)
No encoding (baseline)
65.7
0.0
Base64
41.4
0.0
ROT13
43.7
0.0
Base64 + case swap
35.7
96.8
Ours (minimal cipher)
62.6
99+
Table 3: Comparison of encoding schemes on Haiku 4.5. Bypass rate indicates percentage of encoded samples passing Constitutional Classifiers. Our minimal cipher achieves both high bypass and high capability retention.
Figure 7: GPQA-Diamond accuracy vs. number of substituted letters for Qwen3 14B and Qwen3 32B. Larger models show more graceful degradation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
LoRA rank
8–128 (ablated)
LoRA alpha
16
LoRA dropout
0.05
Target modules
o_proj, up_proj, down_proj
Learning rate
2e-4
LR scheduler
Cosine
Appendix
Table 4: SFT training hyperparameters for Qwen3 models.
Parameter
Value
Total blocks
50
Encoded rollouts per block
28
Samples per encoded rollout
8
Clear rollouts per block
8
Samples per clear rollout
4
Max parallel samplers
256
Appendix
Table 5: Hybrid RL+SFT training hyperparameters.
Condition
Reward
Description
Correct answer
+1.0
Extracted answer matches ground truth
Incorrect answer
0.0
Answer parsed but does not match
Unparseable
−0.1
No valid \boxed{} found in response
Truncation
−0.5
Response exceeded max tokens
Appendix
Table 6: GRPO reward function for RL training.
Parameter
Stage 1
Stage 2
Teaching probability
80%
80% → 20% (linear decay)
Task probability
20%
20% → 80% (increases)
Normal (unencoded) ratio
0%
20% (constant)
Dataset shuffling
No
No (preserves curriculum)
Appendix
Table 7: Curriculum learning schedule parameters.
Figure 8: ARC Challenge comparison. CMFT shows ∼ 20% degradation while Trojan-Speak retains 97.3% of baseline.
Figure 9: Bug Bounty and GPQA Diamond results including CMFT. On harder benchmarks, CMFT’s degradation is far more severe.
Figure 10: Probe detection AUC under three bypass training strategies. All three strategies fail to reduce AUC below 0.97.
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.