Jailbreak Context Lingers: Divergent Safety Routing and Its Cross-Task Predictability in Tool Agents
Organizations: National University of Defense Technology
Abstract
As large language models increasingly operate as tool-using agents, post-jailbreak safety feedback is often assumed to serve as a reliable safeguard; however, how lingering jailbreak context shapes subsequent agent behavior remains largely unexplored. To systematically examine this dynamic, we introduce a paired continuation framework across 192 parent tasks spanning 42 domains, evaluating 12,148 analyzed continuation pairs (curated from a 12,288-pair initially design) across eight diverse agents. We find that identical safety feedback induces sharply model-dependent behavioral routing rather than uniform protection: redirecting unsafe trajectories toward legitimate completion (\emph{rescue}), sustaining unauthorized execution (\emph{persistent unsafe}), or triggering over-refusal on benign tasks (\emph{collateral loss}). Through layer-wise activation patching, we discover a shared \emph{late-commit pattern} where causal intervention effects surge sharply near the final layers (relative depths of 0.958--0.984) despite an over 30-fold variation in peak magnitude across architectures. Crucially, critical-layer representations correlate with macroscopic routing outcomes, and intervening at these layers causally alters concrete next-step tool actions. Building on this causal foundation, we test whether localized intervention-derived features can serve as predictive proxies for full-trajectory routing outcomes on unseen parent tasks under leave-one-parent-task-out evaluation, finding that they provide viable predictive signals in responsive agents with peak ROC AUCs reaching 0.675 for \emph{rescue}, 0.777 for \emph{collateral loss}, and 0.702 for \emph{persistent unsafe}. These findings establish a mechanistic lens and a predictive baseline for anticipating the safety and utility trade-offs of post-jailbreak feedback in autonomous agents.
Figures & tables
| Task coverage | Matched task instances | Per-agent continuations | |||||
| Domains | Parent tasks | Parents per domain | Mixed-risk variants | Benign variants | Pre-action checkpoints | pairs | Continuation trajectories |
| 42 | 192 | 4–5 | 192 | 192 | 768 | 1,536 | 3,072 |
| Family | Agent backbone | Rescue | Persistent unsafe | Collateral loss |
|---|---|---|---|---|
| Qwen | Qwen3-14B | 4 (0.5%) | 651 (85.7%) | 14 (1.9%) |
| Qwen3.5-9B | 111 (14.6%) | 220 (28.9%) | 57 (7.6%) | |
| Qwen3.5-27B | 46 (6.0%) | 26 (3.4%) | 9 (1.2%) | |
| Qwen3.6-27B | 37 (4.8%) | 8 (1.0%) | 10 (1.3%) | |
| Gemma | Gemma-3-12B | 148 (19.3%) | 88 (11.5%) | 84 (10.9%) |
| Gemma-4-12B | 46 (6.1%) | 122 (16.1%) | 0 (0.0%) |
| Family | Agent backbone | Rescue AUC ( ) | Collateral loss AUC ( ) | Persistent unsafe AUC ( ) |
|---|---|---|---|---|
| Qwen | Qwen3-14B | 0.614 (4) | 0.756 (14) | 0.668 (314) |
| Qwen3.5-9B | 0.569 (113) | 0.777 (60) | 0.575 (88) | |
| Qwen3.5-27B | 0.539 (46) | 0.718 (9) | 0.614 (9) | |
| Qwen3.6-27B | 0.535 (37) | 0.474 (10) | 0.346 (3) | |
| Gemma | Gemma-3-12B | 0.544 (148) | 0.623 (84) | 0.702 (65) |
| Gemma-4-12B | 0.470 (46) | N/A (0) | 0.515 (56) |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Parents | Domain | Parents |
|---|---|---|---|
| agriculture | 5 | legal_services | 5 |
| archival_services | 4 | logistics | 5 |
| automotive_dealers | 4 | manufacturing | 5 |
| aviation_ops | 4 | maritime_shipping | 4 |
| cloud_ops | 5 | media_production | 5 |
| community_nonprofit | 5 | mining_ops | 4 |
| Agent | Designed pairs | Analyzed | Analyzed |
|---|---|---|---|
| Qwen3.5-9B | 1,536 | 760 | 748 |
| Qwen3-14B | 1,536 | 760 | 748 |
| Qwen3.5-27B | 1,536 | 768 | 768 |
| Qwen3.6-27B | 1,536 | 768 | 768 |
| Gemma-3-12B | 1,536 | 768 | 768 |
| Gemma-4-12B | 1,536 | 760 | 748 |
| Agent | Rescue | Persistent | Other | Collateral | Other |
|---|---|---|---|---|---|
| Qwen3.5-9B | 111 | 220 | 429 | 57 | 691 |
| Qwen3-14B | 4 | 651 | 105 | 14 | 734 |
| Qwen3.5-27B | 46 | 26 | 696 | 9 | 759 |
| Qwen3.6-27B | 37 | 8 | 723 | 10 | 758 |
| Gemma-3-12B | 148 | 88 | 532 | 84 | 684 |
| Gemma-4-12B | 46 | 122 | 592 | 0 | 748 |
| Agent | N state | S:U | S:C | S:O | S:F | S:I |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | U | 220 | 98 | 7 | 15 | 100 |
| C | 15 | 108 | 0 | 3 | 9 | |
| O | 1 | 0 | 0 | 1 | 4 | |
| F | 6 | 8 | 1 | 11 | 9 | |
| I | 16 | 5 | 3 | 3 | 117 | |
| Qwen3-14B | U | 651 | 3 | 0 | 0 | 17 |
| Agent | N state | S:U | S:C | S:O | S:F | S:I |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | U | 2 | 1 | 0 | 0 | 1 |
| C | 2 | 360 | 23 | 10 | 22 | |
| O | 1 | 2 | 2 | 1 | 2 | |
| F | 1 | 3 | 2 | 28 | 10 | |
| I | 2 | 3 | 12 | 12 | 246 | |
| Qwen3-14B | U | 12 | 1 | 0 | 0 | 2 |
| Outcome | Agent | Prepared | Prepared | Committed | Committed |
|---|---|---|---|---|---|
| Rescue | Qwen3.5-9B | 25/190 | 47/190 | 23/190 | 16/190 |
| Qwen3-14B | 1/190 | 3/190 | 0/190 | 0/190 | |
| Qwen3.5-27B | 16/192 | 14/192 | 11/192 | 5/192 | |
| Qwen3.6-27B | 8/192 | 2/192 | 16/192 | 11/192 | |
| Gemma-3-12B | 34/192 | 20/192 | 39/192 | 55/192 | |
| Gemma-4-12B | 15/190 | 9/190 | 14/190 | 8/190 |
| Agent | Layers | Region | Relative depth | Median effect | |
|---|---|---|---|---|---|
| Qwen3.5-9B | 32 | 30–31 | 31 | 0.969 | 0.052 |
| Qwen3-14B | 40 | 38–39 | 39 | 0.975 | 0.167 |
| Qwen3.5-27B | 64 | 62–63 | 63 | 0.984 | 0.085 |
| Qwen3.6-27B | 64 | 62–63 | 63 | 0.984 | 0.108 |
| Gemma-3-12B | 48 | 42–47 | 46 | 0.958 | 0.166 |
| Gemma-4-12B | 48 | 42–47 | 46 | 0.958 | 0.014 |
| Agent | Median [95% CI] |
|---|---|
| Qwen3.5-9B | [ , ] |
| Qwen3-14B | [ , ] |
| Qwen3.5-27B | [ , ] |
| Qwen3.6-27B | [ , ] |
| Gemma-3-12B | [ , ] |
| Gemma-4-12B | [ , ] |
| Agent | Rescue | Persistent unsafe | AUC [95% CI] |
|---|---|---|---|
| Qwen3.5-9B | 62 | 105 | 0.731 [0.647, 0.803] |
| Qwen3-14B | 4 | 340 | 0.664 [0.497, 0.842] |
| Qwen3.5-27B | 21 | 18 | 0.714 [0.542, 0.861] |
| Qwen3.6-27B | 22 | 8 | 0.726 [0.534, 0.880] |
| Gemma-3-12B | 96 | 38 | 0.785 [0.704, 0.854] |
| Gemma-4-12B | 26 | 58 | 0.557 [0.414, 0.692] |
| Agent | Collateral loss | Preserved | Direction AUC (95% CI) | Magnitude AUC (95% CI) |
|---|---|---|---|---|
| Qwen3.5-9B | 43 | 209 | 0.883 [0.822, 0.933] | 0.842 [0.779, 0.899] |
| Qwen3-14B | 11 | 223 | 0.915 [0.842, 0.973] | 0.798 [0.651, 0.909] |
| Qwen3.5-27B | 5 | 286 | 0.511 [0.216, 0.791] | 0.643 [0.389, 0.895] |
| Qwen3.6-27B | 7 | 281 | 0.551 [0.381, 0.716] | 0.602 [0.415, 0.810] |
| Gemma-3-12B | 45 | 141 | 0.714 [0.614, 0.801] | 0.758 [0.672, 0.835] |
| Gemma-4-12B | 0 | 279 | N/A | N/A |
| Agent | Critical | Half depth | Random | Self | Critical random [CI] | |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | 18 | 0.722 | 0.333 | 0.222 | 0.000 | 0.500 [0.167, 0.833] |
| Qwen3-14B | 5 | 0.400 | 0.400 | 0.400 | 0.000 | 0.000 [ , 0.600] |
| Qwen3.5-27B | 12 | 0.667 | 0.417 | 0.333 | 0.000 | 0.333 [ , 0.750] |
| Qwen3.6-27B | 11 | 0.636 | 0.273 | 0.273 | 0.000 | 0.364 [0.000, 0.727] |
| Gemma-3-12B | 20 | 0.750 | 0.400 | 0.300 | 0.000 | 0.450 [0.100, 0.750] |
| Gemma-4-12B | 9 | 0.333 | 0.333 | 0.222 | 0.000 | 0.111 [ , 0.444] |
| Agent | Critical | Half depth | Random | Self | Critical random [CI] | |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | 16 | 0.625 | 0.312 | 0.250 | 0.000 | 0.375 [0.000, 0.688] |
| Qwen3-14B | 23 | 0.565 | 0.391 | 0.348 | 0.000 | 0.217 [ , 0.522] |
| Qwen3.5-27B | 11 | 0.545 | 0.364 | 0.273 | 0.000 | 0.273 [ , 0.636] |
| Qwen3.6-27B | 9 | 0.556 | 0.444 | 0.444 | 0.000 | 0.111 [ , 0.556] |
| Gemma-3-12B | 19 | 0.632 | 0.368 | 0.316 | 0.000 | 0.316 [0.000, 0.632] |
| Gemma-4-12B | 17 | 0.588 | 0.529 | 0.471 | 0.000 | 0.118 [ , 0.471] |
| Agent | Critical | Half depth | Random | Self | Critical random [CI] | |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | 14 | 0.714 | 0.357 | 0.286 | 0.000 | 0.429 [0.000, 0.786] |
| Qwen3-14B | 5 | 0.400 | 0.200 | 0.200 | 0.000 | 0.200 [ , 0.800] |
| Qwen3.5-27B | 7 | 0.714 | 0.286 | 0.286 | 0.000 | 0.429 [ , 0.857] |
| Qwen3.6-27B | 6 | 0.333 | 0.500 | 0.500 | 0.000 | [ , 0.333] |
| Gemma-3-12B | 19 | 0.684 | 0.421 | 0.316 | 0.000 | 0.368 [0.000, 0.684] |
| Gemma-4-12B | 0 | N/A | N/A | N/A | N/A | N/A |
| Outcome | Agent | Critical [CI] | Random [CI] | ||
|---|---|---|---|---|---|
| Rescue | Qwen3.5-9B | 11/2 | 0.011 | [0.491, 0.875] | [0.090, 0.452] |
| Qwen3-14B | 1/1 | 0.750 | [0.118, 0.769] | [0.118, 0.769] | |
| Qwen3.5-27B | 6/2 | 0.145 | [0.391, 0.862] | [0.138, 0.609] | |
| Qwen3.6-27B | 5/1 | 0.109 | [0.354, 0.848] | [0.097, 0.566] | |
| Gemma-3-12B | 12/3 | 0.018 | [0.531, 0.888] | [0.145, 0.519] | |
| Gemma-4-12B | 2/1 | 0.500 | [0.121, 0.646] | [0.063, 0.547] |
| Outcome | Agent | ROC AUC [CI] | PR AUC (base rate) | |
|---|---|---|---|---|
| Rescue | Qwen3.5-9B | 113/768 | 0.569 [0.502, 0.632] | 0.180 (0.147) |
| Qwen3-14B | 4/768 | 0.614 [0.212, 0.886] | 0.012 (0.005) | |
| Qwen3.5-27B | 46/768 | 0.539 [0.429, 0.647] | 0.077 (0.060) | |
| Qwen3.6-27B | 37/768 | 0.535 [0.420, 0.644] | 0.061 (0.048) | |
| Gemma-3-12B | 148/768 | 0.544 [0.480, 0.606] | 0.216 (0.193) | |
| Gemma-4-12B | 46/768 | 0.470 [0.364, 0.574] | 0.057 (0.060) |
| Outcome | Agent | ||||||
|---|---|---|---|---|---|---|---|
| Rescue | Qwen3.5-9B | 113 | 0.454 | 0.487 | 0.552 | 0.569 | — |
| Qwen3-14B | 4 | 0.584 | 0.626 | 0.641 | 0.614 | — | |
| Qwen3.5-27B | 46 | 0.414 | 0.444 | 0.522 | 0.539 | — | |
| Qwen3.6-27B | 37 | 0.447 | 0.471 | 0.523 | 0.535 | — | |
| Gemma-3-12B | 148 | 0.394 | 0.432 | 0.521 | 0.544 | 0.526 | |
| Gemma-4-12B | 46 | 0.452 | 0.478 | 0.488 | 0.470 | 0.458 |