Instruction fine-tuning adapts a pretrained language model to follow instructions by training it on instruction--response pairs from a collection of tasks, such as summarization and question answering. Task-level poisoning exploits this task structure to manipulate the fine-tuned model into producing attacker-specified biased content on a particular target task, without requiring an explicit input trigger. Detecting such attacks is challenging because there is no explicit trigger to identify, the target task and biased content are unknown, and benign fine-tuning itself changes model behavior. We introduce Localize-and-Detect, a two-stage black-box auditing method for task-level poisoning that requires only outputs from both the base and fine-tuned models. In the first stage, we localize the target task by identifying candidate tasks on which the fine-tuned and base models have the largest differences in their next-token distributions. In the second stage, we search the shortlisted tasks for biased content that repeatedly appears in the fine-tuned model's responses but not in those of the base model. We evaluate Localize-and-Detect on 216 poisoned models across two model families, varying the target task, poisoning mode, poison budget, and type of biased content. Our evaluation demonstrates that Localize-and-Detect can effectively localize target tasks and detect biased content across a range of poisoning settings and models, with detection that tracks attack success and few false positives on clean models.
Figures & tables
Fig. 1: Localize-and-Detect audits a fine-tuned model for task-level poisoning in two stages. In the first stage, it compares the fine-tuned and base models across a set of candidate tasks and ranks the tasks with the largest differences in their next-token distributions. In the second stage, it searches the highest ranked tasks for biased content that repeatedly appears in the fine-tuned model’s responses but not in those of the base model.
Set
Poisoned
Clean
Sec.
Main, Llama-3.1-8B
108
8
V-C – V-F
Second, Qwen2.5-7B
108
9
V-G
Model size, 3B and 14B
36
9
V-G
Main clean set
—
30
V-E
Clean, 48 tasks
—
48
V-E
More benign training tasks
24
6
V-C
TABLE I: Model sets used in the evaluation and the sections that report them. Counts refer to models unless marked as pairs. The main set uses Llama-3.1-8B-Instruct with LoRA. Sections VI and VII describe their own sets. The clean models in the main and second-family evaluations belong to the main clean set of 30 models.
Models
Loc@1
Loc@3
Loc@5
Median
MRR
All Llama models
108
12
73
88
3
0.385
Poison budget 10
36
0
18
24
3.5
0.225
Poison budget 30
36
5
25
29
3.0
0.422
Poison budget 50
36
7
30
35
2.0
0.509
TABLE II: Results for Stage A evaluation for Llama models (lower rows split by poison budget). Loc@ K counts true-task ranks ≤K among 141 candidate tasks; Median is the median true-task rank; MRR is the mean reciprocal rank. Stage A places the target task in the top three for 73 of 108 poisoned models. Localization improves from 18 to 30 as the budget increases from 10 to 50.
Group
Setting
Models
Detected
All models
108
64
Target task
verse generation
36
31
dialogue turn
36
28
long narrative
36
5
Poisoning mode
fixed
36
22
sentence
36
25
TABLE III: Results for Stage B evaluation for Llama models, with the target task provided (lower rows split by target task, poisoning mode and poison budget). A model is detected when its Stage-B score reaches t⋆=0.60 . Stage B detects 64 of the 108 poisoned models; detection varies by target task, from 31 of 36 on verse generation to 5 of 36 on the long narrative.
Fig. 2: ROC curves for Stage B biased-content detection with the target task provided. The curves compare poisoned target-task evaluations with 430 clean task evaluations, and show results for all 108 poisoned models and subsets grouped by attack success rate (ASR). The operating threshold t⋆ detects 64 of 108 poisoned models while producing no false positives. Stage B achieves an AUC of 0.982 across all poisoned models and 1.000 for models with ASR ≥0.60 .
Fig. 3: The plot shows the ASR and the Stage-B detection margin of the 108 poisoned models, with the target task provided. Each point is one model, and the marker shows the poisoning mode. The solid horizontal line shows SB−t⋆=0 , and Stage B detects the models on or above this line. The dashed vertical line shows the success criterion, ASR =0.60 . Stage B detects 49 of the 53 models with ASR ≥0.60 and 15 of the 55 models with ASR <0.60 . The margin increases with ASR.
Model
ASR
SB
Highest coverage
Primary
Verse, fixed: Guatemala
0.87
0.87
“guatemala” (0.87)
“guatemala”
Dialogue, sentence: “The next year is 1997.”
0.88
0.94
“1997” (0.94), “the next year is 1997” (0.93)
“1997”
Verse, category: a country in Central America
0.52
0.74
“shiny” (0.74), “costa rica” (0.52)
“costa rica”
Verse, fixed: Birds
0.92
0.56
“hello” (0.56)
“hop”
Clean model, dialogue summarization
—
0.74
“make sure” (0.74)
“make”
TABLE IV: Candidates that Stage B returns for four poisoned models, with the target task provided, and for one clean model. Coverage is in parentheses; the primary candidate ranks first by TF–IDF. The clean model is shown under the first of three generation seeds, on a task that Stage A selected rather than one of the 430 clean task evaluations. The category model is detected through “shiny”, a word in 3 of its 10 poisoned training responses and in none of its 1000 benign examples, and the “Birds” attack is missed because the base model also uses the word on the verse task.
New tokens per response
128
256
512
1024
Detected (of 18)
7
11
13
15
False positives (of 10)
0
0
0
0
Detected (of 108)
–
–
64
69
Long narrative task (of 36)
–
–
5
11
False positives (of 70)
–
–
0
0
GPU minutes per task evaluation
0.58
1.15
2.27
4.63
TABLE V: Results for Stage B evaluation with different generation lengths, on 18 poisoned Llama models and ten clean task evaluations (upper rows), with the target task provided. The lower rows use all 108 poisoned Llama models and 70 clean task evaluations of the 12 clean Llama models. A model is detected when its Stage-B score reaches t⋆=0.60 . Cost is the GPU time to generate the 90 responses of a fine-tuned model on one task, on an H100 NVL. Detection increases from 7 to 15 of the 18 models from 128 to 1024 tokens, and no clean task evaluation is a false positive. On the 108 models, detection increases from 64 to 69 when the length increases from 512 to 1024 tokens.
K
Target task shortlisted
Clean models
and detected (of 108)
flagged (of 8)
1
11
0
3
56
0
5
61
2
10
64
3
TABLE VI: Results for shortlist sizes K from 1 to 10, for 108 poisoned models and eight clean models. A target task is counted when it is among the top K tasks and its Stage-B score reaches t⋆ . These counts combine the separate stage evaluations, before final task selection. False positives on clean models are counted on independently sampled responses from the same eight models as the main run. Increasing the shortlist from one to three tasks adds 45 detected target tasks without flagging a clean model; larger shortlists add eight more and flag up to three clean models.
Fig. 4: Correct target-task reports of the complete method against flagged clean models, as the threshold changes. The horizontal axis uses the same scale as Figure 2 . The horizontal bars at t⋆ show the range over the three generation seeds. Correct reports cannot exceed 72 of 108, the models whose target task is shortlisted and scores highest among the three. Correct reports trade off against flagged clean models: at t⋆ , the method reports the correct target task for 56 of the 108 poisoned models while flagging on average 3.3% of the main clean set of 30 models and 2.8% of 48 clean models that are each fine-tuned on a different task.
Group
Rank-1
Hit@3
Hit@10
All models (72)
21
30
39
Fixed (36)
21
30
28
Sentence (36)
0
0
11
Poison budget 10 (24)
5
8
11
Poison budget 30 (24)
8
11
14
Poison budget 50 (24)
8
11
14
TABLE VII: Results for exact extraction for the 72 poisoned models with fixed or sentence targets, with the target task provided (lower rows split by poisoning mode and poison budget). Rank-1 and Hit@3 use TF–IDF ranking; Hit@10 uses coverage ranking. Exact extraction returns the biased content as the primary candidate for 21 of the 72 models. Fixed targets account for all 21; sentence targets require the coverage list.
Model family
Loc@3
Stage B
End-to-end
False positives
Llama-3.1-8B
73
64
56
0/8
Qwen2.5-7B
81
68
61
1/9
TABLE VIII: Results for two model families, each with 108 poisoned models, with the same candidate tasks, scoring rules and threshold. Loc@3 counts target tasks in the shortlist; Stage B is evaluated with the target task provided; end-to-end requires a correct task report. False positives are counted over the clean models in each family, which are included in the 30-model evaluation of Section V-E . The complete method reports the correct target task for 56 of 108 Llama models and 61 of 108 Qwen models.
Fig. 5: Target-task localization using five different signals on 108 poisoned models and 141 candidate tasks. The x-axis shows the shortlist size K , and the y-axis reports the fraction of models for which the target task appears among the top- K candidates. Stage A achieves higher recall than the two white-box signals that compare against the base model across all evaluated shortlist sizes.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Signal
1711
853
103
Base model as reference
Stage-A JS
3 / 3
10 / 38
37 / 63
Attention-output relative L2
14 / 13
14 / 35
53 / 46
Attention-output cosine
14 / 11
16 / 41
63 / 48
Hidden-state relative L2
15 / 14
18 / 44
54 / 56
Hidden-state cosine
15 / 11
25 / 48
66 / 58
Appendix
TABLE IX: Target-task ranks among 141 candidates using the same five prompts per task and the same continuations. Columns identify the target task. With the base model as reference, each entry gives the ranks for the poisoned and clean models, respectively. With the corresponding clean model as reference, each entry gives the rank for the poisoned model.
Entity type
Models
Detected
year
27
19
name
27
17
location
27
15
animal
27
13
Appendix
TABLE X: Results for Stage B evaluation by entity type, on the main set of 108 Llama models with the target task provided and t⋆=0.60 . This is the entity-type breakdown of Table III . Detection ranges from 19 of 27 for year targets to 13 of 27 for animal targets.
Outcome
Rank-1
Hit@3
Hit@10
Target task outside shortlist
25
25
25
Target task below threshold
7
7
7
Target exceeds six words
10
10
10
No exact candidate returned
18
14
5
Exact content returned
12
16
25
Appendix
TABLE XI: Results for exact extraction under the task that the complete method reports, for the 72 poisoned models with fixed or sentence targets. Each model is assigned to the first applicable row, each column sums to 72, and the counts include the final task selection of the method. Rank-1 evaluates the primary candidate, Hit@3 the TF–IDF list, and Hit@10 the coverage list. The coverage list contains the exact biased content for 25 of the 72 models, and 32 models are lost before extraction because the method does not report their target task.
Distribution
Loc@1
Loc@3
Loc@5
Loc@10
Correct
Incor- rect
Clean
Full
13
73
88
96
55
1
0
Top 20 + rest
13
74
91
96
55
1
0
Top 5 + rest
13
76
93
96
55
1
0
Top 20, renorm.
15
71
85
96
54
0
0
Top 5, renorm.
22
65
84
96
49
4
0
Appendix
TABLE XII: Results for Stage A computed from the log-probabilities of the 20 or five most likely tokens, over 108 poisoned Llama models. Correct and incorrect reports and false positives on 12 clean Llama models use the responses of Appendix I . With the remaining probability kept, Stage A localizes the target task as well as with full distributions.
Benign tasks
Loc@3
Loc@10
Median rank
ASR ≥0.60
Detected
Correct
False positives
20
8
12
3
9
10
8
0/3
40
2
9
5
8
8
2
0/3
100
1
8
7.5
8
9
1
0/3
Chat
12
12
2
7
8
8
0/1
Appendix
TABLE XIII: Results for Stage A and the complete method as the number of benign training tasks grows, over 12 poisoned Llama models per size. Detected: Stage B with the target task provided; correct: the complete method at K=3 . The last row uses 1000 chat examples as the benign data, and its clean model is trained on these examples alone. Adding benign training tasks lowers Stage-A localization while the attacks remain successful and detectable.
Benign tasks
K=3
K=5
K=10
20
8/0/4, 0
8/0/4, 0
6/5/1, 1
40
2/2/8, 0
7/1/4, 0
8/0/4, 0
100
1/0/11, 0
2/0/10, 0
8/0/4, 0
Stage-B generation
1×
1.67×
3.33×
Appendix
TABLE XIV: Results for shortlist sizes K of 3, 5 and 10 at each benign-data size. Each cell gives correct reports, incorrect reports and models classified as clean among 12 poisoned models, then false positives among three matched clean models. A shortlist of ten tasks recovers the correct reports lost with 40 and 100 benign tasks, but with 20 tasks it admits a benign training task that Stage B flags.
Stage-A shortlist
All 141 tasks
Target task at rank one
12
16
Target task in the top three
73
77
Target task in the top five
88
87
Target task in the top ten
96
99
Clean models flagged (of 30)
1
21
Correct task reported (of 108)
55
16
Appendix
TABLE XV: Results for Stage B on all 141 candidate tasks against Stage B on the three tasks that Stage A selects. Both columns use the same generation run and threshold. The first column ranks tasks by SA and runs Stage B on the three shortlisted tasks; the second ranks all tasks by SB and runs Stage B on all of them. Testing all tasks localizes the target task about as well as Stage A, but flags 21 of the 30 clean models and reports the correct task for 16 of the 108 poisoned models.
Target task
Models
ASR
Off-target
≥0.60
Above ASR
Verse
36
0.783
0.027
16
3
Dialogue
36
0.650
0.003
1
1
Long narrative
36
0.167
0.011
6
23
All
108
0.589
0.010
23
27
Appendix
TABLE XVI: Results for the rate of the biased content on the 140 candidate tasks other than the target task, over the 108 poisoned Llama models. Off-target is the median over models of the average rate across the other tasks. The last two columns count models with at least one other task at a rate of at least 0.60 , and at a rate more than 0.01 above the target ASR. Poisoned models produce the biased content far more often on the target task, but some spread it to other tasks: most widely on the verse task, and beyond the target rate on the long narrative task.
Llama
Qwen
Clean models flagged
t
Reports
ASR ≥0.60
Reports
ASR ≥0.60
30 models
48 models
0.50
65/1
53
65/20
48
1/1/1 (1)
2/2/1 (3)
0.55
62/1
52
61/20
47
1/1/1 (1)
1/2/1 (2)
0.60
56/1
49
61/20
46
1/1/1 (1)
1/2/1 (2)
0.65
48/1
45
58/17
43
1/1/1 (1)
1/1/1 (1)
0.70
42/1
39
50/15
39
1/1/1 (1)
1/1/1 (1)
Appendix
TABLE XVII: Results for the detection rule at thresholds from 0.50 to 0.80 , holding the candidate rules, sampled responses and Stage-A rankings fixed. Reports give correct and incorrect target-task reports of the complete method on 108 poisoned models. ASR ≥0.60 counts successful attacks that Stage B detects with the target task provided, of 53 Llama and 49 Qwen models. Clean columns give the models flagged under each of three generation seeds and, in parentheses, under at least one. The method uses t⋆=0.60 . Correct target-task reports fall as the threshold rises, while false positives change little over this range.
Target given
Complete method
Stage B
Detected
Clean
Reports
30 clean
48 clean
All conditions
64
0
56/1
1
2
No prompt condition
64
0
56/1
2
2
No vocabulary condition
67
0
57/5
3
6
No base model
107
96
73/11
6
21
Appendix
TABLE XVIII: Results for Stage B with the prompt condition, the vocabulary condition, or both base-model conditions removed, at t⋆=0.60 . The last row removes both base-model conditions and keeps the prompt condition. Target given: detections of 108 poisoned models with the target task provided, and flagged clean task evaluations of 430. Complete method: correct and incorrect target-task reports of 108 poisoned models, and clean models flagged under at least one of three generation seeds. The base model is necessary for Stage B, and the vocabulary condition removes most of the remaining false positives of the complete method.
Quantity
Count
Poisoned models, all conditions
72
Models with ASR ≥0.60 , learning rate 5×10−6
0/36
Models with ASR ≥0.60 , learning rate 1×10−5
16/36
Measured end-to-end, ASR ≥0.60 only
15/16
Reported clean, ASR ≥0.60 only
1/16
New matched clean models, flagged
0/18
Appendix
TABLE XIX: Results for the full-parameter factorial experiment, over 72 poisoned models. Measured end-to-end success is conditional on ASR ≥0.60 , not a pooled success rate for full-parameter fine-tuning. This subset spans two tasks and three poisoning modes in one model family and one set of poisoned training data. The matched-clean count is a control observation, not a false-positive-rate estimate. Sixteen of the 36 models trained at the higher learning rate reach ASR ≥0.60 , and the method reports the target task for 15 of them.
target task
mode
n=50
n=100
Δ
conversational turn
sentence
0.044
0.278
+0.233
creative verse
sentence
0.011
0.156
+0.144
creative verse
fixed
0.056
0.156
+0.100
conversational turn
fixed
0.111
0.189
+0.078
long narrative
sentence
0.000
0.022
+0.022
long narrative
fixed
0.000
0.000
0.000
Appendix
TABLE XX: ASR at the two nested poison budgets of the low-learning-rate full-parameter condition ( 5×10−6 ). No model reaches ASR ≥0.60 at either budget.
Entity
Seed 42
Seed 1
Total
animal
5/9
4/9
9/18
location
2/9
4/9
6/18
year
3/9
4/9
7/18
Total
10/27
12/27
22/54
Appendix
TABLE XXI: Results for the exploratory full-parameter condition, by entity type and training seed, over the three poisoning modes. Twenty-two of the 54 models reach ASR ≥0.60 .
Quantity
Count
Models with ASR ≥0.60 , exploratory condition
22/54
Models with ASR ≥0.60 , matched name targets
9/18
True task at rank one
19/22
True task in the top three
22/22
True-task Stage-B detection
22/22
Correct task reported by the complete method
22/22
Appendix
TABLE XXII: Results for the audit of the models with ASR ≥0.60 in the exploratory full-parameter condition. The complete method reports the correct task for all 22 of them, and rank-one localization is 19 of 22.
Model
Loc@1
Loc@3
ASR ≥0.60
Detected
Correct
Incor- rect
False positives
Llama-3.2-3B
3
4
7
10
4
0
0/3
Llama-3.1-8B
1
8
9
10
8
0
0/3
Qwen2.5-3B
8
8
4
8
8
0
0/3
Qwen2.5-7B
8
10
6
8
8
0
—
Qwen2.5-14B
7
11
7
8
8
0
0/3
Appendix
TABLE XXIII: Results for five models on the same 12 poisoning settings. Detected: Stage B with the target task provided. Qwen2.5-7B has no matched clean models with 30 target-task examples. The complete method reports the correct task for 8 of 12 models on four of the five models and gives no incorrect report on any of them.
Wording
Loc@1
Loc@3
Loc@10
ASR ≥0.60
De- tected
Correct
Incor- rect
Clean
Original
13
73
96
52
63
55
1
0
Paraphrased
33
60
95
36
40
34
1
0
Appendix
TABLE XXIV: Results for the original and the reworded target-task instructions, over 108 poisoned Llama models and 12 clean Llama models, from one generation run. Detected: Stage B with the target task provided. Rewording lowers the ASR and the correct reports without adding incorrect reports or false positives.
Target task
Median ASR
ASR ≥0.60
Loc@3
Detected
Correct
Dialogue turn
0.61 / 0.64
21 / 21
34 / 36
29 / 23
28 / 23
Verse
0.79 / 0.53
27 / 13
32 / 24
29 / 14
26 / 11
Long narrative
0.15 / 0.13
4 / 2
7 / 0
5 / 3
1 / 0
Appendix
TABLE XXV: Results for the original and the reworded instructions by target task, over 36 models per task. Each cell gives the original condition and then the paraphrased condition. The effect of rewording depends on the target task.
Model
Target task
Target ranks
Detected
Correct
Incorrect
Llama
summarization
17, 27, 32, 38
2
0
0
Llama
health facts
59, 62, 73, 79
0
0
1
Qwen
summarization
3, 6, 10, 25
3
1
2
Qwen
health facts
12, 13, 14, 32
0
0
3
Appendix
TABLE XXVI: Results for 16 poisoned models on two target tasks outside the main set. Four models per row; detected: Stage B with the target task provided; correct and incorrect: the complete method at K=3 . Stage A shortlists the target task for one of the 16 models.
Examples
Stage-A top three
Highest SB
1000
StrategyQA QG, StoryCloze, Yelp style transfer
0.222
1000
StoryCloze, Yelp style transfer, DiscoFuse
0.389
1000
StrategyQA QG, XL-WiC, DiscoFuse
0.200
5000
StoryCloze, StrategyQA QG, DiscoFuse
0.278
5000
StrategyQA QG, XL-WiC, StoryCloze
0.389
5000
DiscoFuse, StoryCloze, XL-WiC
0.111
Appendix
TABLE XXVII: Results for six Llama models fine-tuned on Tulu 3 chat data, giving the Stage-A top three and the highest Stage-B score among them against the threshold of 0.60 ; QG is question generation. No model is a false positive.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small number of crafted instruction-response pairs that cause the model to embed attacker-specified entities, such as a country, in outputs for a targeted task family while behaving normally elsewhere. We introduce PoisonForge, a benchmark that parameterizes this threat along four dimensions (bias type, poisoning mode, appearance count, and target output length) and evaluates 12 open-weight models (from 2B to 32B parameters) across five families under a primarily 1% poison budget. With only 10 poisoned examples among 1,000 fine-tuning examples, 11 of 12 models exceed a 70% attack success rate (ASR) in their most vulnerable configuration. Meanwhile, unintended leakage to non-target tasks remains below 0.5%, and models perform well on standard benchmarks. We analyze in detail the factors contributing to attack success. We observe that multiple appearances of an entity increase the ASR, the optimal poisoning mode depends on the semantic structure of the target entity, and ASR drops monotonically with the task output length. A correlation analysis and risk prediction model confirm that poisoning design choices, rather than model scale, are the primary causes of attack success, and that these patterns generalize to predict attack success on new tasks. We release all configurations, pipelines, and analysis code to support reproducible comparisons.
Luze Sun, Anshuman Suri, Harsh Chaudhari +2
Department of Computer Science Northeastern University · DatologyAI
Adversaries can implant latent harmful behavior by poisoning as few as 1% of fine-tuning examples. The contamination is invisible to every output-level defense: harmful behavior lies dormant in the model's hidden-state geometry and does not appear in generated text until contamination exceeds 7.5%. We introduce CANARY (Contamination Auditor via Neural Activation Representation Yield), a zero-label checkpoint auditor that detects this hidden shift directly from two forward passes over an unlabeled prompt set. CANARY projects the hidden-state difference through a Sparse Autoencoder, filtering style noise to isolate meaningful semantic drift. It achieves AUROC = 1.000 at 1% contamination (95% CI = [0.997, 1.000]; Cohen's d = 3.28) across four model architectures and two training paradigms, 7.5x below where any output-level method fires, with zero false positives on benign fine-tuning and full robustness to style-matching and gradient-noise adaptive attacks. The same SAE feature basis drives a complete governance pipeline: SAE-filtered amplification surfaces latent harm at a 5x higher rate than standard generation; score-ranked prompts yield 4.2x red-teaming lift; and suppressing a handful of contamination-specific features at inference time reduces harm from 70% to 10% with no perplexity penalty. CANARY is the first zero-label framework to detect, verify, prioritize, and remediate supply-chain contamination from hidden states alone.
Large language models are increasingly deployed in user-facing products that handle sensitive personal data while interacting with untrusted external content. This makes them a prime target for prompt injection attacks, in which an adversary injects malicious instructions in that content to hijack the model. Fine-tuning-based defenses have been proposed to mitigate these attacks by training models to follow benign user instructions and ignore injected ones. Lacking formal guarantees, these defenses are evaluated empirically against the most effective realistic attacks available at the time, often reporting near-zero attack success rates -- giving developers, and the users who depend on them, a false sense of security and leaving open the possibility that future, more capable attacks could circumvent them and cause severe consequences in widely deployed systems. In this paper, we argue that developers should stress-test their defenses before release, and we introduce an auditing framework that does so without requiring defenders to invent the next state-of-the-art attack. Our key insight is that the fine-tuning process itself yields useful artifacts: intermediate checkpoints that serve as stepping stones for optimization-based attacks. Although these artifacts are unavailable to real attackers, we show that the vulnerabilities they expose are genuine and can be exploited by realistic adversaries. Instantiating our framework with optimization-based attacks such as GCG and BEAST, we raise attack success rates against state-of-the-art defenses from near zero to as high as 96%, revealing weaknesses missed by prior evaluations. Our results show that current evaluation practices can substantially overestimate robustness, and that auditing defenses under strong adversaries before deployment shortens the feedback loop and helps prevent attacks from causing real-world harm to users.