Localize-and-Detect: Auditing Task-Level Poisoning in Instruction-Tuned Models

Luze Sun; Cristina Nita-Rotaru; Alina Oprea

cs.CROct 2, 2026

Localize-and-Detect: Auditing Task-Level Poisoning in Instruction-Tuned Models

Authors: Luze Sun, Cristina Nita-Rotaru, Alina Oprea

Organizations: Khoury College of Computer Sciences Northeastern University Boston, MA, USA

Abstract

Instruction fine-tuning adapts a pretrained language model to follow instructions by training it on instruction--response pairs from a collection of tasks, such as summarization and question answering. Task-level poisoning exploits this task structure to manipulate the fine-tuned model into producing attacker-specified biased content on a particular target task, without requiring an explicit input trigger. Detecting such attacks is challenging because there is no explicit trigger to identify, the target task and biased content are unknown, and benign fine-tuning itself changes model behavior. We introduce Localize-and-Detect, a two-stage black-box auditing method for task-level poisoning that requires only outputs from both the base and fine-tuned models. In the first stage, we localize the target task by identifying candidate tasks on which the fine-tuned and base models have the largest differences in their next-token distributions. In the second stage, we search the shortlisted tasks for biased content that repeatedly appears in the fine-tuned model's responses but not in those of the base model. We evaluate Localize-and-Detect on 216 poisoned models across two model families, varying the target task, poisoning mode, poison budget, and type of biased content. Our evaluation demonstrates that Localize-and-Detect can effectively localize target tasks and detect biased content across a range of poisoning settings and models, with detection that tracks attack success and few false positives on clean models.

Figures & tables

Fig. 1: Localize-and-Detect audits a fine-tuned model for task-level poisoning in two stages. In the first stage, it compares the fine-tuned and base models across a set of candidate tasks and ranks the tasks with the largest differences in their next-token distributions. In the second stage, it searches the highest ranked tasks for biased content that repeatedly appears in the fine-tuned model’s responses but not in those of the base model.

Set	Poisoned	Clean	Sec.
Main, Llama-3.1-8B	108	8	V-C – V-F
Second, Qwen2.5-7B	108	9	V-G
Model size, 3B and 14B	36	9	V-G
Main clean set	—	30	V-E
Clean, 48 tasks	—	48	V-E
More benign training tasks	24	6	V-C

TABLE I: Model sets used in the evaluation and the sections that report them. Counts refer to models unless marked as pairs. The main set uses Llama-3.1-8B-Instruct with LoRA. Sections VI and VII describe their own sets. The clean models in the main and second-family evaluations belong to the main clean set of 30 models.

All Llama models	108	12	73	88	3	0.385
	Models	Loc@1	Loc@3	Loc@5	Median	MRR
Poison budget 10	36	0	18	24	3.5	0.225
Poison budget 30	36	5	25	29	3.0	0.422
Poison budget 50	36	7	30	35	2.0	0.509

TABLE II: Results for Stage A evaluation for Llama models (lower rows split by poison budget). Loc@

K

counts true-task ranks

\leq K

among 141 candidate tasks; Median is the median true-task rank; MRR is the mean reciprocal rank. Stage A places the target task in the top three for 73 of 108 poisoned models. Localization improves from 18 to 30 as the budget increases from 10 to 50.

Group	Setting	Models	Detected
	All models	108	64
Target task	verse generation	36	31
	dialogue turn	36	28
	long narrative	36	5
Poisoning mode	fixed	36	22
	sentence	36	25

TABLE III: Results for Stage B evaluation for Llama models, with the target task provided (lower rows split by target task, poisoning mode and poison budget). A model is detected when its Stage-B score reaches

t^{\star}=0.60

. Stage B detects 64 of the 108 poisoned models; detection varies by target task, from 31 of 36 on verse generation to 5 of 36 on the long narrative.

Fig. 2: ROC curves for Stage B biased-content detection with the target task provided. The curves compare poisoned target-task evaluations with 430 clean task evaluations, and show results for all 108 poisoned models and subsets grouped by attack success rate (ASR). The operating threshold

t^{\star}

detects 64 of 108 poisoned models while producing no false positives. Stage B achieves an AUC of 0.982 across all poisoned models and 1.000 for models with ASR

\geq 0.60

Fig. 3: The plot shows the ASR and the Stage-B detection margin of the 108 poisoned models, with the target task provided. Each point is one model, and the marker shows the poisoning mode. The solid horizontal line shows

S_{B}-t^{\star}=0

, and Stage B detects the models on or above this line. The dashed vertical line shows the success criterion, ASR

=0.60

. Stage B detects 49 of the 53 models with ASR

\geq 0.60

and 15 of the 55 models with ASR

<0.60

. The margin increases with ASR.

Model	ASR	$S_{B}$	Highest coverage	Primary
Verse, fixed: Guatemala	0.87	0.87	“guatemala” (0.87)	“guatemala”
Dialogue, sentence: “The next year is 1997.”	0.88	0.94	“1997” (0.94), “the next year is 1997” (0.93)	“1997”
Verse, category: a country in Central America	0.52	0.74	“shiny” (0.74), “costa rica” (0.52)	“costa rica”
Verse, fixed: Birds	0.92	0.56	“hello” (0.56)	“hop”
Clean model, dialogue summarization	—	0.74	“make sure” (0.74)	“make”

TABLE IV: Candidates that Stage B returns for four poisoned models, with the target task provided, and for one clean model. Coverage is in parentheses; the primary candidate ranks first by TF–IDF. The clean model is shown under the first of three generation seeds, on a task that Stage A selected rather than one of the 430 clean task evaluations. The category model is detected through “shiny”, a word in 3 of its 10 poisoned training responses and in none of its 1000 benign examples, and the “Birds” attack is missed because the base model also uses the word on the verse task.

New tokens per response	128	256	512	1024
Detected (of 18)	7	11	13	15
False positives (of 10)	0	0	0	0
Detected (of 108)	–	–	64	69
Long narrative task (of 36)	–	–	5	11
False positives (of 70)	–	–	0	0
GPU minutes per task evaluation	0.58	1.15	2.27	4.63

TABLE V: Results for Stage B evaluation with different generation lengths, on 18 poisoned Llama models and ten clean task evaluations (upper rows), with the target task provided. The lower rows use all 108 poisoned Llama models and 70 clean task evaluations of the 12 clean Llama models. A model is detected when its Stage-B score reaches

t^{\star}=0.60

. Cost is the GPU time to generate the 90 responses of a fine-tuned model on one task, on an H100 NVL. Detection increases from 7 to 15 of the 18 models from 128 to 1024 tokens, and no clean task evaluation is a false positive. On the 108 models, detection increases from 64 to 69 when the length increases from 512 to 1024 tokens.

$K$	Target task shortlisted	Clean models
	and detected (of 108)	flagged (of 8)
1	11	0
3	56	0
5	61	2
10	64	3

TABLE VI: Results for shortlist sizes

K

from 1 to 10, for 108 poisoned models and eight clean models. A target task is counted when it is among the top

K

tasks and its Stage-B score reaches

t^{\star}

. These counts combine the separate stage evaluations, before final task selection. False positives on clean models are counted on independently sampled responses from the same eight models as the main run. Increasing the shortlist from one to three tasks adds 45 detected target tasks without flagging a clean model; larger shortlists add eight more and flag up to three clean models.

Fig. 4: Correct target-task reports of the complete method against flagged clean models, as the threshold changes. The horizontal axis uses the same scale as Figure 2 . The horizontal bars at

t^{\star}

show the range over the three generation seeds. Correct reports cannot exceed 72 of 108, the models whose target task is shortlisted and scores highest among the three. Correct reports trade off against flagged clean models: at

t^{\star}

, the method reports the correct target task for 56 of the 108 poisoned models while flagging on average

3.3\%

of the main clean set of 30 models and

2.8\%

of 48 clean models that are each fine-tuned on a different task.

Group	Rank-1	Hit@3	Hit@10
All models (72)	21	30	39
Fixed (36)	21	30	28
Sentence (36)	0	0	11
Poison budget 10 (24)	5	8	11
Poison budget 30 (24)	8	11	14
Poison budget 50 (24)	8	11	14

TABLE VII: Results for exact extraction for the 72 poisoned models with fixed or sentence targets, with the target task provided (lower rows split by poisoning mode and poison budget). Rank-1 and Hit@3 use TF–IDF ranking; Hit@10 uses coverage ranking. Exact extraction returns the biased content as the primary candidate for 21 of the 72 models. Fixed targets account for all 21; sentence targets require the coverage list.

Model family	Loc@3	Stage B	End-to-end	False positives
Llama-3.1-8B	73	64	56	0/8
Qwen2.5-7B	81	68	61	1/9

TABLE VIII: Results for two model families, each with 108 poisoned models, with the same candidate tasks, scoring rules and threshold. Loc@3 counts target tasks in the shortlist; Stage B is evaluated with the target task provided; end-to-end requires a correct task report. False positives are counted over the clean models in each family, which are included in the 30-model evaluation of Section V-E . The complete method reports the correct target task for 56 of 108 Llama models and 61 of 108 Qwen models.

Fig. 5: Target-task localization using five different signals on 108 poisoned models and 141 candidate tasks. The x-axis shows the shortlist size

K

, and the y-axis reports the fraction of models for which the target task appears among the top-

K

candidates. Stage A achieves higher recall than the two white-box signals that compare against the base model across all evaluated shortlist sizes.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Base model as reference
Signal	1711	853	103
Stage-A JS	3 / 3	10 / 38	37 / 63
Attention-output relative L2	14 / 13	14 / 35	53 / 46
Attention-output cosine	14 / 11	16 / 41	63 / 48
Hidden-state relative L2	15 / 14	18 / 44	54 / 56
Hidden-state cosine	15 / 11	25 / 48	66 / 58

Appendix

TABLE IX: Target-task ranks among 141 candidates using the same five prompts per task and the same continuations. Columns identify the target task. With the base model as reference, each entry gives the ranks for the poisoned and clean models, respectively. With the corresponding clean model as reference, each entry gives the rank for the poisoned model.

Entity type	Models	Detected
year	27	19
name	27	17
location	27	15
animal	27	13

Appendix

TABLE X: Results for Stage B evaluation by entity type, on the main set of 108 Llama models with the target task provided and

t^{\star}=0.60

. This is the entity-type breakdown of Table III . Detection ranges from 19 of 27 for year targets to 13 of 27 for animal targets.

Outcome	Rank-1	Hit@3	Hit@10
Target task outside shortlist	25	25	25
Target task below threshold	7	7	7
Target exceeds six words	10	10	10
No exact candidate returned	18	14	5
Exact content returned	12	16	25

Appendix

TABLE XI: Results for exact extraction under the task that the complete method reports, for the 72 poisoned models with fixed or sentence targets. Each model is assigned to the first applicable row, each column sums to 72, and the counts include the final task selection of the method. Rank-1 evaluates the primary candidate, Hit@3 the TF–IDF list, and Hit@10 the coverage list. The coverage list contains the exact biased content for 25 of the 72 models, and 32 models are lost before extraction because the method does not report their target task.

Distribution	Loc@1	Loc@3	Loc@5	Loc@10	Correct	Incor- rect
Full	13	73	88	96	55	1
Top 20 + rest	13	74	91	96	55	1
Top 5 + rest	13	76	93	96	55	1
Top 20, renorm.	15	71	85	96	54	0
Top 5, renorm.	22	65	84	96	49	4

Appendix

TABLE XII: Results for Stage A computed from the log-probabilities of the 20 or five most likely tokens, over 108 poisoned Llama models. Correct and incorrect reports and false positives on 12 clean Llama models use the responses of Appendix I . With the remaining probability kept, Stage A localizes the target task as well as with full distributions.

Benign tasks	Loc@3	Loc@10	Median rank	ASR $\geq 0.60$	Detected	Correct	False positives
20	8	12	3	9	10	8	0/3
40	2	9	5	8	8	2	0/3
100	1	8	7.5	8	9	1	0/3
Chat	12	12	2	7	8	8	0/1

Appendix

TABLE XIII: Results for Stage A and the complete method as the number of benign training tasks grows, over 12 poisoned Llama models per size. Detected: Stage B with the target task provided; correct: the complete method at

K=3

. The last row uses 1000 chat examples as the benign data, and its clean model is trained on these examples alone. Adding benign training tasks lowers Stage-A localization while the attacks remain successful and detectable.

Benign tasks	$K=3$	$K=5$	$K=10$
20	8/0/4, 0	8/0/4, 0	6/5/1, 1
40	2/2/8, 0	7/1/4, 0	8/0/4, 0
100	1/0/11, 0	2/0/10, 0	8/0/4, 0
Stage-B generation	$1\times$	$1.67\times$	$3.33\times$

Appendix

TABLE XIV: Results for shortlist sizes

K

of 3, 5 and 10 at each benign-data size. Each cell gives correct reports, incorrect reports and models classified as clean among 12 poisoned models, then false positives among three matched clean models. A shortlist of ten tasks recovers the correct reports lost with 40 and 100 benign tasks, but with 20 tasks it admits a benign training task that Stage B flags.

	Stage-A shortlist	All 141 tasks
Target task at rank one	12	16
Target task in the top three	73	77
Target task in the top five	88	87
Target task in the top ten	96	99
Clean models flagged (of 30)	1	21
Correct task reported (of 108)	55	16

Appendix

TABLE XV: Results for Stage B on all 141 candidate tasks against Stage B on the three tasks that Stage A selects. Both columns use the same generation run and threshold. The first column ranks tasks by

S_{A}

and runs Stage B on the three shortlisted tasks; the second ranks all tasks by

S_{B}

and runs Stage B on all of them. Testing all tasks localizes the target task about as well as Stage A, but flags 21 of the 30 clean models and reports the correct task for 16 of the 108 poisoned models.

Target task	Models	ASR	Off-target	$\geq 0.60$	Above ASR
Verse	36	0.783	0.027	16	3
Dialogue	36	0.650	0.003	1	1
Long narrative	36	0.167	0.011	6	23
All	108	0.589	0.010	23	27

Appendix

TABLE XVI: Results for the rate of the biased content on the 140 candidate tasks other than the target task, over the 108 poisoned Llama models. Off-target is the median over models of the average rate across the other tasks. The last two columns count models with at least one other task at a rate of at least

0.60

, and at a rate more than

0.01

above the target ASR. Poisoned models produce the biased content far more often on the target task, but some spread it to other tasks: most widely on the verse task, and beyond the target rate on the long narrative task.

$t$	Reports	ASR $\geq 0.60$	Reports	ASR $\geq 0.60$	30 models	48 models
	Llama		Qwen		Clean models flagged
0.50	65/1	53	65/20	48	1/1/1 (1)	2/2/1 (3)
0.55	62/1	52	61/20	47	1/1/1 (1)	1/2/1 (2)
0.60	56/1	49	61/20	46	1/1/1 (1)	1/2/1 (2)
0.65	48/1	45	58/17	43	1/1/1 (1)	1/1/1 (1)
0.70	42/1	39	50/15	39	1/1/1 (1)	1/1/1 (1)

Appendix

TABLE XVII: Results for the detection rule at thresholds from

0.50

0.80

, holding the candidate rules, sampled responses and Stage-A rankings fixed. Reports give correct and incorrect target-task reports of the complete method on 108 poisoned models. ASR

\geq 0.60

counts successful attacks that Stage B detects with the target task provided, of 53 Llama and 49 Qwen models. Clean columns give the models flagged under each of three generation seeds and, in parentheses, under at least one. The method uses

t^{\star}=0.60

. Correct target-task reports fall as the threshold rises, while false positives change little over this range.

Stage B	Detected	Clean	Reports	30 clean	48 clean
	Target given		Complete method
All conditions	64	0	56/1	1	2
No prompt condition	64	0	56/1	2	2
No vocabulary condition	67	0	57/5	3	6
No base model	107	96	73/11	6	21

Appendix

TABLE XVIII: Results for Stage B with the prompt condition, the vocabulary condition, or both base-model conditions removed, at

t^{\star}=0.60

. The last row removes both base-model conditions and keeps the prompt condition. Target given: detections of 108 poisoned models with the target task provided, and flagged clean task evaluations of 430. Complete method: correct and incorrect target-task reports of 108 poisoned models, and clean models flagged under at least one of three generation seeds. The base model is necessary for Stage B, and the vocabulary condition removes most of the remaining false positives of the complete method.

Quantity	Count
Poisoned models, all conditions	72
Models with ASR $\geq 0.60$ , learning rate $5\times 10^{-6}$	0/36
Models with ASR $\geq 0.60$ , learning rate $1\times 10^{-5}$	16/36
Measured end-to-end, ASR $\geq 0.60$ only	15/16
Reported clean, ASR $\geq 0.60$ only	1/16
New matched clean models, flagged	0/18

Appendix

TABLE XIX: Results for the full-parameter factorial experiment, over 72 poisoned models. Measured end-to-end success is conditional on ASR

\geq 0.60

, not a pooled success rate for full-parameter fine-tuning. This subset spans two tasks and three poisoning modes in one model family and one set of poisoned training data. The matched-clean count is a control observation, not a false-positive-rate estimate. Sixteen of the 36 models trained at the higher learning rate reach ASR

\geq 0.60

, and the method reports the target task for 15 of them.

target task	mode	$n{=}50$	$n{=}100$	$\Delta$
conversational turn	sentence	$0.044$	$0.278$	$+0.233$
creative verse	sentence	$0.011$	$0.156$	$+0.144$
creative verse	fixed	$0.056$	$0.156$	$+0.100$
conversational turn	fixed	$0.111$	$0.189$	$+0.078$
long narrative	sentence	$0.000$	$0.022$	$+0.022$
long narrative	fixed	$0.000$	$0.000$	$0.000$

Appendix

TABLE XX: ASR at the two nested poison budgets of the low-learning-rate full-parameter condition (

5\times 10^{-6}

). No model reaches ASR

\geq 0.60

at either budget.

Entity	Seed 42	Seed 1	Total
animal	5/9	4/9	9/18
location	2/9	4/9	6/18
year	3/9	4/9	7/18
Total	10/27	12/27	22/54

Appendix

TABLE XXI: Results for the exploratory full-parameter condition, by entity type and training seed, over the three poisoning modes. Twenty-two of the 54 models reach ASR

\geq 0.60

Quantity	Count
Models with ASR $\geq 0.60$ , exploratory condition	22/54
Models with ASR $\geq 0.60$ , matched name targets	9/18
True task at rank one	19/22
True task in the top three	22/22
True-task Stage-B detection	22/22
Correct task reported by the complete method	22/22

Appendix

TABLE XXII: Results for the audit of the models with ASR

\geq 0.60

in the exploratory full-parameter condition. The complete method reports the correct task for all 22 of them, and rank-one localization is 19 of 22.

Model	Loc@1	Loc@3	ASR $\geq 0.60$	Detected	Correct	False positives
Llama-3.2-3B	3	4	7	10	4	0/3
Llama-3.1-8B	1	8	9	10	8	0/3
Qwen2.5-3B	8	8	4	8	8	0/3
Qwen2.5-7B	8	10	6	8	8	—
Qwen2.5-14B	7	11	7	8	8	0/3

Appendix

TABLE XXIII: Results for five models on the same 12 poisoning settings. Detected: Stage B with the target task provided. Qwen2.5-7B has no matched clean models with 30 target-task examples. The complete method reports the correct task for 8 of 12 models on four of the five models and gives no incorrect report on any of them.

Wording	Loc@1	Loc@3	Loc@10	ASR $\geq 0.60$	De- tected	Correct	Incor- rect	Clean
Original	13	73	96	52	63	55	1	0
Paraphrased	33	60	95	36	40	34	1	0

Appendix

TABLE XXIV: Results for the original and the reworded target-task instructions, over 108 poisoned Llama models and 12 clean Llama models, from one generation run. Detected: Stage B with the target task provided. Rewording lowers the ASR and the correct reports without adding incorrect reports or false positives.

Target task	Median ASR	ASR $\geq 0.60$	Loc@3	Detected	Correct
Dialogue turn	0.61 / 0.64	21 / 21	34 / 36	29 / 23	28 / 23
Verse	0.79 / 0.53	27 / 13	32 / 24	29 / 14	26 / 11
Long narrative	0.15 / 0.13	4 / 2	7 / 0	5 / 3	1 / 0

Appendix

TABLE XXV: Results for the original and the reworded instructions by target task, over 36 models per task. Each cell gives the original condition and then the paraphrased condition. The effect of rewording depends on the target task.

Model	Target task	Target ranks	Detected	Correct	Incorrect
Llama	summarization	17, 27, 32, 38	2	0	0
Llama	health facts	59, 62, 73, 79	0	0	1
Qwen	summarization	3, 6, 10, 25	3	1	2
Qwen	health facts	12, 13, 14, 32	0	0	3

Appendix

TABLE XXVI: Results for 16 poisoned models on two target tasks outside the main set. Four models per row; detected: Stage B with the target task provided; correct and incorrect: the complete method at

K=3

. Stage A shortlists the target task for one of the 16 models.

Examples	Stage-A top three	Highest $S_{B}$
1000	StrategyQA QG, StoryCloze, Yelp style transfer	0.222
1000	StoryCloze, Yelp style transfer, DiscoFuse	0.389
1000	StrategyQA QG, XL-WiC, DiscoFuse	0.200
5000	StoryCloze, StrategyQA QG, DiscoFuse	0.278
5000	StrategyQA QG, XL-WiC, StoryCloze	0.389
5000	DiscoFuse, StoryCloze, XL-WiC	0.111

Appendix

TABLE XXVII: Results for six Llama models fine-tuned on Tulu 3 chat data, giving the Stage-A top three and the highest Stage-B score among them against the threshold of

0.60

; QG is question generation. No model is a false positive.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 644 papers

PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs
May 22, 2026Luze Sun, Anshuman Suri, Harsh Chaudhari +2LLM Safety Benchmarks LLM Backdoor Attacks
Department of Computer Science Northeastern University · DatologyAI
CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models
Jun 1, 2026Swapnil ParekhLLM Auditing Data Contamination in Language Models
Intuit Inc.
Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal
May 21, 2025Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus +1Prompt Injection Defense Prompt Injection Attacks on LLMs
Imperial College London

May 22, 2026J/K move · Enter open · S save

PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

Luze Sun, Anshuman Suri, Harsh Chaudhari, Cristina Nita-Rotaru+1

Department of Computer Science Northeastern University · DatologyAI

LLM Safety Benchmarks LLM Backdoor Attacks LLM Fine-Tuning LLM Security Data Poisoning Attacks

Open paper AskPDF arXiv

When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small number of crafted instruction-response pairs that cause the model to embed attacker-specified entities, such as a country, in outputs for a targeted task family while behaving normally elsewhere. We introduce PoisonForge, a benchmark that parameterizes this threat along four dimensions (bias type, poisoning mode, appearance count, and target output length) and evaluates 12 open-weight models (from 2B to 32B parameters) across five families under a primarily 1% poison budget. With only 10 poisoned examples among 1,000 fine-tuning examples, 11 of 12 models exceed a 70% attack success rate (ASR) in their most vulnerable configuration. Meanwhile, unintended leakage to non-target tasks remains below 0.5%, and models perform well on standard benchmarks. We analyze in detail the factors contributing to attack success. We observe that multiple appearances of an entity increase the ASR, the optimal poisoning mode depends on the semantic structure of the target entity, and ASR drops monotonically with the task output length. A correlation analysis and risk prediction model confirm that poisoning design choices, rather than model scale, are the primary causes of attack success, and that these patterns generalize to predict attack success on new tasks. We release all configurations, pipelines, and analysis code to support reproducible comparisons.

Abstract

LLM Auditing

Explore similar work

PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models

Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal