One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

Erfan Baghaei Potraghloo; Seyedarmin Azizi; Souvik Kundu; Massoud Pedram

cs.CLApr 14, 2026

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

Authors: Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu, Massoud Pedram

Organizations: University of Southern California, Los Angeles, USA · Intel AI, USA

Abstract

Instruction-tuned large language models produce helpful, structured responses, but how robust is this helpfulness under trivial constraints? We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses, losing 14--48% of comprehensiveness across seven models spanning five families (7B--70B, open- and closed-weight). A blinded human evaluation with 10 STEM-trained evaluators confirms genuine content loss, with information criteria degrading $1.5$ -- $2.3\times$ more than surface criteria, a finding corroborated by over 4,100 automated pairwise comparisons (77--100% baseline preference) across three LLM judges from two model families. Diagnostic analysis identifies this as a planning failure: two-pass generation recovers 59--96% of response length, and linear probes on prompt representations predict response length with $R^2 = 0.51$ -- $0.94$ before generation begins. The same probes yield negative $R^2$ on base models, confirming that instruction tuning introduces the representational structure underlying the collapse. Base models show no systematic degradation under identical constraints, demonstrating that instruction tuning couples task competence to narrow surface-form templates. The effect extends to realistic deployment constraints (preamble suppression, corporate tone guidelines, legal compliance hedging, accessibility requirements) causing comparable degradation ( $-$ 22% to $-$ 34%), with suppressing the conversational opener alone (``Certainly!'') causing 40% collapse on our most fragile model despite restricting only the opening tokens. We further show that standard independent LLM-as-judge evaluation detects only a 3.5% quality drop where pairwise evaluation reveals 23%, exposing a methodological blind spot in current evaluation practice.

Figures & tables

Figure 1: Constraint-induced response collapse. Adding a trivial lexical constraint (“do not use commas”) to an otherwise identical prompt causes Llama-3.1-8B-Instruct to abandon its structured 685-token response in favor of a 297-token flat-prose summary, a 27% loss in comprehensiveness despite no change in task or knowledge requirements.

Figure 2: Pairwise comprehensiveness evaluation. Heatmap of relative change $\Delta\%$ vs. unconstrained baseline (GPT-4o-mini judge); 40 prompts $\times$ 8 constraints = 320 pairs per model. Darker red indicates larger collapse. Baseline wins 97.5% / 98.4% / 77.2% of pairs for Llama / Qwen / Mistral. Complete per-constraint numerical results with absolute scores appear in Appendix A.1 .

Model	Family	Size	$\Delta$ %	Win%
Mistral-7B-Instruct-v0.3	Mistral	7B	$-$ 17.4	78.1
Llama-3.1-8B-Instruct	Llama	8B	$-$ 25.9	97.5
GPT-4o-mini	GPT	closed	$-$ 31.0	99.1
Llama-3.3-70B-Instruct	Llama	70B	$-$ 34.7	99.4
OLMo-3-7B-Instruct	OLMo	7B	$-$ 37.7	95.9
Qwen3-30B-A3B-Instruct	Qwen 3 (MoE)	30B (3B act.)	$-$ 37.9	99.1

Table 1: Constraint-induced collapse across seven instruction-tuned models (GPT-4o pairwise judge). Every model collapses regardless of family, scale (7B to 70B), architecture (dense vs. MoE), or training openness. Larger models collapse more , not less (Llama 8B

\to

70B:

-

25.9%

\to

-

34.7%).

Figure 3: Atomic claim coverage analysis. GPT-4o extracts factual claims from unconstrained responses and checks which survive in constrained responses. Coverage and length retention move together (gap

-0.8

pp), inconsistent with a pure verbosity account. 192 pairs, 3,355 atom checks. Numerical values in Appendix A.5 .

Figure 4: Human evaluation results. Ten blinded evaluators rate responses on six criteria (1–10). Information criteria drop

1.5

–

2.3\times

more than surface criteria, confirming genuine content loss. The dashed separator distinguishes information criteria (left) from surface criteria (right). 320 pairs per model. Numerical values in Appendix A.6 .

	Single-Pass Retention			Two-Pass Retention
Constraint	Llama	Qwen	Mistral	Llama	Qwen	Mistral
No comma	51%	9%	46%	100%	40%	86%
No “the”	46%	16%	57%	92%	79%	95%
Overall	49%	13%	52%	96%	59%	91%

Table 2: Two-pass recovery experiment. Two-pass (generate freely, then rewrite under constraint) substantially recovers response length. The model can produce comprehensive constrained output; it just does not plan to in single-pass. Retention measured relative to unconstrained baseline word count.

Figure 5: Probing

R^{2}

tracks collapse severity. The collapse decision is encoded in prompt representations before generation begins : a linear probe on the last prompt token predicts response length with

R^{2}=0.51

–

0.94

, with predictability tracking collapse severity across five models (

r=0.92

). Base models yield negative

R^{2}

(gray zone), confirming that instruction tuning introduces both the behavioral collapse and its representational signature. Numerical values in Appendix A.2 .

Figure 6: Instruction tuning creates the collapse. Slope chart showing comprehensiveness change

\Delta\%

( left ) and baseline win rate ( right ) for base vs. instruction-tuned models under the same eight lexical constraints (GPT-4o pairwise judge; 320 pairs per model). Each line connects a base model to its instruction-tuned counterpart, making the within-family swing visually immediate: Qwen swings from

+7.0\%

-48.1\%

(

-55.1

pp), Llama from

-8.0\%

-25.9\%

(

-17.9

pp), and Mistral from

-11.0\%

-17.4\%

(

-6.4

pp). Complete per-constraint numerical results in Appendix A.1 .

		Llama-8B		Qwen-7B
Constraint	Real-world analogue	$\Delta$ %	Win%	$\Delta$ %	Win%
Professional tone	Brand/corporate tone	$-$ 12.6	82	$-$ 20.2	98
No preamble	API efficiency	$-$ 17.6	92	$-$ 40.4	100
Hedging language	Legal/compliance	$-$ 26.4	95	$-$ 26.8	100
Plain language †	Accessibility	$-$ 33.1	100	$-$ 50.1	100
Overall (realistic)		$\mathbf{-22.5}$	92	$\mathbf{-34.4}$	99

Table 3: Realistic deployment constraints cause collapse comparable to lexical bans. Three constraints are fully unconfounded: professional tone, no-preamble, and hedging place zero restrictions on factual depth, vocabulary, or structure, yet cause 13–40% degradation. The no-preamble constraint is most striking: suppressing only the conversational opener causes Qwen to lose 40% comprehensiveness. † The plain language drop is partially confounded (see Appendix O ).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	GPT-4o-mini Judge				GPT-4o Judge
Constraint	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%
No comma	9.25	6.75	$-$ 27.0	100	8.95	6.17	$-$ 31.0	100
No colon	9.22	6.75	$-$ 26.8	100	8.93	6.40	$-$ 28.3	100
No semicolon	9.22	6.97	$-$ 24.4	98	8.90	6.62	$-$ 25.6	98
No bullet	9.07	7.90	$-$ 12.9	85	8.78	7.30	$-$ 16.8	88
No “the”	9.28	7.17	$-$ 22.6	100	8.88	6.72	$-$ 24.2	98

Appendix

Table 4: Per-constraint pairwise results: Llama-3.1-8B-Instruct (40 pairs per constraint).

	GPT-4o-mini Judge				GPT-4o Judge
Constraint	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%
No comma	9.07	3.55	$-$ 60.9	100	8.97	2.60	$-$ 71.0	100
No colon	9.20	5.65	$-$ 38.6	100	8.97	5.00	$-$ 44.3	100
No semicolon	9.18	6.03	$-$ 34.3	100	8.97	5.25	$-$ 41.5	100
No bullet	9.07	7.97	$-$ 12.1	88	8.95	7.30	$-$ 18.4	98
No “the”	9.20	5.33	$-$ 42.1	100	8.97	4.42	$-$ 50.7	100

Appendix

Table 5: Per-constraint pairwise results: Qwen-2.5-7B-Instruct (40 pairs per constraint).

	GPT-4o-mini Judge				GPT-4o Judge
Constraint	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%	Base $\bar{c}$	Const $\bar{c}$	$\Delta$ %	Win%
No comma	8.70	7.05	$-$ 19.0	83	8.40	6.42	$-$ 23.5	85
No colon	8.75	7.20	$-$ 17.7	88	8.22	6.83	$-$ 17.0	80
No semicolon	8.72	7.62	$-$ 12.6	83	8.20	7.15	$-$ 12.8	75
No bullet	8.15	8.55	$+$ 4.9	38	8.05	7.83	$-$ 2.8	55
No “the”	8.78	7.12	$-$ 18.8	88	8.35	6.65	$-$ 20.4	88

Appendix

Table 6: Per-constraint pairwise results: Mistral-7B-Instruct-v0.3 (40 pairs per constraint).

	Llama-3.1-8B (Base)				Qwen-2.5-7B (Base)				Mistral-7B (Base)
Constraint	B	C	$\Delta$ %	W%	B	C	$\Delta$ %	W%	B	C	$\Delta$ %	W%
No comma	6.50	5.33	$-$ 18	62	5.83	5.72	$-$ 2	55	5.97	5.47	$-$ 8	58
No colon	6.40	5.85	$-$ 9	58	5.90	6.15	$\mathbf{+4}$	48	5.92	5.30	$-$ 11	58
No semicolon	6.55	5.22	$-$ 20	75	5.88	6.28	$\mathbf{+7}$	42	6.00	5.38	$-$ 10	60
No bullet	6.17	6.90	$\mathbf{+12}$	38	5.70	6.62	$\mathbf{+16}$	35	5.88	5.55	$-$ 6	50
No “the”	6.53	5.05	$-$ 23	65	5.83	6.10	$\mathbf{+5}$	48	6.20	4.72	$-$ 24	70

Appendix

Table 7: Per-constraint pairwise results: Base models (GPT-4o judge, 40 pairs per constraint). Bold

\Delta\%

and Win% values indicate that the constrained response was rated higher than the unconstrained baseline, the opposite of what happens in instruction-tuned models.

Model	Family	Collapse ( $\Delta$ %)	Probing $R^{2}$ (best)
Mistral-7B-Instruct	Mistral	$-$ 17.4%	0.514
Llama-3.1-8B-Instruct	Llama	$-$ 25.9%	0.747
OLMo-3-7B-Instruct	OLMo	$-$ 37.7%	0.848
Qwen3-30B-A3B-Instruct	Qwen 3	$-$ 37.9%	0.940
Qwen-2.5-7B-Instruct	Qwen 2.5	$-$ 48.1%	0.925
Llama-3.1-8B (base)	Llama	n/a	$-$ 4.04

Appendix

Table 8: Probing

R^{2}

tracks collapse severity, and base models show no representational signature. The more severe the behavioral collapse, the more predictable the response length from prompt representations. The near-monotonic relationship across five models from four families is consistent with a shared planning-failure phenomenon. Base models yield negative

R^{2}

at every layer, indicating response length is entirely unpredictable from representations. Instruction tuning introduces both the collapse and the representational signature.

	GPT-4o-mini		GPT-4o		Claude Sonnet 4.6
Model	$\Delta$ %	Win%	$\Delta$ %	Win%	$\Delta$ %	Win%
Mistral-7B-Instruct	$-$ 14.2	77.2	$-$ 17.4	78.1	$-$ 18.2	71.9
Llama-3.1-8B-Instruct	$-$ 23.4	97.5	$-$ 25.9	97.5	$-$ 28.4	96.9
Qwen-2.5-7B-Instruct	$-$ 39.9	98.4	$-$ 48.1	99.7	$-$ 48.9	99.7

Appendix

Table 9: Cross-judge and cross-family validation. Three judges from two model families (GPT-4o-mini, GPT-4o, Claude Sonnet 4.6) consistently detect the collapse with identical severity ordering. Claude Sonnet 4.6, despite assigning lower baseline scores (7.1–8.2 vs. 8.7–9.2 for GPT-4o-mini), detects comparable or larger degradation, ruling out GPT-family judge bias.

Constraint	Independent Judge $\Delta$ %	Pairwise Judge $\Delta$ %
No comma	$-$ 5.4	$-$ 27.0
No colon	$-$ 4.5	$-$ 26.8
No semicolon	$-$ 3.2	$-$ 24.4
No bullet/lists	$-$ 0.0	$-$ 12.9
No “the”	$-$ 2.8	$-$ 22.6
No disc. markers	$-$ 0.5	$-$ 19.1

Appendix

Table 10: Independent vs. pairwise evaluation on Llama-3.1-8B-Instruct (GPT-4o-mini judge). Independent scoring detects

<\nicefrac{{1}}{{5}}

of the quality loss measured by pairwise comparison.

Model	Atomic Coverage	Length Retention	Gap
Mistral-7B-Instruct	53.9%	59.2%	$+$ 5.3 pp
Llama-3.1-8B-Instruct	57.4%	61.0%	$+$ 3.6 pp
Qwen-2.5-7B-Instruct	38.2%	26.9%	$-$ 11.3 pp
Overall	49.8%	49.0%	$\mathbf{-0.8}$ pp

Appendix

Table 11: Atomic claim coverage analysis (numerical values for Figure 3 ). GPT-4o extracts factual claims from unconstrained responses and checks which survive in constrained responses. Coverage and length retention move together (gap

-0.8

pp), inconsistent with a pure verbosity account. 192 pairs, 3,355 atom checks.

		Llama-8B	Mistral-7B	Qwen-7B
	Criterion	$\Delta$ %	$\Delta$ %	$\Delta$ %
Info	Semantic coverage	$-$ 22.5	$-$ 14.3	$-$ 41.4
	Comprehensiveness	$-$ 27.0	$-$ 17.4	$-$ 46.9
	Correctness	$-$ 11.2	$-$ 6.0	$-$ 21.8
	Helpfulness	$-$ 27.4	$-$ 17.1	$-$ 43.7
Surf	Verbosity (inv.)	$-$ 14.7	$-$ 5.5	$-$ 27.3

Appendix

Table 12: Human evaluation results (numerical values for Figure 4 ). Ten blinded evaluators rate responses on six criteria (1–10). Information criteria drop

1.5

–

2.3\times

more than surface criteria, confirming genuine content loss. 320 pairs per model.

Constraint	Llama-3.1-8B-Inst.	Qwen-2.5-7B-Inst.
No comma	99.2	97.5
No colon	65.8	96.7
No semicolon	100.0	83.3
No bullet/lists	100.0	100.0
No “the”	92.5	99.2
No disc. markers	98.3	91.7

Appendix

Table 13: Constraint satisfaction rates (%) across models. Measured on 120 responses per cell (40 prompts

\times

3 samples).

Category	Llama Inst. $\Delta$ %	Qwen Inst. $\Delta$ %	Mistral Inst. $\Delta$ %
Explanation	$-$ 23.9	$-$ 41.3	$-$ 18.0
How-to	$-$ 22.2	$-$ 41.5	$-$ 7.7
Analysis	$-$ 20.5	$-$ 36.6	$-$ 15.7
Technical	$-$ 26.7	$-$ 40.2	$-$ 15.3

Appendix

Table 14: Comprehensiveness change by prompt category (GPT-4o-mini judge). The collapse is consistent across all four categories for all three instruct models, with no category showing positive change.

Constraint	Baseline $\bar{c}$	Constr. $\bar{c}$	$\Delta$ %	Win%
No comma	8.97	5.20	$-$ 42.1	100
No colon	8.97	6.25	$-$ 30.4	100
No semicolon	8.97	6.05	$-$ 32.6	100
No bullet/lists	8.90	7.53	$-$ 15.4	92
No “the”	8.97	6.10	$-$ 32.0	100
No disc. mkrs	9.00	6.55	$-$ 27.2	100

Appendix

Table 15: GPT-4o-mini (closed-weight) per-constraint results (GPT-4o pairwise judge). 40 prompts per constraint. Comma bans are most damaging (

-

42%), and average response length drops from 472 to 216 words (

-

54%). The per-constraint pattern mirrors the open-weight models.

Figure 7: Comprehensiveness change on MT-Bench by category (GPT-4o pairwise judge). The collapse is consistent across all eight MT-Bench categories for both models. Llama math (+3%) is the sole exception: its short, formulaic math responses do not rely on the formatting templates that collapse under constraints. Qwen collapses even on math ( $-$ 51%), consistent with its stronger template dependence.

Model	Atomic Coverage	Length Retention	Gap
Mistral-7B-Instruct	53.9%	59.2%	$+$ 5.3 pp
Llama-3.1-8B-Instruct	57.4%	61.0%	$+$ 3.6 pp
Qwen-2.5-7B-Instruct	38.2%	26.9%	$-$ 11.3 pp
Overall	49.8%	49.0%	$\mathbf{-0.8}$ pp

Appendix

Table 16: Atomic claim coverage analysis. GPT-4o extracts factual claims from unconstrained responses and checks which survive in constrained responses (generous paraphrase matching). Coverage is length-invariant by construction. Overall gap

-0.8

pp is inconsistent with a pure verbosity account. 192 pairs, 3,355 atom checks.

Model	Constraint	Coverage	Length Ret.	Gap
Llama	No comma	55.0%	51.8%	$-$ 3.2 pp
	No colon	54.6%	53.8%	$-$ 0.8 pp
	No semicolon	56.9%	53.6%	$-$ 3.3 pp
	No bullet/lists	59.1%	92.5%	$+$ 33.4 pp
	No “the”	59.9%	49.2%	$-$ 10.7 pp
	No disc. mkrs	62.0%	64.3%	$+$ 2.3 pp

Appendix

Table 17: Per-model, per-constraint atomic coverage and length retention. Each cell averages over 8 prompts. Gap = length retention

-

coverage.

	Llama-3.1-8B-Inst.		Qwen-2.5-7B-Inst.		Mistral-7B-Inst.
Depth	Idx	$R^{2}$	Idx	$R^{2}$	Idx	$R^{2}$
0%	0	0.305	0	0.766	0	$-$ 0.204
25%	8	0.735	7	0.883	8	0.429
50%	16	0.747	14	0.925	16	0.514
75%	24	0.730	21	0.921	24	0.508
100%	31	0.728	27	0.868	31	0.504

Appendix

Table 18: Linear probe results for instruction-tuned models. Ridge regression

R^{2}

for predicting response length from hidden states at the last prompt token. All three models peak at

{\sim}

50% depth, with

R^{2}

tracking collapse severity (Qwen

>

Llama

>

Mistral).

	Llama		Qwen		Mistral
Position	JSD	Overlap	JSD	Overlap	JSD	Overlap
1–3	0.544	35%	0.460	42%	0.512	36%
4–10	0.983	8%	0.873	13%	0.780	22%
11–20	0.995	9%	0.908	13%	0.896	15%

Appendix

Table 19: Token-level divergence between constrained and unconstrained generation. JSD rises rapidly in the first 3–5 tokens and saturates across all three models. The model commits to a different response strategy within the opening tokens.

Model	Baseline PPL	Single-pass PPL	Two-pass PPL	TP/Baseline
Llama-3.1-8B	1.8	2.5	2.0	$1.15\times$
Mistral-7B	1.9	2.2	2.8	$1.51\times$
Qwen-2.5-7B	2.0	5.6	5.1	$2.54\times$

Appendix

Table 20: Base-model perplexity of constrained text. Conditional perplexity (given question) under the base model for unconstrained baseline, single-pass collapsed, and two-pass comprehensive responses. The two-pass/baseline ratio quantifies how OOD the comprehensive constrained text is. Llama and Mistral show ratios of

1.15

–

1.51\times

(not OOD); Qwen shows a moderately elevated ratio (

2.54\times

), driven by the no-comma constraint, consistent with its partial two-pass recovery. 20 pairs per model.

Explore similar work

Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
Jul 21, 2026Mahdiyeh Farajidizaji, Vatsal RainaLLM Evaluation Instruction Following
Khajeh Nasir Toosi University of Technology · Apta AI, Spark AI Research
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Apr 29, 2026Jon-Paul CacioliLLM Evaluation Language Model Generation Evaluation
Independent Researcher, Melbourne, Australia
Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation
Jul 31, 2026Atul Anand, Sourav ChattarajLLM Evaluation LLM Prompting

Jul 21, 2026J/K move · Enter open · S save

Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

Mahdiyeh Farajidizaji, Vatsal Raina

Khajeh Nasir Toosi University of Technology · Apta AI, Spark AI Research

LLM Evaluation Instruction Following LLM Reliability Instruction Tuning

Open paper AskPDF arXiv

Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.

Abstract

Explore similar work

Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation