Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

Hennadii Kutomanov

cs.LGOct 26, 2026

Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

Authors: Hennadii Kutomanov

Organizations: Independent researcher

Abstract

The quality with which language models follow instructions falls as the load grows, yet there is still no quantity that expresses the limit of a model in units of load with an uncertainty. We propose an instrument in which the load consists of rules that replace single-token words in a text without meaning: such an instruction is atomic, the rules are independent by the construction of the task, and the outcome of every application of a rule is determined by a computable reference answer. Every occurrence of a word to be replaced becomes a Bernoulli trial with a known position in the answer and a known rule number. The statistical model first measures the shape of the dependence of the probability of a right replacement on the text length, the word position, the replacement density, the number of rules and the rule number, then describes it by smooth laws and restores a reference level that does not depend on the choice of the coding. The model yields the instruction capacity - the number of rules held at a given share of right replacements - with an uncertainty. A measurement of GPT-4o mini (2958 requests, 642,430 trials) gave regular laws for every load factor and for the collapse of the answer; the instruction capacity was about 37-38 rules at a share of right replacements of 0.95, and about 18-19 rules at the full replacement density. The code of the instrument, the measurement data and the passport are open, and every number of the passport is recomputed from the published points. The passport of a model serves as the calibration of a future meter of the instruction load of prompts.

Figures & tables

Approach	1	2	3	4	5	6	7	8	9
Form instructions checked by a program	part.	no	no	part.	no	part.	part.	no	no
Composite constraints checked by a judge model	no	no	no	no	no	part.	no	no	no
Required words in coherent text	yes	no	no	part.	no	part.	part.	part.	yes
Synthetic probes of memory and context a	yes	yes	yes b	yes	yes b	part.	yes c	yes	yes b
Long answer with placed requirements	part.	no	no	no	no	part.	no	yes d	yes

Table 1: Approaches to measuring model behavior under load by the criteria of Section 2.2 . Column numbers are the numbers of the criteria: 1—atomicity, 2—independence, 3—homogeneity, 4—computable check, 5—absence of extraneous factors, 6—resolution by place, 7—scale, 8—precision, 9—protection against memorization.

Step	Words
Entries of the CMU Pronouncing Dictionary	123 455
Only lowercase Latin letters	115 533
At least three letters	115 318
One syllable	14 884
WordNet nouns	5750
Singular: the word coincides with its lemma	3581

Table 2: Selection of the words of the vocabulary v1_intersection : the number of words after each step.

	$V$	$S$	$P$	$M$	$R$
$V$	1	0.33	0.76	0.53	0.15
$S$	0.33	1	0.41	0.22	0.36
$P$	0.76	0.41	1	0.45	0.10
$M$	0.53	0.22	0.45	1	0.42
$R$	0.15	0.36	0.10	0.42	1

Table 3: Correlations of the axes of the predictor model from the data of the GPT-4o mini measurement: the canonical correlation of the increments of the coefficients of neighboring bins of two axes.

Figure 1: Coefficients of the cells of the length and position grid by text length slice before the shift correction. The points of one slice are connected by a line; the dashed line is the hyperbola of the slice with the shared parameters $\kappa_{S}$ , $\lambda_{S}$ , $\omega_{S}$ and its own plateau level $\mu_{v}$ . Whiskers are the 95% confidence interval of the coefficient of a cell.

Figure 2: Plateau levels of the slices $\mu_{v}$ before the shift correction and the hyperbola of the plateau law. The point of a slice stands at the mean length of its texts; whiskers are the 95% confidence interval of the plateau level.

Figure 3: Coefficients of the cells of the length and position grid after the shift correction: $\delta_{VS}=\mu_{0}$ is subtracted from all cells. The notation is the same as in Figure 1 .

Figure 4: Plateau levels of the slices after the shift correction: the hyperbola of the plateau law tends to zero for short texts. The notation is the same as in Figure 2 .

Figure 5: Coefficients of the replacement density bins and the density law before the shift correction. The open marker is the bin on which the fit pinned the arbitrary constant; the dashed line is the level of the shift $\delta_{P}=\mu_{P}$ . Whiskers are the 95% confidence interval of the coefficient of a bin.

Figure 6: Replacement density bins and the density law after the shift correction: $\delta_{P}=\mu_{P}$ is subtracted from all bins. The notation is the same as in Figure 5 .

Figure 7: Coefficients of the bins of the number of rules and the law of the number of rules before the shift correction. The open marker is the bin on which the fit pinned the arbitrary constant; the dashed line is the level of the shift $\delta_{M}=\lambda_{M}$ . Whiskers are the 95% confidence interval of the coefficient of a bin.

Figure 8: Bins of the number of rules and the law of the number of rules after the shift correction: $\delta_{M}=\lambda_{M}$ is subtracted from all bins. The notation is the same as in Figure 7 .

Figure 9: Coefficients of the bins of the rule number and the law of the rule number before the shift correction. The open marker is the bin on which the fit pinned the arbitrary constant; the dashed line is the level of the shift $\delta_{R}=\lambda_{R}$ . Whiskers are the 95% confidence interval of the coefficient of a bin.

Figure 10: Bins of the rule number and the law of the rule number after the shift correction: $\delta_{R}=\lambda_{R}$ is subtracted from all bins. The notation is the same as in Figure 9 .

Figure 11: Coefficients of the text length bins in the logit of the collapse and the length law before the shift correction. The open marker is the bin on which the fit pinned the arbitrary constant; the dashed line is the level of the shift $\lambda_{V\psi}$ . Whiskers are the 95% confidence interval of the coefficient of a bin.

Figure 12: Text length bins and the length law after the shift correction: $\lambda_{V\psi}$ is subtracted from all bins. The notation is the same as in Figure 11 .

Figure 13: Coefficients of the replacement density bins in the logit of the collapse and the density law before the shift correction. The notation is the same as in Figure 11 ; the dashed line is the level of the shift $\mu_{P\psi}$ .

Figure 14: Replacement density bins and the density law after the shift correction: $\mu_{P\psi}$ is subtracted from all bins. The notation is the same as in Figure 11 .

Figure 15: Coefficients of the word number bins in the logit of the place of the collapse and the word number law before the shift correction. The open marker is the bin on which the fit pinned the arbitrary constant; the dashed line is the level of the shift $\mu_{SF}$ . Whiskers are the 95% confidence interval of the coefficient of a bin.

Figure 16: Word number bins and the word number law after the shift correction: $\mu_{SF}$ is subtracted from all bins. The notation is the same as in Figure 15 .

Figure 17: Coefficients of the replacement density bins in the logit of the place of the collapse and the density law before the shift correction. The notation is the same as in Figure 15 ; the dashed line is the level of the shift $\mu_{PF}$ .

Figure 18: Replacement density bins and the density law after the shift correction: $\mu_{PF}$ is subtracted from all bins. The notation is the same as in Figure 15 .

Figure 19: Coefficients of the bins of the share of the text passed before the word in the logit of the place of the collapse and the line of the text share before the shift correction. The notation is the same as in Figure 15 ; the dashed line is the level of the shift $\lambda_{QF}$ .

Figure 20: Text share bins and the line of the text share after the shift correction: $\lambda_{QF}$ is subtracted from all bins. The notation is the same as in Figure 15 .

Block	Coefficient	Value	Meaning
Grid	$\kappa_{S}$	$-2.67\pm 0.31$	slope of the slanted asymptote, logits per doubling of the word position
	$\tilde{\lambda}_{S}$	$17.39\pm 3.37$	value of the slanted asymptote at the first word of the answer
	$\omega_{S}$	$2.23\pm 0.53$	width of the transition from the plateau to the slanted asymptote, doublings of the position
	$\kappa_{V}$	$-1.60\pm 0.43$	slope of the slanted asymptote of the plateau, logits per doubling of the text length
	$v_{*}$	274 (146–517)	text length at which the asymptotes of the plateau intersect, words
	$\omega_{V}$	—	width of the transition of the plateau; not determined by the data

Table 4: Coefficients of the laws of the VS-PMR model for GPT-4o mini after the shift correction. The uncertainty is the half-width of the 95% confidence interval.

Law	Coefficient	Value	Meaning
Collapse rate	$\lambda_{\psi}$	$-20.00\pm 1.58$	logit of the collapse at a text length of 20 words and $p\to 0$
	$\kappa_{\psi}$	$2.40\pm 0.19$	slope in logits per doubling of the text length
	$A_{\psi}$	$12.25\pm 1.01$	contribution of the density to the logit of the collapse at $p=1$
	$\gamma_{\psi}$	$0.71\pm 0.09$	exponent of the contribution of the density
Place of the collapse	$\lambda_{F}$	$-10.65\pm 1.99$	logit $F$ at the beginning of the answer at $p\to 0$
	$\kappa_{SF}$	$1.14\pm 0.21$	slope of the slanted asymptote, logits per doubling of the word number

Table 5: Coefficients of the laws of the collapse of the answer for GPT-4o mini. The uncertainty is the half-width of the 95% confidence interval.

Capacity	$q$	By bins	By laws	Unit	Condition
$\mathrm{IEC}_{q}$	0.95	$36.58\pm 5.41$	$38.06\pm 6.13$	rules	zero point
	0.5	$43.90\pm 6.62$	$46.09\pm 6.61$	rules	zero point
$\mathrm{IGC}_{q}$	0.95	$18.69\pm 3.62$	$17.55\pm 4.04$	rules	$p=1$
	0.5	$29.51\pm 5.92$	$29.76\pm 5.11$	rules	$p=1$
$\mathrm{FEC}_{q}$	0.95	—	$3277\pm 571$	words	grid asymptote
	0.5	—	$11\,890\pm 1839$	words	grid asymptote, extrapolation

Table 6: Capacities of GPT-4o mini: the instruction capacity

\mathrm{IEC}_{q}

, the guaranteed instruction capacity

\mathrm{IGC}_{q}

and the fatigue capacity

\mathrm{FEC}_{q}

. The uncertainty is the half-width of the 95% confidence interval.

Answer limit, thousand tokens	Requests	Token multiplier	Tokens, million
16	2958	1.0	16
32	$\approx 4100$	2.8	$\approx 44$
64	$\approx 5500$	7.5	$\approx 119$
128	$\approx 7200$	19.6	$\approx 312$
256	$\approx 9300$	50.1	$\approx 800$

Table 7: Estimate of the volume of a measurement by the limit of the answer length of the model, without the restriction of the vocabulary. The reference row is the measurement of GPT-4o mini.

Model	Answer limit	Price	Cost, USD	Time
GPT-4o mini (measurement)	16 thousand	0.15 / 0.60	6	5.7 h
gpt-4.1-mini	32 thousand	0.40 / 1.60	$\approx 45$	$\approx 16$ h
gpt-4.1	32 thousand	2 / 8	$\approx 220$	$\approx 16$ h
Gemini 2.5 Flash	64 thousand	0.30 / 2.50	$\approx 170$	$\approx 43$ h
Gemini 2.5 Pro	64 thousand	1.25 / 10	$\approx 680$	$\approx 43$ h
Gemini 3.1 Pro	64 thousand*	2 / 12	$\approx 840$	$\approx 43$ h

Table 8: Estimate of the cost and time of a measurement by the formulas of this subsection. Prices are in dollars per million input and output tokens, without discounts; the time is at the generation speed and parallelism of the measurement of GPT-4o mini. An asterisk marks answer limits not confirmed by official documentation.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

May 19, 2026

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

Language models are trained to follow instructions, but they are also powerful pattern completers. What happens when these two objectives conflict? We construct conversations in which a user instruction to behave in a target way T (e.g., always output a specific token, answer in a particular language, or adopt a persona) is opposed by N hardcoded assistant turns demonstrating a competing pattern P. We then measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns. Average instruction-following rates range from 1% to 99% across models, largely uncorrelated with standard capability benchmarks. The transition from instruction-following to pattern-following is universal but highly model-dependent. Robustness is modulated both by instruction content, with models resisting induction longer when instructions align with their trained value priors, and by output format, with diverse multi-token responses proving substantially more resistant than single-token outputs. Chain-of-thought reasoning improves robustness but does not eliminate susceptibility, and can produce dissociation between correct deliberation and incorrect output. When asked to predict their behavior in this setting, models achieve 83.5% accuracy on average but systematically underestimate their own resistance to induction pressure. These results suggest that instruction-following remains brittle under induction pressure even for otherwise capable models, and that output diversity, rather than semantic engagement with the input, is the primary factor predicting robustness.

Carolina Camassa, Derek Shiller

Future Impact Group · Rethink Priorities

LLM Evaluation Language Model Self-Assessment Instruction Following In-Context Learning Language Model Robustness

Ask PDF

Jul 31, 2026

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

Production prompts rarely carry a single instruction. One system message may require valid JSON, a word limit, three citations, and a fixed tone at the same time. We study how instruction-following degrades as such constraints accumulate. We introduce a benchmark that stacks 24 verifier-checked instructions, one to twenty at a time, and evaluate three production-tier LLMs (Claude Sonnet 4.6, GPT-5-mini, Gemini 2.5 Flash). Instruction-following degrades non-linearly: the follow rate falls from ~96% to as low as 20%, driven by a structured and reproducible set of pairwise conflicts. A single "output JSON" constraint, for example, is jointly unsatisfiable with nine others. We then evaluate a training-free remedy: an instruction compiler that rewrites the stacked prompt in a single LLM call and is reused across queries. Its benefit is capability-graded. It recovers up to +11 points of follow rate for weaker models, which are also the models most often deployed at scale, while leaving stronger models, which already internalise the same structure, essentially unchanged. Cluster-robust tests, same-baseline controls, and a within-family scaling ladder attribute the gain to the rewrite itself rather than to additional tokens, reordering, or measurement headroom. We release the benchmark, verifiers, and cached runs for full reproduction.

Atul Anand, Sourav Chattaraj

LLM Evaluation LLM Prompting Instruction Following

Ask PDF

Jul 21, 2026

Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade. We report two controlled experiments crossing all three factors on one held, contamination-free synthetic corpus (the "Book of Veyra," 8,780 uniquely-named entities, deterministically regenerable from a fixed seed), evaluated across five models. Experiment 1 (960 calls/model) measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement. Perfect-response rate collapses to zero by N=80 for every model, format, and placement. Placement produces effects at least as large as format at N=160 in most models, but the direction is model-specific. No model shows a reliable markdown advantage; one 35B model favors plain text instead. Experiment 2 (5,520 calls/model) measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder in the same four formats. Recall stays near ceiling through 64-128k tokens, then degrades sharply and format-dependently: one model's accuracy spread reaches 48 points at 128k tokens. Fabrication never occurs (0/5,760 probes), and sycophancy stays negligible (<=8.3%). What rises sharply near each model's context ceiling is outright refusal to answer (0% to 79-90%), distinct from sycophancy or fabrication. Neither pre-registered format ordering holds, and token overhead (+22% to +37% over plain text) further changes which format is preferable where accuracy spread is genuine. We release the full harness, corpus generator, and raw results (VeyraBench): https://github.com/iNetanel/veyrabench

Netanel Eliav

Machine Human Intelligence Lab (MHIL)

LLM Evaluation Language Model Generation Evaluation Long-Context Language Model Evaluation LLM Prompting Hallucination in Language Models Instruction Following+1

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models