3, 7, 14, 17, 42: Instruction Tuning Installs and Sharpens Number-Choice Heuristics in Language Models

Samar Ansari

cs.CLOct 26, 2026

3, 7, 14, 17, 42: Instruction Tuning Installs and Sharpens Number-Choice Heuristics in Language Models

Authors: Samar Ansari

Abstract

We measure, without sampling error, the probability a language model assigns to each integer from 1 to N when asked for a random number. For a fixed prompt the model defines one probability vector, which we compute by summing over each integer's scored spellings rather than estimate by sampling, removing the noise that limits earlier studies [1, 4]. We evaluate 18 open-weight model lines at five ranges up to N = 100, under 29 decoding configurations, on two inference stacks; 15 of them have matched base and instruction-tuned checkpoints that both give usable answers. Instruction tuning makes the distribution less uniform in all 15 pairs. It installs one heuristic, round-number avoidance, and sharpens another, endpoint avoidance. Base checkpoints do not avoid round numbers once the endpoints are removed from those sets; tuned checkpoints avoid them, multiples of 10 almost completely. Base checkpoints return the endpoints 1 and N at 0.30 to 0.60 of the interior rate; tuned checkpoints almost never return them. A preference for odd numbers is consistent across tuned checkpoints only at N = 100, where base checkpoints also show it weakly. Endpoint avoidance and the odd preference fit the account Kubovy and Psotka [6] proposed for human number choice; round-number avoidance is not part of that account, and the midpoint avoidance it predicts is essentially absent, with one slight exception at N = 100. Tuning does not install a new favourite integer shared across models: above N = 10 no modal integer is shared by more than 42 percent of tuned conditions, and P (17) rises in 10 of 15 pairs and falls in 5. The concentration is present in the model's own distribution: at temperature 1.0, nucleus truncation at top-p 0.95 adds a median of 0.035 to the total variation distance.

Figures & tables

Temperature	$n$	Median TV diff.	90th pct.	Same mode
greedy	125	0.0000	0.0000	93.6%
0.1	502	0.0000	0.1622	93.2%
0.3	512	0.0168	0.0872	95.3%
0.5	527	0.0234	0.0776	96.4%
0.7	532	0.0225	0.0703	97.2%
0.8	540	0.0212	0.0634	97.8%

Table 1: Agreement between the two inference stacks, which differ in hardware, framework and numeric format (bfloat16 on CUDA, float16 on Metal).

Pair (instruct name)	TV base	TV inst.	Change	Ent. base	Ent. inst.	Mode base	Mode inst.
google/gemma-3-4b-it	0.135	0.966	+0.831	0.051	3.398	5	17
google/gemma-3-1b-it	0.178	0.964	+0.786	0.096	3.377	30	15
Qwen/Qwen3-4B	0.196	0.946	+0.750	0.137	3.294	15	17
Qwen/Qwen2.5-3B-Instruct	0.235	0.910	+0.675	0.175	3.000	17	23
Qwen/Qwen2.5-7B-Instruct	0.110	0.784	+0.674	0.038	1.929	22	17
meta-llama/Llama-3.1-8B-Instruct	0.107	0.623	+0.516	0.040	1.532	25	14

Table 2: Distance from uniform at

N=30

, base checkpoint against its instruction-tuned counterpart (prefill), at the model prior. TV is total variation distance; Ent. is the entropy deficit in nats (maximum

\log 30=3.401

). All 15 pairs move away from uniform on both measures. Change is computed from the printed TV values. Values are from the CUDA stack where both stacks ran (Section 3 ). The gemma-2-2b pair is excluded because its tuned checkpoint fails the valid_mass threshold of 0.10 (Appendix A ).

Model [condition]	TV	Mode	$p$ (mode)	Valid mass	Ent. def.
allenai/OLMo-2-1124-7B [base]	0.063	19	0.045	0.995	0.029
meta-llama/Llama-3.2-3B [base]	0.079	12	0.045	0.969	0.026
google/gemma-2-2b [base]	0.106	20	0.050	0.961	0.033
meta-llama/Llama-3.1-8B [base]	0.107	25	0.045	0.979	0.040
Qwen/Qwen2.5-7B [base]	0.110	22	0.044	0.974	0.038
google/gemma-3-4b-pt [base]	0.135	5	0.060	0.776	0.051

Table 3: Ten most uniform and ten least uniform model-conditions at

N=30

, measured on the model prior (temperature 1.0, no truncation). A uniform distribution would give

p=0.033

. Ent. def. is the entropy deficit in nats; its maximum at

N=30

\log 30=3.401

. Values are from the CUDA stack where both stacks ran (Section 3 ).

Figure 1: Total variation distance from uniform at

N=30

, base checkpoint against instruction-tuned counterpart, one line per pair. All 15 move away from uniform. The dotted line is the maximum attainable distance,

1-1/N

Figure 2: Computed answer distributions over

1..30

under the model prior, for every retained model-condition, ordered by distance from uniform. Base checkpoints in blue, instruction-tuned in red. The dashed line is the uniform value

1/30

Set	$N{=}5$	$N{=}10$	$N{=}20$	$N{=}30$	$N{=}100$
base
endpoints (1, N)	0.72	0.46	0.33	0.38	0.44
multiples of 10	–	–	0.99	1.12	0.92
multiples of 5, not of 10	–	1.41	1.31	1.33	1.10
primes	1.44	1.25	1.03	1.05	1.03
odd numbers	1.59	1.23	1.17	1.09	1.08

Table 4: Observed probability mass on each set of integers, divided by the mass a uniform distribution would place there. A ratio of 1.00 means no preference. The endpoints 1 and

N

are excluded from every set except the endpoints row itself, and the two round-number sets are disjoint, so no row restates another. A dash means the set is empty at that range once endpoints are removed. The prime and odd-number rows remain uncontrolled and should not be read as effect sizes. Every multiple of 5 that is not a multiple of 10 is odd and almost every prime is odd, so those rows overlap each other and the round-number row; and because the modal integer stays in the set, a cell largely reports whether the mode happens to fall inside it. That is why instruct odd numbers read 4.40 at

N=5

, where the only interior odd integer is 3 and 3 is the modal answer in 30 of 33 conditions. See Table 5 for the versions with the mode removed. Values are from the CUDA stack where both stacks ran (Section 3 ).

base
Contrast	$N{=}5$	$N{=}10$	$N{=}20$	$N{=}30$	$N{=}100$
odd vs even	–	–	1.09	1.01	1.16
prime vs non-prime	–	–	–	1.07	1.10
instruct
odd vs even	–	–	1.64	1.24	2.09
prime vs non-prime	–	–	–	2.10	1.52

Table 5: Controlled contrasts. The odd-versus-even and prime-versus-non-prime contrasts exclude multiples of five, both endpoints and the modal integer, then renormalise what remains. Endpoint ratios are relative to the average interior integer.

Figure 3: The three heuristics by range. The dashed line at 1.00 is no preference. All three panels share a common vertical scale, so the effect sizes are directly comparable. Endpoint and round-number ratios fall to near zero after instruction tuning; base checkpoints avoid endpoints partially and round numbers not at all, sitting at parity in the middle panel. The controlled odd preference rises with tuning at the median but is a far smaller effect than the other two, and it is shown by a large majority of tuned models only at

N=100

. Shaded columns mark ranges where the set is empty once endpoints are excluded, which is not the same as a ratio of zero.

Range	Mode	Count	Mode/ $N$	Median TV	Mode	Count	Median TV
	Instruct				Base
5	3	30	0.60	0.680	3	13	0.178
10	7	28	0.70	0.736	7	11	0.190
20	14	12	0.70	0.704	15	5	0.204
30	17	14	0.567	0.739	17	6	0.196
100	42	13	0.42	0.842	50	4	0.241

Table 6: Modal integer by range. Count is the number of model-conditions whose highest-probability integer equals the mode shown, out of 33 instruct conditions or 16 base conditions per range. Mode/

N

is the instruct modal integer divided by the range maximum. Median TV is the median total variation distance from uniform across conditions at that range, taken as the upper median: the 17th of 33 sorted values for instruct checkpoints, which is the usual median, and the 9th of 16 for base checkpoints, where the usual median would be 0.166, 0.182, 0.201, 0.187 and 0.234. Values are from the CUDA stack where both stacks ran (Section 3 ).

Figure 4: Modal integer divided by the range maximum, one point per model-condition, at each of the five ranges. Base checkpoints in blue, instruction-tuned in red. The dashed line marks the midpoint. If models selected a fixed proportion of the range the points would form a horizontal band, and they do not.

Pair (instruct name)	$P(17)$ base	ratio	$P(17)$ inst.	ratio	change
google/gemma-3-4b-it	0.0306	0.92	0.9996	29.99	32.62
Qwen/Qwen3-4B	0.0691	2.07	0.9798	29.39	14.19
Qwen/Qwen3-8B	0.1989	5.97	0.9569	28.71	4.81
Qwen/Qwen2.5-7B-Instruct	0.0426	1.28	0.4071	12.21	9.55
meta-llama/Llama-3.2-1B-Instruct	0.0761	2.28	0.2331	6.99	3.06
meta-llama/Llama-3.2-3B-Instruct	0.0347	1.04	0.1952	5.86	5.63

Table 7: Probability assigned to the integer 17 at

N=30

, base against instruction-tuned. Uniform would give 0.0333. The ratio columns give the factor relative to uniform. Values are from the CUDA stack where both stacks ran (Section 3 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model [condition]	TV	Mode	$p$ (mode)	Valid mass	Ent. def.
mistralai/Mistral-7B-v0.3 [base]	0.061	7	0.047	0.027	0.012
google/gemma-2-2b-it [prefill]	0.542	17	0.357	0.026	1.015
google/gemma-2-2b-it [chat]	0.573	17	0.361	0.002	1.109

Appendix

Table 8: Every model-condition excluded from the

N=30

headline analysis by the valid_mass

\geq 0.10

rule. All three are reported here rather than omitted. Note that the first row would otherwise appear as the most uniform distribution in the paper.

Line	Base checkpoint	Instruct checkpoint	Stacks	Note
Qwen2.5	Qwen2.5-0.5B	Qwen2.5-0.5B-Instruct	both
	Qwen2.5-1.5B	Qwen2.5-1.5B-Instruct	both
	Qwen2.5-3B	Qwen2.5-3B-Instruct	both
	Qwen2.5-7B	Qwen2.5-7B-Instruct	Metal
Qwen3	Qwen3-0.6B-Base	Qwen3-0.6B	both	thinking off
	Qwen3-1.7B-Base	Qwen3-1.7B	both	thinking off

Appendix

Table 9: Every checkpoint attempted. Stacks lists the stacks that gave valid output; the Note column gives exceptions by condition. Base checkpoints ran in the completion condition, instruction-tuned checkpoints in the chat and prefill conditions. Stacks: CUDA (RTX 4070 Ti, bfloat16) and Metal (Mac Mini, float16). Model names omit the vendor prefix (Qwen, google, meta-llama, allenai, HuggingFaceTB, mistralai, microsoft).

Instruction Tuning

Instruction tuning adapts pretrained language models to follow natural-language requests using curated or synthesized instruction-response examples.

See all 82 papers

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe
Jul 28, 2026Chaemin Jang, Dongman Lee, Jihee KimLanguage Model Decoding LLM Prompting
School of Computing, KAIST · School of Business and Technology Management, KAIST · Graduate School of Data Science, KAIST
Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
Aug 13, 2026Irina Proskurina, Mayank Kumar, Oyindolapo O. KomolafeConfidence Estimation in Language Models Question Answering
Cohere Labs Community · Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · School of Computer Science Engineering and Technology (SCSET), Bennett University, Greater Noida, India +1
Probabilistic Calibration Is a Trainable Capability in Language Models
May 12, 2026Davide Baldelli, Sruthi Kuriakose, Maryam Hashemzadeh +2LLM Fine-Tuning Controllable Text Generation
Chandar Research Lab · Mila – Quebec AI Institute · LAMA-WeST Lab +3

Jul 28, 2026J/K move · Enter open · S save

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

Chaemin Jang, Dongman Lee, Jihee Kim

School of Computing, KAIST · School of Business and Technology Management, KAIST · Graduate School of Data Science, KAIST

Language Model Decoding LLM Prompting LLM Fine-Tuning User Simulation

Open paper AskPDF arXiv

Silicon sampling uses language models as proxies for human survey respondents, treating each model call as an independent draw from the persona's response distribution. We show this draw does not exist: instruction-tuned models do not sample from distributions, they collapse to a single output. The same persona on the same question returns the same answer on more than half of items in a public-opinion benchmark, and the model's internal probabilities concentrate on a single option. The failure is associated with, and amplified by, instruction-targeted post-training: instruction-tuned models are worse than their own bases in every family we can compare, the gap widens at each successive post-training stage and with the size of the tuning update, and continued pretraining on non-instruction tokens leaves it unchanged. Yet the knowledge survives: the same model that cannot sample from a distribution can describe it accurately in a single call. We call this gap the KNOWS/DOES split. Exploiting the split, a single call that asks the model to describe the response distribution more than halves the error against human survey data compared to persona aggregation. When per-persona outputs are required, we propose Prompt-Perturbed Argyle (PPA), which reduces the same error by 21%, spreading each persona's answers to mirror real population differences at no added cost.

Abstract

Instruction Tuning

Explore similar work

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

Probabilistic Calibration Is a Trainable Capability in Language Models