Forty Questions: Identifying a Language Model from One-Word Answers

Tapan Parikh

cs.CLOct 20, 2026

Forty Questions: Identifying a Language Model from One-Word Answers

Authors: Tapan Parikh

Organizations: Cornell Tech

Abstract

A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet". A model's profile records how often it gives each answer to each question. We attribute new answers to the model most likely to have given them. On a panel of 94 models, 40 answers identify the exact model 92.5% of the time, against 1.1% by chance. Profiles built through OpenRouter identified all ten models served by DeepInfra under blinded labels. A system prompt lowers accuracy by 6 to 16 points. We applied the method to 96 endpoints of 13 open models on OpenRouter, and 92 identify as the model they claim. Of the other four, one matches the claimed model with reasoning turned off. One serves Gemma 4 under the name Llama 4 Maverick: it names itself Gemma 4, and it is billed Gemma 4's token counts on every message. Two remain unresolved. Billing records also show context added to first-party models. Grok 4.7 is billed 1,247 input tokens for a one-line question that costs GPT-4o only 12 tokens, and five models state the current date without being billed for it. The questions come from a public census of model defaults, and the method costs cents per endpoint.

Figures & tables

	one answer each		two answers each
questions	exact	family	exact	family
5	28.1%	39.9%	41.5%	53.0%
10	54.5%	63.6%	68.0%	76.2%
20	78.2%	84.9%	90.7%	94.1%
40	92.5%	96.0%	96.9%	99.1%
96	98.4%	100.0%	98.7%	100.0%

Table 1: Identification among 94 models by number of questions and answers per question. Chance is 1.1%.

questions (two answers each)	selected	random
5	62.0%	38.8%
10	81.1%	67.5%
15	90.7%	80.6%
20	95.8%	87.9%
25	97.1%	92.7%

Table 2: Selected against random question sets, mean of the two halves. Both halves’ choices are in Appendix F .

condition	10 $\times$ 2	20 $\times$ 2	31 $\times$ 2	31 $\times$ 4
no system prompt, a later day	66%	90%	98%	100%
“You are a helpful assistant.”	60%	82%	89%	94%
“You are Ava, a friendly assistant.”	56%	74%	86%	91%

Table 3: Identification of 14 models under a system prompt, among all 94 profiles (20 draws per cell).

claimed model	declared precision	identified as	rank of claim	$q$
Llama 4 Maverick	fp8	Gemma 4 31B	91	0.006
Mistral Small 3.2	unstated	Mistral Small 2603	5	0.006
Kimi K2.5	unstated	Kimi K2	2	0.006
DeepSeek V4 Flash	fp4	Qwen 3.8 27B	4	0.048

Table 4: The four endpoints that do not identify as claimed. Rank is the claimed model’s rank among 94 profiles.

model	billed input tokens
GPT-4o (reference)	12
Grok 4.7	1,247
Palmyra X5	243
Grok 4.5 and 4.6	211
MiniMax M3	181
Kimi K3	90 to 93

Table 5: Billed input tokens for “What is today’s date?”, no system prompt.

		four days later
claimed model	audit	plain	our system prompt	longer message
Llama 4 Maverick	$+3$	$+3$	$+4$	$+3$
Mistral Small 3.2	$+12$	$+12$	$+12$	$+12$
Kimi K2.5	$+19$	$+19$	$+1$	$+19$
DeepSeek V4 Flash	$+79$	$+79$	$+79$	$+79$

Table 6: Billed input tokens above the peers’ count, per mismatch.

claimed model	audit	later day	caller’s system prompt	surplus
Llama 4 Maverick	Gemma 4 31B (91)	Gemma 4 31B (91)	Gemma 4 31B (91)	$+4$
Mistral Small 3.2	Mistral Small 2603 (5)	Mixtral 8x22B (3)	Mistral Small 2603 (5)	$+12$
Kimi K2.5	Kimi K2 (2)	Kimi K2 (2)	Kimi K2 (2)	$+1$
DeepSeek V4 Flash	Qwen 3.8 27B (4)	Grok 4.5 (7)	Grok 4.5 (9)	$+79$

Table 7: Identification of the four mismatches across runs, with the claimed model’s rank in parentheses. Surplus is billed input tokens above the peers under the caller’s system prompt.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

model	family	collected
claude-fable-5	claude	07-04 to 09-30
claude-fable-5.1	claude	09-03 to 09-30
claude-haiku-4.5	claude	07-03 to 09-30
claude-opus-4.1	claude	09-25 to 09-30
claude-opus-4.5	claude	09-25 to 09-30
claude-opus-4.6	claude	09-25 to 09-30

Appendix

Table 8

	chosen on samples 0–3		chosen on samples 4–7
$k$	question	accuracy	question	accuracy
1	mammal	15%	building	11%
2	sound	34%	reptile	28%
3	bird	47%	poet	50%
4	instrument	51%	landmark	56%
5	reptile	61%	smell	63%

Appendix

Table 9

endpoint	precision	price	identified as	rank	margin	pinned	$q$
deepseek-v3.2/A	fp8	1.11	deepseek-v3.2	1	+1.63	1	0.041
deepseek-v3.2/B	fp8	0.38	deepseek-v3.2	1	+1.16	1	0.292
deepseek-v3.2/C	fp8	0.42	deepseek-v3.2	1	+0.56	1	1.000
deepseek-v3.2/D	fp4	0.38	deepseek-v3.2	1	+1.27	1	0.345
deepseek-v3.2/E	unstated	0.96	deepseek-v3.2	1	+1.39	1	0.557
deepseek-v3.2/F	unstated	1.50	deepseek-v3.2	1	+1.51	1	0.006

Appendix

Table 10

Language Model Fingerprinting

Language model fingerprinting identifies and verifies models or their lineage from distinctive outputs, internal representations, or embedded signals.

See all 81 papers

The One-Word Census: Answer-Choice Conformity Across 105 Language Models
Jul 14, 2026Tapan ParikhLLM Evaluation Language Model Generation Evaluation
Cornell Tech
System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer
Sep 23, 2026Dmitrij ŻatuchinLLM Auditing Language Model Fingerprinting
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia.
Structured Output Collapses Answer Diversity Across 44 Language Models
Jul 20, 2026Tapan ParikhLLM Evaluation Structured Prompting
Cornell Tech

Jul 14, 2026J/K move · Enter open · S save

The One-Word Census: Answer-Choice Conformity Across 105 Language Models

Tapan Parikh

Cornell Tech

LLM Evaluation Language Model Generation Evaluation Text Generation Output Diversity in Language Models

Open paper AskPDF arXiv

When a language model must choose one answer from a large space of equally valid options, which answer does it choose, and how often is it the answer every other model chooses? Asked to "pick a word," 105 language models from more than twenty labs chose serendipity 46% of the time. We measure this convergence, and each model's share in it, with 96 single-turn prompts that each name a category with many valid one-word answers ("Name a tree."), asked eight times per model and scored by exact match, with no embeddings and no judge. A model's answer-choice surprisal is the average -log2 probability of its answers under the pooled answers of all other models. In 28 of 96 categories a single answer takes at least 80% of all answers. The concentration does not depend on small or persona-tuned models: the 87 major-lab models are at least as concentrated as the full field. Lightly post-trained and persona-tuned models are the most divergent; heavily post-trained assistants from the major labs are the most conformist. Models that avoid the modal answer mostly land on the same runner-up. Within the major providers' lineages, release order shows no panel-wide trend once model tier is controlled; GPT, Gemini, Grok and Qwen become more conformist across releases, and Claude's generation-5 releases reverse. On open checkpoints of three post-training pipelines, supervised fine-tuning is the largest step toward the field's answers. Against human category-production norms, the field is more concentrated than people in 18 of 20 shared categories. All prompts, transcripts, and code are public.

Abstract

Language Model Fingerprinting

Explore similar work

The One-Word Census: Answer-Choice Conformity Across 105 Language Models

System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Structured Output Collapses Answer Diversity Across 44 Language Models