Typological Feature Induction as a Benchmark for Inductive Metalinguistic Reasoning

Chen Zhang; Seth Aycock; Yusuke Miyao

cs.CLNov 3, 2026

Typological Feature Induction as a Benchmark for Inductive Metalinguistic Reasoning

Authors: Chen Zhang, Seth Aycock, Yusuke Miyao

Organizations: NII LLMC · University of Amsterdam · The University of Tokyo

Abstract

Existing metalinguistic reasoning benchmarks require both inference of rules and their application, making failures hard to diagnose. We introduce typological feature induction (TFI) which targets the former step. Given sentences in an unnamed language paired with translations in a known pivot language, models must predict a typological feature of the unnamed language. From Grambank and multi-parallel corpora, we automatically build evaluation questions over 195 languages. We find that LLMs identify 59--80% of these languages from only 16 sentence pairs; and misaligning the translations barely affects their accuracy, suggesting reliance on prior knowledge of the language. Enciphering text to prevent recognition increases the corpus' utility, but mainly for features where target and pivot languages agree. On languages which models cannot identify, neither open model outperforms a majority-value baseline. Since induction from raw text aligns with field linguists' use-case, this work indicates the models tested are not yet ready for this role. TFI offers a scalable path to track progress in this direction.

Figures & tables

Figure 1: A TFI item for Hausa, with an example sentence pair, and reasoning excerpts from Qwen3.6. In parallel , the model names the language and answers correctly by recall. With the target side enciphered, it aligns three months word by word with watanni uku (‘months three’) and answers with the English order.

	Description
parallel	$n$ aligned pairs
mono	$n$ target sentences only
misaligned	$n$ pairs, target side shuffled
parallel+name	$n$ aligned pairs $+$ language name
cipher-parallel	$n$ pairs, target side enciphered
cipher-mono	mono on enciphered text

Table 1: Experimental settings and baselines.

Figure 2: Accuracy of TFI on the core evaluation set as the number of sentence pairs

n

grows. Error bars show the standard deviation over three corpus seeds for the open models; GPT-5.6 Luna was run on one.

(a) Language naming. Qwen3.6, parallel ,

n{=}128

; Estonian, GB075 (postpositions); correct.

“Based on the provided corpus, the target language ( Estonian ) exhibits a strong preference for postpositions […] In Estonian , many spatial and temporal relationships are expressed using nouns in the genitive case followed by a postpositional noun […]”

In cipher-parallel the same model names no language and reaches the same answer after 18,847 characters of analysis (2,620 in plaintext).

(b) Corpus citation. Gemma 4, misaligned ,

n{=}128

; Mískito, GB044 (bound plural marking); correct.

“Reviewing the corpus: […] 5. ‘issues’

\rightarrow

niks munia . […]”

The target line shown with the issues sentence translates a recipe ( Diara mangki niks munia dukia nani: 1 kg tawa, … ; ‘The ingredients needed include: 1 kg potatoes, …’), so the gloss is invented. In parallel , Gemma 4 glosses issues as aisan nani .

Table 2: Two reasoning patterns: (a) Naming the language and answering from recall. (b) Citing corpus pairs as evidence, including pairs that do not match.

		$n{=}16$	$n{=}64$	$n{=}256$
before cipher (all, 195)	Gemma 4	0.660	0.662	0.658
	Qwen3.6	0.593	0.581	0.576
	GPT-5.6 Luna	0.800	0.809	0.805
before cipher (Latin, 127)	Gemma 4	0.643	0.646	0.640
	Qwen3.6	0.583	0.562	0.559
	GPT-5.6 Luna	0.795	0.811	0.806

Table 3: Language identification accuracy, i.e. the proportion of languages each model names correctly from

n

unlabelled sentence pairs.

Figure 3: Proportions of outputs that name a specific language, in parallel and cipher-parallel , on the Latin-script evaluation set.

Figure 4: Accuracy of TFI on the 127 Latin-script languages with the target side enciphered. Error bars show the standard deviation over three corpus seeds.

Figure 5: Change in accuracy from adding different pivot sides to an enciphered corpus (

n{=}128

; cipher-parallel

-

cipher-mono ).

Figure 6: Distribution of answers on Grambank features, for items where the target’s value differs from the pivot’s, without the pivot side ( cipher-mono ) and with it ( cipher-parallel ).

Figure 7: Accuracy on the extended set as the number of sentence pairs

n

grows. The top panel shows all 3,775 items, the bottom panel the 952 items whose answer is not the most frequent value in Grambank. Error bars show the standard deviation over three corpus seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	Lang.	Feat.	Items	Used for
Core	195	22	4,084	§ 4.1
Latin-Script	127	22	2,657	§ 4.2 –§ 4.4
Extended	24	195	3,775	§ 4.5

Appendix

Table 4: Statistics of the three evaluation sets. Feat. is the number of distinct Grambank features asked about. The extended set asks about every feature coded for each of its languages.

		$n{=}128$			$n{=}512$			$n{=}1024$
		all	answered	unans.	all	answered	unans.	all	answered	unans.
Qwen3.6-27B	parallel	0.788	0.789	5	0.777	0.786	47	0.775	0.784	51
	mono	0.790	0.792	11	0.749	0.777	146	0.734	0.770	191
	misaligned	0.787	0.792	29	0.749	0.786	193	0.746	0.784	198
Gemma 4 31B	parallel	0.807	0.807	0	0.810	0.810	1	0.810	0.812	7
	mono	0.803	0.803	1	0.804	0.805	3	0.800	0.801	8

Appendix

Table 5: Accuracy on all 4,084 items (unanswered counted as wrong, as in the main text), accuracy on answered items only, and the number of unanswered items per run. Mean over three corpus seeds.

		Named correctly	Not named
Gemma 4	$n{=}16$	.784 $\rightarrow$ .620	.760 $\rightarrow$ .612
Gemma 4	$n{=}128$	.821 $\rightarrow$ .676	.761 $\rightarrow$ .623
Qwen3.6	$n{=}16$	.793 $\rightarrow$ .588	.715 $\rightarrow$ .642
Qwen3.6	$n{=}128$	.810 $\rightarrow$ .646	.724 $\rightarrow$ .638

Appendix

Table 6: Accuracy in parallel

\rightarrow

cipher-parallel on the Latin-script items, split by whether the plaintext output names the target correctly (first corpus seed).

Figure 8: Change in accuracy when the pivot shares the target’s value for the feature, with a fixed effect for each target and feature, so that pivots are compared on the same question (

n{=}128

, 15 pivots). Bars are 95% confidence intervals with standard errors clustered by target and pivot.

	Gemma 4		Qwen3.6
	Unid.	Others	Unid.	Others
majority	.676	.603	.676	.603
parallel	.695	.798	.701	.775
cipher-parallel	.614	.646	.626	.630

Appendix

Table 7: Accuracy on the Latin-script core items at

n{=}128

, mean over three corpus seeds, split by whether the language is unidentified, i.e. belongs to the extended set (15 languages, 318 items), or not (112 languages, 2,339 items).

Language	ISO	Glottocode	Family	Script	Src.	Core	L	E
Achinese	ace	achi1257	Austronesian	Latn	F	22	✓
Afar	aar	afar1241	Afro-Asiatic	Latn	B	21	✓
Aguaruna	agr	agua1253	Chicham	Latn	B	20	✓
Amharic	amh	amha1245	Afro-Asiatic	Ethi	F+B	21
Amis	ami	amis1246	Austronesian	Latn	B	22	✓
Arhuaco	arh	arhu1242	Chibchan	Latn	B	22	✓	✓

Appendix

Table 8: All 195 target languages, with family from Grambank. Src. is the source corpus (F: FLORES+, B: BOUQuET); Core is the number of core-set items; L and E mark the Latin-script and extended sets.

Language	ISO	Glottocode	Family	Script	Src.	Core	L	E
Hassaniyya	mey	hass1238	Afro-Asiatic	Latn	B	22		✓
Hausa	hau	haus1257	Afro-Asiatic	Latn	F+B	19	✓
Hebrew	heb	hebr1245	Afro-Asiatic	Hebr	F+B	22
Hehe	heh	hehe1240	Atlantic-Congo	Latn	B	21	✓	✓
Highland Puebla Nahuatl	azz	high1278	Uto-Aztecan	Latn	B	14	✓
Hindi	hin	hind1269	Indo-European	Deva	F+B	22

Appendix

Table 9: All 195 target languages (continued).

Language	ISO	Glottocode	Family	Script	Src.	Core	L	E
Nuer	nus	nuer1246	Nilotic	Latn	F+B	20	✓	✓
Occitan (post 1500)	oci	occi1239	Indo-European	Latn	F	22	✓
Odia	ory	oriy1255	Indo-European	Orya	F+B	22
Pangasinan	pag	pang1290	Austronesian	Latn	F	22	✓
Panjabi	pan	panj1256	Indo-European	Guru	F+B	22
Paraguayan Guaraní	gug	para1311	Tupian	Latn	F+B	10	✓

Appendix

Table 10: All 195 target languages (continued).

Group	ID	Grambank question	Values	Items	Gold
Word order	GB024	What is the order of numeral and noun in the NP?	[1] Num–N; [2] N–Num; [3] both	186	117/49/20
	GB025	What is the order of adnominal demonstrative and noun?	[1] Dem–N; [2] N–Dem; [3] both	186	102/58/26
	GB065	What is the pragmatically unmarked order of adnominal possessor noun and possessed noun?	[1] Possessor–Possessed; [2] Possessed–Possessor; [3] both	190	75/75/40
	GB130	What is the pragmatically unmarked order of S and V in intransitive clauses?	[1] SV; [2] VS; [3] both	194	157/26/11
	GB131	Is a pragmatically unmarked constituent order verb-initial for transitive clauses?	[0] absent; [1] present	191	159/32
	GB132	Is a pragmatically unmarked constituent order verb-medial for transitive clauses?	[0] absent; [1] present	191	90/101

Appendix

Table 11: The 22 core features. Items is the number of the 195 languages Grambank codes for the feature; Gold is how many of them have each value, in the order listed.

Inductive Reasoning in Language Models

See all 16 papers

Typological Feature Prediction with Large Language Models: An In-Context Learning Approach
Sep 3, 2026Qianwen Wang, York Hay Ng, Aditya Khan +1In-Context Learning Linguistic Typology
University of Toronto, Canada · Columbia · Ontario Tech University, Canada
Zero-Compute Cross-Lingual Transferability Estimation Using Typological Feature Proxies
Sep 30, 2026Dalton Raphael Harmsen, Swier Garst, Thomas van Osch +2Task Transferability Estimation Linguistic Typology
AMOR/e Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · SURF, Amsterdam, The Netherlands
LLM Agents as Computational Typologists
Sep 7, 2026Changbing Yang, Christopher Hammerly, Freda Shi +1LLM Agents Linguistic Typology
University of British Columbia · University of Waterloo · Vector Institute

Sep 3, 2026J/K move · Enter open · S save

Typological Feature Prediction with Large Language Models: An In-Context Learning Approach

Qianwen Wang, York Hay Ng, Aditya Khan, En-Shiun Annie Lee

University of Toronto, Canada · Columbia · Ontario Tech University, Canada

In-Context Learning Linguistic Typology

Open paper AskPDF arXiv

Typological features are widely used in multilingual NLP, and the prediction of such features holds downstream utility. However, existing methods to predict missing values lack interpretable justifications for predictions, while their performance across resource levels and feature types remains underexplored. Given LLMs' abilities in meta-linguistic reasoning and in providing rationales, we investigate LLMs' performance in typological feature prediction via an in-context learning approach with linguistic data from URIEL+ and Glottolog. We find that zero-shot prompting is insufficient, but when given phylogenetic and geographic neighbour evidence, LLMs substantially outperform all baselines without disadvantaging low-resource languages. We further find that most LLM rationales are consistent with the provided evidence, offering a step toward explainable typological feature prediction.

Abstract

Inductive Reasoning in Language Models

Explore similar work

Typological Feature Prediction with Large Language Models: An In-Context Learning Approach

Zero-Compute Cross-Lingual Transferability Estimation Using Typological Feature Proxies

LLM Agents as Computational Typologists