Training Data Size Sensitivity in Unsupervised Rhyme Recognition

Petr Plecháč; Artjoms Šeļa; Silvie Cinková; Mirella De Sisto; Lara Nugues; Neža Kočnik; Antonina Martynenko; Ben Nagy

cs.CLApr 9, 2026

Training Data Size Sensitivity in Unsupervised Rhyme Recognition

Authors: Petr Plecháč, Artjoms Šeļa, Silvie Cinková, Mirella De Sisto, Lara Nugues, Neža Kočnik, Antonina Martynenko, Ben Nagy,

Organizations: Institute of Czech Literature, Czech Academy of Sciences, Czechia · University of Tartu, Estonia · Charles University, Czechia · Tilburg University, Netherlands ·

Abstract

Rhyme is deceptively intuitive: what is or is not a rhyme is constructed historically, scholars struggle with rhyme classification, and people disagree on whether two words are rhymed or not. This complicates automated rhymed recognition and evaluation, especially in multilingual context. This article investigates how much training data is needed for reliable unsupervised rhyme recognition using RhymeTagger, a language-independent tool that identifies rhymes based on repeating patterns in poetry corpora. We evaluate its performance across seven languages (Czech, German, English, French, Italian, Russian, and Slovene), examining how training size and language differences affect accuracy. To set a realistic performance benchmark, we assess inter-annotator agreement on a manually annotated subset of poems and analyze factors contributing to disagreement in expert annotations: phonetic similarity between rhyming words and their distance from each other in a poem. We also compare RhymeTagger to three large language models using a one-shot learning strategy. Our findings show that, once provided with sufficient training data, RhymeTagger consistently outperforms human agreement, while LLMs lacking phonetic representation significantly struggle with the task.

Figures & tables

$a_{1}:a_{2}$ ,	$a_{1}:a_{3}$ ,	$a_{1}:a_{4}$ ,	$a_{2}:a_{3}$ ,	$a_{2}:a_{4}$ ,	$a_{3}:a_{4}$ ,
$b_{1}:b_{2}$ ,	$b_{1}:b_{3}$ ,	$b_{1}:b_{4}$ ,	$b_{2}:b_{3}$ ,	$b_{2}:b_{4}$ ,	$b_{3}:b_{4}$

Table 1

sample	$F_{1}$ -score
cs	0.90
de	0.88
en	0.88
fr	0.86
it	0.94
ru	0.96

Table 1: Inter-annotator agreements in individual language samples

	Mean	SD	HDI 3%	HDI 97%
phon_dist	-0.487	0.032	-0.544	-0.424
line_dist	-0.842	0.052	-0.943	-0.750
cs	3.356	0.180	3.027	3.690
de	2.237	0.116	2.026	2.459
en	2.338	0.109	2.126	2.534
fr	2.630	0.123	2.402	2.865

Table 2: Logistic regression results by corpus. The mean effect sizes are shown, along with the 94% High Density Interval that represents the ‘credible range’ of the estimated effect.

Figure 1: $F_{1}$ -scores achieved by RhymeTagger against human annotators. 100 models per single model size

sample	GPT-4o		Claude 3.7 Sonnet		DeepSeek-V3		rhymeTagger
sample	annot. 1	annot. 2	annot. 1	annot. 2	annot. 1	annot. 2	rhymeTagger
cs	0.61	0.57	0.67	0.59	0.46	0.48	0.91
de	0.68	0.64	0.53	0.51	0.57	0.60	0.89
en	0.57	0.59	0.63	0.64	0.66	0.67	0.86
fr	0.77	0.81	0.76	0.86	0.62	0.58	0.89
it	0.82	0 .79	0.74	0.75	0.43	0.41	0.97

Table 3: Agreement (

F_{1}

-score) between LLMs (one-shot learning) and human annotators. Highest scores for each language highlighted in bold. The last column gives the value for the largest available rhymeTagger model (higher of the two

F_{1}

averages for given model size)

Unsupervised Learning

Unsupervised learning finds structure, useful representations, or generative models in data without task labels.

See all 57 papers

Zero-shot Large Language Models for Automatic Readability Assessment
Apr 27, 2026Riley Grossman, Yi ChenLLM Prompting
New Jersey Institute of Technology Newark, NJ
LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian
Sep 30, 2026Claudiu Creanga, Liviu P. DinuLLM Evaluation LLM-as-a-Judge
Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania
Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese
Jun 5, 2026Xing Yue, Yongliang Shen, Weiming LuLLM Evaluation
Zhejiang University

Apr 27, 2026J/K move · Enter open · S save

Zero-shot Large Language Models for Automatic Readability Assessment

Riley Grossman, Yi Chen

New Jersey Institute of Technology Newark, NJ

LLM Prompting

Open paper AskPDF arXiv

Unsupervised automatic readability assessment (ARA) methods have important practical and research applications (e.g., ensuring medical or educational materials are suitable for their target audiences). In this paper, we propose a new zero-shot prompting methodology for ARA and present the first comprehensive evaluation of using large language models (LLMs) as an unsupervised ARA method by testing 10 diverse open-source LLMs (e.g., different sizes and developers) on 14 diverse datasets (e.g., different text lengths and languages). Our findings show that our proposed prompting methodology outperforms prior methods on 13 of the 14 datasets. Furthermore, we propose LAURAE, which combines LLM and readability formula scores to improve robustness by capturing both contextual and shallow (e.g., sentence length) features of readability. Our evaluation demonstrates that LAURAE robustly outperforms prior methods across languages, text lengths, and amounts of technical language.

Abstract

Unsupervised Learning

Explore similar work

Zero-shot Large Language Models for Automatic Readability Assessment

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese