Training Data Size Sensitivity in Unsupervised Rhyme Recognition
Authors: Petr Plecháč, Artjoms Šeļa, Silvie Cinková, Mirella De Sisto, Lara Nugues, Neža Kočnik, Antonina Martynenko, Ben Nagy, Luca Giovannini, Robert Kolár
Organizations: Institute of Czech Literature, Czech Academy of Sciences, Czechia · University of Tartu, Estonia · Charles University, Czechia · Tilburg University, Netherlands · University of Basel, Switzerland · University of Ljubljana, Slovenia · Institute of Polish Language, Polish Academy of Sciences, Poland · University of Potsdam, Germany
Rhyme is deceptively intuitive: what is or is not a rhyme is constructed historically, scholars struggle with rhyme classification, and people disagree on whether two words are rhymed or not. This complicates automated rhymed recognition and evaluation, especially in multilingual context. This article investigates how much training data is needed for reliable unsupervised rhyme recognition using RhymeTagger, a language-independent tool that identifies rhymes based on repeating patterns in poetry corpora. We evaluate its performance across seven languages (Czech, German, English, French, Italian, Russian, and Slovene), examining how training size and language differences affect accuracy. To set a realistic performance benchmark, we assess inter-annotator agreement on a manually annotated subset of poems and analyze factors contributing to disagreement in expert annotations: phonetic similarity between rhyming words and their distance from each other in a poem. We also compare RhymeTagger to three large language models using a one-shot learning strategy. Our findings show that, once provided with sufficient training data, RhymeTagger consistently outperforms human agreement, while LLMs lacking phonetic representation significantly struggle with the task.
Figures & tables
a1:a2 ,
a1:a3 ,
a1:a4 ,
a2:a3 ,
a2:a4 ,
a3:a4 ,
b1:b2 ,
b1:b3 ,
b1:b4 ,
b2:b3 ,
b2:b4 ,
b3:b4
Table 1
sample
F1 -score
cs
0.90
de
0.88
en
0.88
fr
0.86
it
0.94
ru
0.96
Table 1: Inter-annotator agreements in individual language samples
Mean
SD
HDI 3%
HDI 97%
phon_dist
-0.487
0.032
-0.544
-0.424
line_dist
-0.842
0.052
-0.943
-0.750
cs
3.356
0.180
3.027
3.690
de
2.237
0.116
2.026
2.459
en
2.338
0.109
2.126
2.534
fr
2.630
0.123
2.402
2.865
Table 2: Logistic regression results by corpus. The mean effect sizes are shown, along with the 94% High Density Interval that represents the ‘credible range’ of the estimated effect.
Figure 1: F1 -scores achieved by RhymeTagger against human annotators. 100 models per single model size
sample
GPT-4o
Claude 3.7 Sonnet
DeepSeek-V3
rhymeTagger
annot. 1
annot. 2
annot. 1
annot. 2
annot. 1
annot. 2
cs
0.61
0.57
0.67
0.59
0.46
0.48
0.91
de
0.68
0.64
0.53
0.51
0.57
0.60
0.89
en
0.57
0.59
0.63
0.64
0.66
0.67
0.86
fr
0.77
0.81
0.76
0.86
0.62
0.58
0.89
it
0.82
0 .79
0.74
0.75
0.43
0.41
0.97
Table 3: Agreement ( F1 -score) between LLMs (one-shot learning) and human annotators. Highest scores for each language highlighted in bold. The last column gives the value for the largest available rhymeTagger model (higher of the two F1 averages for given model size)
Unsupervised automatic readability assessment (ARA) methods have important practical and research applications (e.g., ensuring medical or educational materials are suitable for their target audiences). In this paper, we propose a new zero-shot prompting methodology for ARA and present the first comprehensive evaluation of using large language models (LLMs) as an unsupervised ARA method by testing 10 diverse open-source LLMs (e.g., different sizes and developers) on 14 diverse datasets (e.g., different text lengths and languages). Our findings show that our proposed prompting methodology outperforms prior methods on 13 of the 14 datasets. Furthermore, we propose LAURAE, which combines LLM and readability formula scores to improve robustness by capturing both contextual and shallow (e.g., sentence length) features of readability. Our evaluation demonstrates that LAURAE robustly outperforms prior methods across languages, text lengths, and amounts of technical language.
Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.
Claudiu Creanga, Liviu P. Dinu
Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania
Language is a vehicle for thought, intricately tied to sounds, symbols, and meaning. However, most large language model (LLM) research focuses on meaning (semantics) and symbols (spelling) while largely overlooking sounds. Existing benchmarks on LLMs' phonological abilities are either solvable through rote memorization or intertwined with other abilities, making them inadequate to measure LLMs' genuine ability in phonological understanding. Here, we present Phun-Bench, a purpose-built Chinese benchmark with diverse tasks and settings across three dimensions (Homophony, Rhyme, and Phonetic Similarity), designed to systematically evaluate LLMs' phonological understanding. Our results show that while LLMs excel at recalling correct pronunciations, they generally struggle to leverage phonological knowledge in the flexible and intuitive way that human speakers do. Moreover, through detailed analyses, we propose a hypothesis regarding the underlying mechanism of LLMs' phonological understanding and "perception", highlighting an underexplored frontier for future research.