Towards cross-cultural study of folksong lyrics with machine translation
Authors: Anna Dvořáková, Anna Aljanaki, Danbinaerin Han, Peter van Kranenburg, Matěj Kratochvíl, Inna Lisniak, Zdeněk Vejvoda, Jan Hajič
Organizations: Charles University Czech Republic · University of Music and Performing Arts Graz Austria · Graduate School of Culture Technology KAIST Republic of Korea · Utrecht University Netherlands · Institute of Ethnology Czech Academy of Sciences Czech Republic · Estonian Literary Museum; M. T. Rylsky Institute of Art Studies, Folkloristics and Ethnology NAS of Ukraine
Music is universally present in human societies. Ethnomusicologists have long been documenting the diverse expressions of human musicality, and comparative musicology has recently brought several studies of folksong to a more global scale. Such cross-cultural research has not been conducted on lyrics: the language barrier has so far prevented work with multi-lingual data. However, Natural Language Processing (NLP) technologies have reached a stage where this language barrier may no longer be prohibitive. Combining folksong lyrics corpora across five languages, we machine-translate them to a pivot language with a pre-trained neural topic model, and we examine the relationship between content and social function within each language, and across languages for wedding songs. As expected, human evaluation of translation results shows that non-Indo-European languages suffer from overall worse translation quality. Experiments with topic models then indicate that the content of lyrics is at best partially related to the social function of folksongs across all languages. These experiments are just first steps into cross-cultural folk musics lyrics analysis; however, they do indicate that a previously unobserved web of cross-cultural relationships beyond ethnomusicological typologies may be uncovered through the study of what people sing across the world's diverse folk musics.
Figures & tables
Language
No. of songs
No. of wedding songs
No. of songs monoling. exps.
Czech
3320
211
2829
Dutch
3041
149
2612
Estonian
4515
100
3553
Korean
15861
480
7767
Ukrainian
1764
257
1585
Table 1: Number of songs with lyrics and number of wedding songs (Korean: marriage songs) in each corpus and number of songs that were used in the experiments from each corpus. The high reduction on the Korean part was caused by a large number of missing typology labels needed for the experiments
Language
# words evaluated
Mean importance
Mean correctness
Mean overall
Czech
764
0.84
0.97
3.67
Dutch
1396
0.48
0.93
4.18
Estonian
903
0.79
0.76
2.94
Korean
584
0.95
0.79
3.00
Ukrainian
1144
0.64
0.90
3.32
Table 2: MT manual evaluation results.
Corpus
Avg. weighted F1 score
Avg. # PCA components
no. effective training songs
Czech
0.26
411
1685
Dutch
0.53
230
499
Estonian
0.51
461
2415
Korean
0.63
488
3552
Ukrainian
0.52
293
1040
Weddings
0.63
216
500
Table 3: Classification results on monolingual corpora and on wedding songs (all languages, and the 3 well-translated ones): avg. weighted F1 score, and avg. number of PCA components explaining 95 % of the embeddings variance, and the number of training songs after subsampling.
Figure 1: Classification results: row-normalised confusion matrices for top 10 labels in each language, and on the languages for wedding songs. We report recall because we care most about the question: “Is this label hard to identify?”.
Cross-lingual transfer is central to low-resource machine translation, but its behavior within closely related language families remains insufficiently characterized. We study transfer among five Turkic languages; Turkish, Azerbaijani, Uzbek, Kazakh, and Kyrgyz; using pairwise transfer matrices. In this setting, each model is fine-tuned with one transfer source and evaluated on a different transfer target while the translation target remains the same. Across mT5 experiments, we find that transfer is strongest between closely related Turkic pairs, especially Turkish-Azerbaijani and Kazakh-Kyrgyz. We also show that transfer direction matters, and that the same transfer source-transfer target pair can behave differently when the translation target changes. Latinization improves BLEU and chrF in several script-mismatched settings, but its effect is not uniform across metrics. Additional analyses show that transfer sources are mostly stable across different datasets and model settings.
Omer Burak Cinar, Mehmet Mert Dalkilic, Cagri Toraman
Middle East Technical University · Computer Engineering Department
Cross-lingual transfer describes how knowledge in a source language benefits a target language. Measuring it quantitatively requires broad multilingual pre-training, as prior work has done with cross-lingual transfer matrices. We ask whether transfer is predictable from freely available typological features, and whether the prominence of high-resource source languages reflects typology or data quality and quantity. We show that typological databases contain cheap and dense signals about cross-lingual transfer. Our typology-only random forest on a 24-language prior-work transfer matrix scores leave-one-language-out ρ=0.705 and R2=0.49, beating a non-typological control at ρ=0.62, which verifies the ability of typology-only predictions to reconstruct costly measured cross-lingual transfer. The signal survives leave-one-script-out and leave-one-family-out protocols, so script and family confounding do not explain the effect. By decomposing the transfer into a typology term and a resource-and-script bias term, we find the best-source ranking sensitive to this bias. In contrast, typology is not affected by this bias, which makes it a zero-compute screening tool that replaces hundreds of training runs with a model fit. Our code is available here.
Dalton Raphael Harmsen, Swier Garst, Thomas van Osch +2
AMOR/e Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · SURF, Amsterdam, The Netherlands
Moral language is subtle and culturally variable, making it difficult to translate faithfully across languages. Idiomatic expressions, slang, and cultural references introduce hard-to-avoid translation artifacts. Yet automated moral values classification depends on language-specific annotated corpora that exist almost exclusively in English. We investigate whether LLM-based translation can bridge this gap, taking Polish as a test case. Using ~50k morally-annotated social media posts from a diverse range of topics, we apply a principled four-method validation pipeline: LaBSE cross-lingual embedding similarity, Centered Kernel Alignment (CKA), LLM-as-judge evaluation, and deep learning classifier parity tests. We show that despite shortcomings in handling slang, vulgarity, and culturally-loaded expressions, direct translation preserves subtle moral cues well enough to be harvested by cross-lingual machine learning - with a mean cosine similarity of 0.89 and classification accuracy gaps of 0.01-0.02 AUROC across foundations. These results demonstrate that machine translation is a practical and cost-effective path to moral research in languages currently under-resourced in this domain. We demonstrate this for Polish as a representative Slavic language, with expected generalization to related languages.
Maciej Skorski
University of Luxembourg Faculty of Science, Technology and Medicine Luxembourg, Luxembourg · University of Warsaw Institute of Informatics Warsaw, Poland