Paralinguistic speech tasks are often considered relatively language-agnostic, as they rely on extralinguistic acoustic cues rather than lexical content. However, prior studies report performance degradation under cross-lingual conditions, indicating non-negligible language dependence. Still, these studies typically focus on isolated language pairs or task-specific settings, limiting comparability and preventing a systematic assessment of task-level language dependence. We introduce the Cross-Lingual Transfer Matrix (CLTM), a systematic method to quantify cross-lingual interactions between pairs of languages within a given task. We apply the CLTM to two paralinguistic tasks, gender identification and speaker verification, using a multilingual HuBERT-based encoder, to analyze how donor-language data affects target-language performance during fine-tuning. Our results reveal distinct transfer patterns across tasks and languages, reflecting systematic, language-dependent effects.
Figures & tables
Figure 1 : Typical learning curve for a single language, showing the dynamic interval and derivative regimes.
Figure 2 : Learning curves for both tasks, showing performance as a function of training samples for a representative subset of languages. The task-specific dynamic interval [N,2N] used to compute the CLTM is highlighted.
Figure 3 : Architecture for gender recognition.
Figure 4 : Speaker verification pipeline: SID training via a classification head, then embeddings are L2-normalized and compared with cosine similarity for verification.
Figure 5 : Reduced CLTMs (16 representative languages) for gender recognition and speaker verification. Colors show how much adding donor-language data affects performance on a target language compared to adding the same amount of target-language data.
Metric
Gender Recognition
Speaker Verification
RFD1
0.162
2.970
Asymrel
0.175
1.084
prop+
99.97%
8.93%
reciprocity+
99.93%
1.69%
cosrows
0.990
0.615
intra-family+
4.98%
41.68%
Table 1 : Aggregate CLTM diagnostics computed on the full 44×44 matrices for gender recognition and speaker verification.
Language pair
dcent (Euclidean)
CLTM values
Russian–Belarusian
1.37
1.35 / 1.46
Kurmanji–Central Kurdish
1.53
1.22 / 0.91
German–Portuguese
2.06
-2.02 / -0.32
Galician–Central Kurdish
2.39
-1.09 / -1.13
Table 2 : Euclidean centroid distances between language speaker embeddings and CLTM values for selected SV pairs.