cs.CLMar 30, 2026
SaveOn the limited utility of parallel data for learning shared multilingual representations
Organizations: University of Helsinki
Abstract
Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.
Figures & tables
Figure 1: English and Finnish translation sentence pair representations of each model projected to the first two principal components. Finnish sentences are in blue and English sentences in yellow. Each model is exposed to different amounts of parallel data from no parallel data in lm-0 (top row) to 5% parallel data in lm-5 (bottom row).
| Model | English | Finnish | Parallel |
|---|---|---|---|
| lm-0 | 0 | ||
| lm-1 | |||
| lm-2 | |||
| lm-5 |
Table 1: The distributions of the training data in tokens for the four language models.
Figure 2: The average cosine similarities across the layers for the fully trained models and the untrained model. Translation refers to the average cosine similarity between the Finnish-English translation pairs, whereas random refers to the average cosine similarity between random Finnish and English sentence pairs.
Figure 3: The PWCCA scores for the final checkpoints of all the models and the untrained model. Random refers to the scores obtained when shuffling the parallel dataset, thus breaking the possible correlation.
Figure 4: The distribution of the 1000 most language-specific neurons across the layers of each model.
Figure 5: The percentage of neurons on each layer that score lower than or equal to 0.55 average precision in predicting both of the languages.
| Finnish | lm-0 | lm-1 | lm-2 | lm-5 |
|---|---|---|---|---|
| >= 0.75 | 4666 | 4485 | 4538 | 4531 |
| >= 0.9 | 496 | 423 | 398 | 437 |
| >= 0.95 | 134 | 106 | 92 | 108 |
| >= 0.99 | 16 | 19 | 16 | 15 |
| English | lm-0 | lm-1 | lm-2 | lm-5 |
| >= 0.75 | 3448 | 3208 | 3165 | 3201 |
Table 2: The total number of neurons in each model with varying average precision thresholds. Finnish refers to average precision predicting Finnish, whereas English to average precision predicting English. The total number of neurons in each model is 132,096.
| Fluency | Coherence | |||
| lm-0 | ||||
| FI | 3.818 | 0.221 | 2.899 | 0.479 |
| EN | 1.008 | 0.052 | 3.123 | 0.425 |
| 0.09 | 3.713 | 0.266 | 2.324 | 0.636 |
| lm-1 | ||||
| FI | 3.827 | 0.211 | 2.92 | 0.478 |
Table 3: The mean ( ) scores with standard deviations ( ) from the LLM-judge for fluency and coherence for the continuations to the 200 stories. 0.09 refers to the controlled continuations, FI to the Finnish baseline continuations without any control vectors applied, and EN to English baseline continuations.
Figure 6: PWCCA scores computed across layers for early checkpoints of each model.
Figure 7: Difference in mean fluency and coherence scores between the controlled and Finnish baseline completions for checkpoints during training.
Figure 8: The development of the number of neurons given four thresholds for average precisions.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Parameters | |
| Non-embedding parameters | |
| Number of layers | |
| Model dimension | |
| Attention type | Multi-headed attention |
| Number of heads |
Table 4: Hyperparameters for the model architecture used in this study.
| Input | Kia had always had a distaste for mushrooms. But then one day she ordered pasta at a restaurant. The sauce was wonderful, and she asked what was in it. To her surprise, the waiter said it was mostly mushrooms! |
|---|---|
| lm-0 | Kia ei ollut ainoa, joka oli ihastunut tähän ruokaan. Pian muutkin asiakkaat alkoivat kertoa syövänsä sieniruokaa. Sitten Kia päätti Kia wasn’t the only one who was in love with this dish. Soon other customers started telling me they were eating mushroom food. Then Kia decided |
| lm-1 | Kia oli täysin järkyttynyt. Hän ei ollut koskaan syönyt mitään niin hyvää. Mikä ihme on tämä? Mutta siinä vaiheessa hän tajusi, että Kia was completely shocked. She had never eaten anything so good. What the hell is this? But at that moment she realized that |
| lm-2 | Kia ei ollut tästä moksiskaan, vaan ajatteli, että totta kai hän söi tuon. Mutta sitten hän huomasi, että ruoka oli Kia wasn’t at all surprised by this, and thought that of course he ate that. But then he noticed that the food was |
| lm-5 | Kia ei ollut koskaan aiemmin ostanut ravintolasta mitään muuta kuin ruokaa, joten hän ei ollut kovin yllättynyt kuullessaan, että ateria oli tehty sienistä Kia had never bought anything other than food from a restaurant before, so she wasn’t too surprised to hear that the meal was made with mushrooms. |
Table 5: Example text completions from the language control vector experiment. English translation in italics.