Do neural machine translation models learn language-universal conceptual representations, or do they merely cluster languages by surface similarity? We investigate this question by probing the representation geometry of Meta's NLLB-200, a 200-language encoder-decoder Transformer, through six experiments that bridge NLP interpretability with cognitive science theories of multilingual lexical organization. Using the Swadesh core vocabulary list embedded across 135 languages, we find that the model's embedding distances significantly correlate with phylogenetic distances from the Automated Similarity Judgment Program (ρ=0.13, p=0.020), demonstrating that NLLB-200 has implicitly learned the genealogical structure of human languages. We show that frequently colexified concept pairs from the CLICS database exhibit significantly higher embedding similarity than non-colexified pairs (U=42656, p=1.33×10−11, d=0.96), indicating that the model has internalized universal conceptual associations. Per-language mean-centering of embeddings improves the between-concept to within-concept distance ratio by a factor of 1.19, providing geometric evidence for a language-neutral conceptual store analogous to the anterior temporal lobe hub identified in bilingual neuroimaging. Semantic offset vectors between fundamental concept pairs (e.g., man to woman, big to small) show high cross-lingual consistency (mean cosine = 0.84), suggesting that second-order relational structure is preserved across typologically diverse languages. We release InterpretCognates, an open-source interactive toolkit for exploring these phenomena, alongside a fully reproducible analysis pipeline.
Figures & tables
Figure 1: Embedding geometry for the concept “water” across 29 languages. (a) 3D PCA projection colored by language family shows tight clustering despite orthographic diversity. (b) Pairwise similarity heatmap reveals that same-family languages (e.g., Romance, Slavic) cluster, but cross-family similarity remains high ( >0.93 for most pairs).
Figure 2: Swadesh convergence ranking vs. surface-form similarity. (a) Orthographic similarity (normalized Levenshtein distance on Latin-script word forms, R2=0.012 ) and (b) phonological similarity (after crude phonetic normalization, R2=0.004 ) plotted against embedding convergence (isotropy-corrected). Points are colored by semantic category. Neither measure predicts convergence: over 98% of the convergence signal is attributable to semantic rather than surface-form factors. Concepts in the upper-left quadrant converge strongly in embedding space despite low surface-form similarity—the strongest candidates for genuine conceptual universals.
Figure 3: Convergence by semantic category. Violin plots with individual data points for each Swadesh semantic category (isotropy-corrected). The dashed line marks the overall mean. Nature and People categories converge most strongly; Pronouns converge least, consistent with their high cross-linguistic grammaticalization variability.
Figure 4: Per-concept convergence scores grouped by semantic category (sorted by category mean, highest at top). Each dot is one Swadesh concept; the dashed line marks the overall mean. Shaded bands delineate category boundaries, enabling identification of within-category outliers such as polysemous items that depress their category’s aggregate score.
Figure 5: Isotropy correction validation. (a) Scatter of raw vs. corrected convergence scores (Spearman ρ=0.990 ), colored by semantic category; points below the diagonal indicate concepts whose convergence decreased after correction. (b) Top-10 and bottom-10 concepts under each regime; the overlap is substantial, with a few concepts reranked. (c) Sensitivity of the convergence ranking to the ABTT hyperparameter k : all pairwise Spearman correlations with the reference k=3 ranking span 0.98–1.00, confirming robustness.
Figure 6: Carrier sentence robustness analysis. (a) Scatter of contextualized vs. decontextualized convergence scores (Spearman ρ=0.867 ). Points near the identity line indicate concepts whose convergence is insensitive to the carrier sentence. (b) Slopegraph of the top-20 concepts under each condition, showing minimal reranking.
Figure 7: Layer-wise emergence of language-universal semantic structure across 12 encoder layers (computed on a 39-language subset). (a) Mean Swadesh convergence increases monotonically, with a sharp rise around layer 1, paralleling the “NLP pipeline” effect. (b) The Conceptual Store Metric (both raw and mean-centered ratios) shows a similar trajectory, with the centered ratio exhibiting a phase transition at layer 6. (c) Per-concept convergence heatmap reveals that concrete, perceptually grounded concepts (top rows) achieve high cross-lingual convergence earlier than abstract or polysemous concepts (bottom rows).
Figure 8: Phylogenetic structure in the NLLB-200 embedding space. Left: heatmap of pairwise embedding coherence (normalized to [0,1] ; white = 0 , red = 1 ) between 88 languages, ordered by hierarchical clustering. Right: dendrogram derived from the embedding distance matrix. Major language families (e.g., Indo-European, Austronesian, Niger-Congo) form recognizable clusters.
Figure 9: Mantel test scatter: pairwise embedding distance vs. ASJP phonetic distance across 88 languages, colored by phylogenetic relationship tier—same subfamily (blue), cross-branch Indo-European (amber), and cross-family (red). Dashed lines show per-group linear fits with Spearman ρ values. The overall Mantel ρ=0.13 ( p=0.020 ).
Figure 10: 2D PCA projection of Swadesh concept embeddings pooled across all 19 language families. Small translucent dots show per-family centroid positions (one per concept per family, ∼1,900 points total); convex hulls delineate each semantic category’s spread; large opaque dots mark the overall cross-lingual centroids. Body parts and nature terms occupy distinct regions of the space, while pronouns cluster tightly—confirming that the model’s geometry is organized by meaning rather than arbitrary lexical associations.
Figure 11: Cosine similarity as a function of colexification frequency for 1431 Swadesh concept pairs. Each red point is a pair attested by at least one CLICS 3 language family; grey points are non-colexified controls. The dashed line is a linear fit. Spearman ρs=0.17 ( p=2.15e−10 ). Selected pairs are labelled to illustrate the semantic content at different frequency levels.
Figure 12: PCA projection of 11 Berlin & Kay basic color terms across 136 languages. (a) 2D chromatic plane: small translucent dots show per-language embeddings; convex hulls delineate spread; large circles mark cross-lingual centroids. Warm and cool colors occupy opposing regions, recovering the circular topology of perceptual color space. (b) 3D view: the third principal component separates achromatic terms (white, black, grey; square markers) from the chromatic plane, revealing a luminance axis orthogonal to the hue circle—consistent with the achromatic–chromatic distinction in the Berlin & Kay hierarchy.
Figure 13: Semantic offset invariance across languages. (a) Each bar shows the mean cosine similarity between per-language offset vectors and the centroid offset for a given concept pair; the best-performing pair is fire–water . (b) Per-family disaggregation: each cell shows the mean consistency averaged over languages within each family. Rows are concept pairs (sorted by overall consistency); columns are language families. Warmer colors indicate higher consistency.
Figure 14: Offset vector demonstration for the top-4 concept pairs. (a) Joint PCA projection showing per-language embeddings (translucent), centroid positions (white markers), and offset arrows for each pair (colored). Thin arrows show individual per-language offsets; bold arrows show centroid offsets. (b) All offset vectors plotted from a common origin, revealing directional consistency: per-language offsets cluster tightly around their centroid direction for each pair, confirming language-invariant relational structure.