q-bio.NCOct 17, 2026
SaveMultilingual Brain Decoding Reveals Shared Cortical Semantic Geometry
Organizations: Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · University of Chinese Academy of Sciences, Beijing, China. · The University of Hong Kong, Hong Kong, China. · Hong Kong Baptist University, Hong Kong, China.
Abstract
Human languages differ markedly in lexical and syntactic form yet can convey similar semantic content. The neural basis of this cross-language semantic commonality remains unclear. Here we developed a multilingual brain decoding framework and analysed brain recordings from 227 participants spanning Chinese, English and French. We found that cortical semantic representations exhibited consistent semantic geometry relationships across languages and individuals, revealing a shared cortical semantic geometry. Importantly, the cross-language consistency of cortical semantic geometry increased from sensorimotor to higher-order association cortex, forming a shared semantic geometry gradient closely aligned with the principal cortical functional gradient. Our work establishes generalizable multilingual brain decoding and identifies stable cortical semantic geometry as a neural basis for semantic commonality across languages.
Figures & tables
Figure 1: Overview of multilingual brain decoding and shared cortical semantic geometry. a , SemDecoder maps neural responses to Chinese, English and French stimuli into a unified semantic space for text reconstruction. b , Illustration of different cortical response patterns and preserved relative semantic relationships across languages. Corresponding semantic contents are shown in two-dimensional representational configurations. c , Cortical maps of the principal functional gradient and shared semantic geometry gradient, with their correspondence across Schaefer-400 parcels. Surface colours indicate percentile ranks within each map. Each point in the scatter plot denotes a parcel; the black line shows a least-squares fit. The Spearman correlation and one-sided spin-test value from 1,000 rotations are indicated. d , Zero-shot cross-language text decoding for English to Chinese, English to French and Chinese to English. BGEScore compares SemDecoder decoding with a control in which the model generates text without brain data (Random). Violin plots show participant distributions; dashed and dotted lines indicate medians and quartiles, respectively. e , Reference texts and corresponding decoded examples in English, Chinese and French. Green, yellow and red indicate exact matches, semantically similar content and unmatched content, respectively. **** .
Figure 2: Semantic decoding across languages and datasets. a , Decoding performance of SemDecoder and Tang2023 Tang et al. (2023) across datasets. Open and filled circles indicate Tang2023 and SemDecoder, respectively; colours indicate datasets. Scores are -normalized relative to text generated without brain data and oriented so that higher values indicate better performance; dashed line, randomized baseline. b,c , Participant-level ( b ) and BGEScore ( c ) across datasets and languages. , 35, 28, 6, 6 and 12 participants for LPPC-fMRI(EN), LPPC-fMRI(CN), LPPC-fMRI(FR), Chen2026(EN), Chen2026(CN) and SMN4Lang(CN), respectively. Comparisons in b,c used two-sided paired -tests with Holm–Šidák correction across the six comparisons for each metric; asterisks indicate adjusted values. d , Temporal BGEScore matrices comparing decoded texts with reference texts, averaged across participants. Horizontal and vertical axes indicate decoded and reference times, respectively. e , Cortical contribution to within-language semantic decoding, quantified by the decrease in brain–semantic cosine similarity after 8-mm searchlight occlusion. Colours show percentile ranks among positively contributing vertices; non-positive contributions are black. White lines mark major sulci. f , Participant-level contributions in eight temporal cortical regions, measured as the decrease in decoding performance after regional occlusion; insets identify each region. Asterisks indicate higher contributions than participants’ cortex-wide means (one-sided paired Wilcoxon signed-rank tests with Holm correction). Dotted line, zero contribution. , 49 and 28 participants for Chinese, English and French, respectively. Boxes in b,c,f show medians and interquartile ranges (IQRs); whiskers extend to IQR, and points beyond whiskers denote outliers. * , ** , *** , **** . L, left; R, right; STS, superior temporal sulcus; TTS, transverse temporal sulcus; STG, superior temporal gyrus; PT, planum temporale; HG, Heschl’s gyrus; MTG, middle temporal gyrus. EN, English; CN, Chinese; FR, French.
Figure 3: Cortical semantic geometry is shared across languages. a , Group-level cortical representational geometry before and after Procrustes alignment. Points denote corresponding narrative events (E1–E3), with distances reflecting dissimilarity between cortical response patterns. Similar relative configurations illustrate preserved relationships among events across languages. b , Cross-language RDM similarity between participants and the other language’s group average, pooled across both directions for each language pair. Points denote participants; dashed and dotted lines indicate medians and quartiles, respectively. c , Concreteness, socialness and arousal directions in the aligned cortical representational spaces. Arrows point from abstract to concrete, non-social to social and low to high arousal, respectively. Parallel arrows across languages indicate consistent semantic attribute directions. d , Cross-language parallelism of attribute directions versus participant-specific median random controls ( participants; one-sided paired Wilcoxon signed-rank tests, Holm-corrected across three attributes). e , Correspondence of neural distances between matched event pairs across languages. Points denote event pairs; lines show least-squares fits. Spearman correlations are indicated. f , Cross-language RSA along narrative progress. Solid lines and shading show the mean and s.e.m. across windows grouped by narrative progress; dashed lines show the phase-randomized control. g , Searchlight brain–semantic RSA for each language. h , Spatial Spearman correlations between individual brain–semantic RSA maps and the group averages of the other two languages, averaged across those languages. i , Searchlight cross-language RSA for each language pair. Cyan contours in g,i outline the top 20% of mean cross-language RSA. j , Spatial Spearman correlations between individual brain–semantic and mean cross-language RSA maps. In h,j , , 49 and 28 participants for Chinese, English and French, respectively. Asterisks in h,j indicate correlations above participant-specific spin-null medians (one-sided Wilcoxon signed-rank tests, Holm-corrected across languages within each panel). Boxes in d,h,j show medians and interquartile ranges (IQRs); whiskers extend to IQR, and points beyond whiskers denote outliers. *** , **** . s.e.m., standard error of the mean; CN, Chinese; EN, English; FR, French.
Figure 4: Cross-language generalization follows a gradient of shared cortical semantic geometry. a , Cross-language text decoding in LPPC-fMRI across all six language directions. BGEScore compares SemDecoder with text generated without brain data (Random). Arrows indicate training to test language. b , Bidirectional Chinese–English decoding in Chen2026. c , Decoders trained on LPPC-fMRI Chinese data were tested in NEBULA101 native French speakers listening to French ( ) or English ( ). An untrained network served as the control. Passage identification was measured by normalized rank among 12 candidates; dotted line, chance (0.5). d , Semantic geometry consistency versus single-region cross-language identification accuracy, averaged across six directions. Each point denotes a Schaefer-400 parcel. Identification uses two-alternative forced-choice matching with one parcel retained at a time. e , Cross-language text decoding with cortical inputs restricted to the low, middle or high third of the shared semantic geometry gradient for each language pair. Insets illustrate the Chinese–English groups. f , Cross-language neural response prediction (Pearson ) in low, middle and high thirds of the shared semantic geometry gradient ( source participants, each averaged over two target languages). Comparisons used two-sided paired Wilcoxon signed-rank tests with Holm correction across three pairwise comparisons. g , Principal functional gradient, extending from sensorimotor to transmodal cortex, and shared semantic geometry gradient. Colours indicate percentile ranks within each gradient. h , Correspondence between the gradients across 400 cortical parcels, coloured by semantic geometry consistency group. Lines in d,h show least-squares fits; Spearman correlations and values are indicated. i , Overlap between independently defined thirds of the two gradients. Cells give the fraction of parcels in each functional-gradient third assigned to a third of the shared semantic geometry gradient; rows sum to one. Boxes in a–c,f show medians and interquartile ranges (IQRs); whiskers extend to IQR, and points beyond whiskers denote outliers. ** , *** , **** . CN, Chinese; EN, English; FR, French.
Figure 5: Cross-language transfer improves decoding with limited target-language data. a–c , Cross-subject cross-language transfer in LPPC-fMRI, with English ( a ), Chinese ( b ) and French ( c ) as the target language. Boxes compare BGEScore for cross-language transfer models and a baseline fitted to the same target-language data. Comparisons used two-way repeated-measures ANOVA with Šidák-adjusted comparisons between models at each proportion (three comparisons per target language). d , Relative cortical contribution to decoding by transfer models fitted with 30% target-language data, measured by the reduction in brain–semantic cosine similarity after parcel occlusion. Colours show percentile ranks across Schaefer-400 parcels. Cyan contours mark the upper third of the shared semantic geometry gradient. e , Distributions of the relative contributions in d across low, middle and high thirds of the shared semantic geometry gradient. Points denote parcels ( , 133 and 134, respectively); dashed and dotted lines indicate medians and quartiles. Mid Low and High Mid were assessed using one-sided spin tests (1,000 rotations), with Holm correction across the two comparisons; asterisks indicate adjusted values. f , 12-way normalized-rank passage identification after cross-language transfer in native French speakers from NEBULA101 ( ), using cortical inputs from low, middle or high semantic geometry consistency groups; chance, 0.5. Comparisons used two-sided paired Wilcoxon signed-rank tests with Holm correction across three pairwise comparisons. Insets in e,f illustrate the semantic geometry consistency groups. g , Reference passages and texts decoded by the baseline and SemDecoder using 30% target-language training data. Green and orange highlight exact matches and semantically related content, respectively; red marks unmatched decoded content. Boxes in a–c,f show medians and interquartile ranges (IQRs); whiskers extend to IQR, and points beyond whiskers denote outliers. ** , *** , **** . EN, English; CN, Chinese; FR, French.
| Dataset | Language | Modality | Participants | Participant Time | Total Time |
|---|---|---|---|---|---|
| LPPC-fMRI (EN) Li et al. (2022) | English | fMRI | 49 | 1.5 h | 73.5 h |
| LPPC-fMRI (CN) | Chinese | fMRI | 35 | 1.5 h | 52.5 h |
| LPPC-fMRI (FR) | French | fMRI | 28 | 1.5 h | 42.0 h |
| Chen2026 (EN) Chen et al. (2026) | English | fMRI | 6 | 2.0 h | 12.0 h |
| Chen2026 (CN) | Chinese | fMRI | 6 | 2.0 h | 12.0 h |
| SMN4Lang Wang et al. (2022) | Chinese | fMRI | 12 | 5.1 h | 60.7 h |
Extended Data Table 1: Datasets.
| Dataset | Stimuli | Data split | Stimulus index | Words | Time (min) |
|---|---|---|---|---|---|
| LPPC-fMRI (EN) | 9 | Train | Story 1–8 | 13400 | 80 |
| Test | Story 9 | 1974 | 12 | ||
| LPPC-fMRI (CN) | 9 | Train | Story 1–8 | 20013 | 85 |
| Test | Story 9 | 2914 | 13 | ||
| LPPC-fMRI (FR) | 9 | Train | Story 1–8 | 13581 | 86 |
| Test | Story 9 | 1810 | 11 |
Extended Data Table 2: Training and evaluation partitions for the primary datasets.
Extended Data Fig. 1: Correspondence between shared semantic geometry and cortical functional gradients. a , Cortical maps of the shared semantic geometry gradient and the first three functional gradients (G1–G3). Shared semantic geometry was defined by cross-language searchlight representational similarity analysis (RSA), averaged across the three language pairs using the LPPC-fMRI training set. Colours indicate percentile ranks within each map. Lateral and medial views of both hemispheres are shown. b , Associations between shared semantic geometry and each functional gradient across Schaefer-400 parcels ( ). Each point denotes a parcel; red lines show least-squares fits. Correlations use raw gradient and RSA values. Spearman correlations and two-sided spin-test values from 1,000 rotations are indicated. LH, left hemisphere; RH, right hemisphere; a.u., arbitrary units.
Extended Data Fig. 2: Functional associations along the principal functional and shared semantic geometry gradients. Neurosynth association maps for 24 cognitive terms are summarized along the principal functional gradient (left) and the shared semantic geometry gradient (right). Schaefer-400 parcels were divided into 20 equal-sized percentile bins along each gradient; cells show mean association -statistics, with negative bin means set to zero. Rows are ordered by the weighted mean position of each term along the principal functional gradient and kept identical in both panels. Red boxes mark the peak bins of auditory, speech and language terms. Enrichment of these terms at the apex (top 5% of parcels) of the shared semantic geometry gradient, relative to the apex of the principal functional gradient, exceeded that of 1,000 surrogate maps accounting for correspondence between the two gradients (auditory, speech and language, all ; one-sided).
Extended Data Fig. 3: Shared semantic geometry and decoding across the cortical functional hierarchy. a , Semantic geometry consistency across deciles of the principal functional gradient, before and after adjustment for language-specific split-half reliability. Reliability was estimated by correlating sentence representational dissimilarity matrices (RDMs) between participant halves within each language. Original ranks and reliability-adjusted residuals were separately -standardized. Points show means; error bars show standard errors of the mean (s.e.m.) across parcels within each decile. Spearman and reliability-adjusted partial correlations are indicated, with one-sided spin tests using 1,000 rotations. b , Null distribution of the association between semantic geometry consistency and single-region cross-language decoding after 1,000 within-story permutations of cross-language sentence correspondence (grey histogram). The blue line marks the association for matched sentences. c , Single-region zero-shot cross-language decoding in parcels with lower or higher semantic geometry consistency within each third of the principal functional gradient (Low, Mid and High). Parcels within each third were divided at the median semantic geometry consistency. Boxes summarize parcel-wise accuracies; centre lines indicate medians and boxes indicate interquartile ranges. Within each third of the principal functional gradient, higher and lower semantic geometry consistency groups were compared using one-sided spin tests (1,000 rotations), with Holm correction across the three comparisons; asterisks indicate adjusted values: * . Analyses used Schaefer-400 parcels in LPPC-fMRI. Semantic geometry consistency was averaged across three language pairs using the training set. Decoding in b,c used two-alternative forced-choice identification with one parcel retained at a time, averaged across six language directions.
Extended Data Fig. 4: Within-language and cross-language neural response prediction. Models were fitted separately for each source participant using leave-one-story-out cross-validation. a , Neural response prediction within English and Chinese (top row) and across both language directions (bottom row) in Chen2026. Models were fitted and evaluated within each bilingual participant. b , Neural response prediction within Chinese, English and French (top row) and across all six language directions (middle and bottom rows) in LPPC-fMRI. Predictions were compared with target-language group-mean responses; within-language evaluation excluded the source participant from the group mean. Arrows indicate training to test language. Colours on flattened cortical surfaces show Pearson correlations between predicted and observed responses, with separate scales for the two datasets. Only correlations surviving one-sided sign-flip tests with family-wise error correction are coloured ( ; vertex-level max- in LPPC-fMRI, cluster-mass correction in Chen2026). SFS, superior frontal sulcus; IFS, inferior frontal sulcus; CS, central sulcus; IPS, intraparietal sulcus; STS, superior temporal sulcus; LF, lateral fissure.
Extended Data Fig. 5: Regional semantic geometry consistency and cross-language decoding. Associations between cross-language RSA and single-region decoding accuracy in LPPC-fMRI, shown for each of the six directions of cross-language decoding. Arrows indicate training to test language. Each point denotes a Schaefer-400 parcel ( ). Decoding used two-alternative forced-choice identification, retaining only the indicated parcel’s standardized cortical input. Solid lines show least-squares fits; dotted lines indicate chance accuracy (0.5). Spearman correlations and one-sided spin-test values (1,000 rotations), Holm-corrected across the six directions, are indicated. RSA, representational similarity analysis; CN, Chinese; EN, English; FR, French.
More in this topic
Cross-Lingual Representation Alignment
Cross-lingual representation alignment studies how models map meaning across languages into comparable internal or embedding spaces.
See all 68 papers