Understanding how the human brain constructs meaning from real-world language remains a fundamental challenge, as semantics depend on context that varies in natural settings. To enable the study of real-world linguistic meaning in the brain, we introduce SCRIBE, a framework grounded in a neuroscience-informed semantic space that leverages an ensemble of open-source large language models to generate interpretable, word-level semantic representations of arbitrary text. In a behavioral study with 921 participants, we show that human semantic feature judgments vary systematically with context, and that SCRIBE captures these shifts, reaching 75-81% of the estimated human noise ceiling. Using brain activity recorded while participants listen to naturalistic stories, we further demonstrate that SCRIBE predicts neural responses in language-selective brain regions, significantly outperforming non-contextual semantic representations. Variance partitioning reveals that SCRIBE explains unique brain-predictive information beyond syntactic features, suggesting that its representations capture aspects of linguistic meaning not accounted for by the measured syntactic features. Together, these findings establish SCRIBE as a scalable and interpretable foundation for studying natural language semantics in the brain.
Figures & tables
Figure 1: SCRIBE constructs interpretable, context-aware semantic features that are used to predict brain activity, enabling the modeling of the dynamic nature of meaning in the brain. It combines three information sources: a cognitively grounded semantic feature space ( Binder et al., 2016 ) , a target concept, and the linguistic context in which the concept appears. For each semantic property in the feature space, LLMs are prompted independently to assign a score reflecting how strongly the property applies to the concept in context. These property-wise scores are then aggregated across models to produce the final representations, which are used as input to a brain encoding model that predicts fMRI responses.
Model
Pearson ↑
pr
MSE ↓
pMSE
Mistral 7B Instruct
0.61
<0.001
2.16
<0.001
Qwen 2.5 7B Instruct
0.69
<0.001
1.32
<0.001
LLaMA 3.1 8B Instruct
0.67
<0.001
1.70
<0.001
LLaMA 3.1 70B Instruct
0.74
<0.001
1.53
<0.001
Ensemble
0.77
<0.001
1.07
<0.001
Table 1: Alignment between SCRIBE scores and human ratings for isolated concepts (i.e., without additional context) ( Binder et al., 2016 ) . We report Pearson correlation ( r ) and mean squared error (MSE), with corresponding permutation-based p -values. Overall, SCRIBE aligns significantly with human semantic ratings, with the ensemble achieving the best performance.
(a) Contextual modulation
Annotator
LT
O
HT
LT<O
HT>O
HT>LT
Human
2.67
3.23
4.23
<.001
<.001
<.001
Ensemble
1.58
2.18
3.54
<.001
<.001
<.001
Table 2: Human contextual judgments and their alignment with the SCRIBE ensemble. (a) Mean semantic feature scores for human judgments (computed from all ratings) and SCRIBE across the low-target ( LT ), original ( O ), and high-target ( HT ) conditions, with one-sided Wilcoxon signed-rank tests over concept–property pairs for the predicted directional contrasts. (b) Human reliability and human–SCRIBE alignment in each condition. ICC(1,8) : reliability of the mean of 8 participants; ceiling =ICC(1,8) , the maximum correlation attainable given human noise. r : Pearson correlation between SCRIBE and the mean human rating; r /ceiling: correlation normalized by the noise ceiling, with 95% bootstrap confidence intervals over concept–property pairs. MSE: mean squared error between SCRIBE and the mean human rating, with the chance level obtained by shuffling SCRIBE scores across pairs ( 1000 permutations) in parentheses. Values are averaged over 20 balanced subsamples of 8 ratings per item, as required for comparable reliability estimates across items. All ICCs, correlations and MSEs are significant ( p≤.001 , permutation tests).
Figure 2: (a) Voxel-wise differences in brain alignment between contextual and out-of-context variants of SCRIBE ( SCRIBEC−SCRIBEO ) for a representative subject (subject 3). Red regions show higher predictive accuracy for the contextual model. Improvements due to contextualization are predominantly observed in language-related cortical regions. (b) Brain alignment for SCRIBEO , QAEmb, and SCRIBEC across the whole brain and language-related ROIs. Bars represent mean performance across subjects, with individual subject values overlaid and error bars indicating the standard error. Red asterisks ( ∗ ) denote statistically significant differences (Wilcoxon signed-rank test). Contextual SCRIBE ( SCRIBEC ) consistently outperforms its out-of-context ( SCRIBEO ) variant, while achieving performance comparable to QAEmb.
Figure 3: Unique neural variance explained by SCRIBEC and QAEmb. (a) Spatial distribution of voxel-wise unique variance for a representative subject (subject 2). Red indicates greater unique variance explained by SCRIBEC , whereas blue indicates greater unique variance explained by QAEmb . (b) Region-wise comparison of uniquely explained variance across semantically relevant ROIs. Bars show mean performance across subjects, with individual values overlaid and error bars indicating the standard error. SCRIBEC explains more unique variance than QAEmb on average in all four ROIs, with significant differences in LTC ( p=0.039 ) and AG ( p=0.023 ).
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Type
Modality
Submodality
Description
Vision
Sensory
Vision
General
Something that you can easily see
Bright
Sensory
Vision
Intensity
Visually light or bright
Dark
Sensory
Vision
Intensity
Visually dark
Color
Sensory
Vision
Surface
Has a characteristic or defining color
Pattern
Sensory
Vision
Surface
Has a characteristic visual texture or surface pattern
Large
Sensory
Vision
Size
Large in size
Appendix
Table 3: Overview of the semantic properties from the feature space of Binder et al. (2016) , grouped by modality and cognitive domain. Each concept is represented as a vector over these interpretable semantic features. The table lists the 68 properties that are queried individually: the three bipolar properties Temperature , Texture , and Weight of the original 65 -dimensional space are each split into their two poles (Hot/Cold, Smooth/Rough, Light/Heavy), and the two resulting scores are recombined by taking their maximum, yielding the final 65 properties (Appendix B.2.1 ).
Figure 4: Distribution of semantic feature scores derived from human ratings in the Binder et al. (2016) dataset (composed of 535 words), shown for each of the 65 semantic properties.
Component
Variant
Modification
Baseline
Base prompt
Prompt used throughout the main experiments (Appendix A.2.1 ).
Role
Remove expert role
Remove the instruction: “You are an expert in rating to what degree words are associated with semantic properties.”
Task
Property name only
Provide only the property name instead of its relation and definition, e.g., “associated with Motion?”
Task
Property name + definition
Provide the property name followed by its relation and definition, e.g., “associated with Motion (showing a lot of visually observable movement)?”
Task
Relation: as
Replace “associated with” by “as”, e.g., “To what degree is the word {word} as {property relation} {property explanation}?”
Task
Relation: characterized by
Replace “associated with” by “characterized by”, e.g., “To what degree is the word {word} characterized by {property relation} {property explanation}?”
Appendix
Table 4: Prompt variants evaluated in the prompt ablation study. Each variant modifies a single component of the base prompt while leaving all other components unchanged.
Model
Ablation
MSE
Pearson Corr.
llama3.1-8b-instruct
Base
1.693
0.674
No role
2.615
0.580
Property name only
2.684
0.498
Property name + definition
1.867
0.639
Relation “as”
1.717
0.663
Relation “characterized by”
1.700
0.659
Appendix
Table 5: Prompt ablation results across individual models and their ensemble on the 535 words of Binder et al. (2016) . Pearson correlation and MSE are computed per word across the 65 properties and averaged over words. The base prompt is the main prompt; its values differ slightly from Table 1 because of non-deterministic GPU inference across runs.
Example
Context size
Context
1
k=3
of those people
k=5
I’m one of those people
k=10
raised with a religion and I’m one of those people
k=50
so a lot of people are raised with a religion and I’m one of those people
2
k=3
the word pretty
k=5
extension of the word pretty
Appendix
Table 6: Examples of the context provided to SCRIBE for different context window sizes. The target word is shown in bold.
Figure 5: Distribution of (concept, property) pairs across discretized baseline bins under two binning strategies. (a) Quantile-based thresholds ( 0.34 , 1.5 ) yield more uniform distributions across bins; however, the threshold for the high bin is low relative to the full rating scale ( 0 – 6 ), leading to less interpretable groupings. (b) Manual thresholds aligned with the original rating scale ( [0,2) , [2,4) , [4,6] ) produce more interpretable assignments of low, mid, and high association levels across properties. These observations motivate the use of fixed thresholds in our synthetic dataset construction.
Property
Concept
Low-context sentence
High-context sentence
Angry
thunder
The thunderstorm was a beautiful display of nature.
The loud thunder made me furious.
Motion
bus
The bus was parked at the station.
The bus sped down the highway.
Social
dinner
I ate dinner alone at home.
We had dinner together with friends at the restaurant.
Pleasant
fireworks
The loud fireworks startled the baby.
Watching fireworks on the 4th of July was a joyful experience.
Fast
bus
The bus moved slowly through traffic.
The bus sped down the highway.
Motion
activist
The activist sat quietly in the corner.
The activist marched through the streets.
Appendix
Table 7: Examples of controlled contextual manipulations for selected property-concept pairs. For each pair, the low-context sentence is designed to reduce the target semantic property, while the high-context sentence increases it, demonstrating how context systematically modulates semantic feature annotations.
Table 8: Brain regions and corresponding ROI labels.
Figure 6: Distribution of mean squared error (MSE) for out-of-context semantic annotation across models. For each model, we show the distribution of the MSE values computed on the true concept–property assignments (left) and under a permutation baseline obtained by randomly shuffling concept–property pairs (right). Across all models, the true MSE distribution is substantially lower than the permutation baseline, indicating that LLM-based annotations capture meaningful semantic structure beyond chance. Results are shown for individual models and the ensemble.
Figure 7: Distribution of Pearson correlation between LLM-generated semantic annotations and human ratings across models. For each model, we show the distribution of correlation values computed on the true concept–property assignments (left) and under a permutation baseline obtained by randomly shuffling concept–property pairs (right). Across all models, the true correlation distribution is substantially higher than the permutation baseline, indicating that LLM-based annotations capture meaningful semantic structure beyond chance. Results are shown for individual models and the ensemble.
Round 1
Round 2
Version 1
Version 2
Version 1
Version 2
Survey responses
270
276
256
258
Completed and matched to Prolific
249
255
250
250
After removing duplicates
245
255
250
250
Failed attention check(s)
– †
– †
25
29
Mean response time <7 s
2
13
4
5
Appendix
Table 9: Participant flow per data-collection round and questionnaire version. Each version covers half of the concept–property pairs.
Annotator
Concept
Average scores
Wilcoxon p -value (two-sided)
Wilcoxon p -value (one-sided)
Bin
LT
O
HT
LT vs O
HT vs O
HT vs LT
LT<O
HT>O
HT>LT
Human
All O
2.67
3.23
4.23
<.001
<.001
<.001
<.001
<.001
<.001
L O
1.73
1.78
3.36
0.666
<.001
<.001
0.333
<.001
<.001
M O
2.75
3.43
4.33
<.001
<.001
<.001
<.001
<.001
<.001
H O
3.52
4.47
4.98
<.001
<.001
<.001
<.001
<.001
<.001
Ensemble
All O
1.58
2.18
3.54
<.001
<.001
<.001
<.001
<.001
<.001
Appendix
Table 10: Contextual modulation of semantic feature scores for the human-evaluated subset. We report mean scores for human judgments and the SCRIBE ensemble across the original ( O ), low-target ( LT ), and high-target ( HT ) conditions, together with two- and one-sided Wilcoxon signed-rank tests. Results are shown across all concept–property pairs ( AllO ) and stratified by original out-of-context association strength ( LO , MO , HO ). Human means are computed from all ratings of each item. For pairs with low baseline association ( LO ), low-target contexts do not further decrease human ratings, consistent with a floor effect.
Model
Average scores
Wilcoxon p -value (two-sided)
Wilcoxon p -value (one-sided)
LT
O
HT
LT vs O
HT vs O
HT vs LT
LT<O
HT>O
HT>LT
Human
2.67
3.23
4.23
<.001
<.001
<.001
<.001
<.001
<.001
Llama3.1-70B
1.71
2.53
4.09
<.001
<.001
<.001
<.001
<.001
<.001
Llama3.1-8B
1.77
2.27
3.22
<.001
<.001
<.001
<.001
<.001
<.001
Mistral-7B
1.39
1.60
3.50
0.026
<.001
<.001
0.013
<.001
<.001
Qwen2.5-7B
1.43
2.32
3.36
<.001
<.001
<.001
<.001
<.001
<.001
Appendix
Table 11: Comparison of low-target ( LT ), original/no-context (O), and high-target ( HT ) scores assigned by human annotators, individual LLMs, and LLM ensembles to the evaluated subset of the synthetic dataset. O denotes each annotator’s own out-of-context score. All annotators exhibit the intended ordering LT<O<HT , with significant differences in all three expected directions ( LT<O , HT>O , and HT>LT ; all p<.05 , one-sided Wilcoxon signed-rank tests over concept–property pairs). Two-sided tests are also reported for completeness. The ensemble excluding Llama3.1-8B provides a control excluding the model used to generate the synthetic sentences.
Condition
ICC(1,1)
ICC(1,8) [95% CI]
Ceiling
F
Raters for 0.90
O
0.40
0.84 [0.82, 0.87]
0.92
6.39
14
LT
0.27
0.74 [0.70, 0.78]
0.86
3.91
25
HT
0.24
0.72 [0.66, 0.76]
0.85
3.55
29
Appendix
Table 12: Reliability of human judgments per condition (mean over 20 balanced subsamples of n=8 ratings per item; 95% bootstrap CI over items). The last column reports the number of raters needed for a reliability of 0.90 (Spearman–Brown).
Figure 8: Reliability of the mean human rating as a function of the number of raters per item. Lines show the Spearman–Brown prediction from ICC(1,1) (shaded: ± SD over 20 subsamples). Markers show empirical estimates from random subsets of raters: circles, ICC(1,n) computed on n rater columns; crosses, the correlation between the means of two disjoint groups of n raters (error bars: SD over subsamples). The dotted line marks n=8 , the number of raters used in the main analyses; dashed grey lines mark reliabilities of 0.7 , 0.8 and 0.9 .
r /ceiling ↑
MSE ↓
Model
O
LT
HT
O
LT
HT
Llama3.1-70B
0.79
0.76
0.64
2.98
3.97
3.30
Llama3.1-8B
0.70
0.68
0.66
3.49
2.77
4.27
Mistral-7B
0.57
0.65
0.67
5.82
4.41
3.69
Qwen2.5-7B
0.76
0.70
0.63
2.35
3.41
2.59
Ensemble
0.80
0.81
0.75
2.59
2.64
2.31
Appendix
Table 13: Alignment of individual models and the ensemble with human ratings (mean over 20 balanced subsamples of 8 ratings per item). Ceilings: 0.92 ( O ), 0.86 ( LT ), 0.85 ( HT ).
O
LT
HT
m=1 rater
0.54
0.39
0.37
m=2 raters
0.65
0.50
0.47
m=3 raters
0.70
0.56
0.52
m=4 raters
0.73
0.59
0.56
SCRIBE ensemble
0.68
0.62
0.56
Appendix
Table 14: Correlation with the mean of 4 held-out raters, for the mean of m other raters and for the SCRIBE ensemble ( 50 random splits ×20 subsamples).
Compared
Cond.
MSE [95% CI]
Chance
Floor
Offset yˉ−hˉ
Offset term
Scale term
Corr. term
SCRIBE
O
2.59 [2.22, 2.98]
6.53
0.36
−1.04
1.09
0.07
1.43
SCRIBE
LT
2.64 [2.31, 3.00]
5.71
0.45
−1.09
1.19
0.11
1.35
SCRIBE
HT
2.31 [1.91, 2.74]
4.80
0.35
−0.68
0.47
0.41
1.43
Binder norms
O
1.45 [1.22, 1.69]
5.54
0.36
−0.38
0.15
0.07
1.24
Appendix
Table 15: MSE between SCRIBE (or the Binder norms) and the mean of 8 human ratings, with 95% bootstrap CIs, the permutation baseline, the MSE floor, and the decomposition into offset, scale and correlation terms.
Model
Concept
Average scores
Wilcoxon p -value (two-sided)
Wilcoxon p -value (one-sided)
Bin
L T
O
H T
L T vs O
H T vs O
H T vs L T
L T < O
H T > O
H T > L T
Llama3.1-8B
All O
1.86
2.16
3.04
<.001
<.001
<.001
<.001
<.001
<.001
L O
0.67
0.62
1.39
0.168
<.001
<.001
0.916
<.001
<.001
M O
1.86
2.06
3.08
0.009
<.001
<.001
0.005
<.001
<.001
H O
3.06
3.81
4.64
<.001
<.001
<.001
<.001
<.001
<.001
Mistral-7B
All O
1.37
1.55
3.11
<.001
<.001
<.001
<.001
<.001
<.001
Appendix
Table 16: Per-model results for the controlled contextual manipulation experiment (see Section 4.2 and Appendix A.3 ) using all the contextual generated sentences. We report average semantic scores and Wilcoxon signed-rank tests for each individual model and their ensemble. O denotes each model’s own out-of-context score. For all models and the ensemble, semantic feature values shift in the intended direction ( HT>O>LT ) across all pairs and within the mid and high bins. For pairs with low baseline association ( LO ), llama3.1-8b and mistral-7b do not lower scores further in the low-target condition, mirroring the floor effect observed in human judgments (Table 10 ); the ensemble shows significant effects in all bins.
Word type
Context Size
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Nouns
out-of-context
0.1372
0.1122
0.1190
0.1066
0.0933
0.0896
0.0931
0.1073 ± 0.0065
window/k3
0.1420
0.1237
0.1301
0.1217
0.0789
0.0881
0.0807
0.1093 ± 0.0098
window/k5
0.1442
0.1202
0.1330
0.1129
0.1051
0.0873
0.0884
0.1130 ± 0.0081 ∗
window/k10
0.1621
0.1362
0.1384
0.1200
0.1101
0.0869
0.0858
0.1199 ± 0.0106 ∗
window/k50
0.1760
0.1485
0.1207
0.1381
0.1116
0.0831
0.0928
0.1244 ± 0.0123
up to concept
0.1566
0.1191
0.1001
0.1048
0.0732
0.0777
0.0681
0.1000 ± 0.0118
Appendix
Table 17: Brain encoding correlations across context sizes using llama3.1-8b-instruct . We compare out-of-context annotations, fixed-size context windows ( k∈{3,5,10,50} ), and an up to concept condition including all preceding stimulus context. Results are reported for three POS subsets: Nouns , Nouns, Verbs, Adjectives , and All Semantic POS . Values correspond to noise-ceiling-normalized correlations. Bold indicates the best context within each POS group, and underlining indicates the overall best configuration. ∗ denotes significant improvements over the out-of-context baseline (one-sided Wilcoxon signed-rank test, p<0.05 ).
Word type
Context Size
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Nouns
out-of-context
0.1368
0.0917
0.1012
0.0848
0.0821
0.0871
0.0693
0.0933 ± 0.0081
window/k3
0.1449
0.1248
0.1000
0.0938
0.0445
0.0939
0.0721
0.0963 ± 0.0124
window/k5
0.1506
0.1337
0.1156
0.0975
0.0794
0.1048
0.0898
0.1102 ± 0.0095 ∗
window/k10
0.1729
0.1521
0.1340
0.1300
0.0897
0.1103
0.0944
0.1262 ± 0.0115 ∗
window/k50
0.1622
0.1441
0.1248
0.1206
0.0871
0.0975
0.0880
0.1177 ± 0.0109 ∗
up to concept
0.1318
0.1171
0.1016
0.1051
0.0790
0.0832
0.0720
0.0985 ± 0.0082
Appendix
Table 18: Brain encoding correlations across context sizes using mistral-7b-instruct . We compare out-of-context annotations, fixed-size context windows ( k∈{3,5,10,50} ), and an up to concept condition including all preceding stimulus context. Results are reported for three POS subsets: Nouns , Nouns, Verbs, Adjectives , and All Semantic POS . Values correspond to noise-ceiling-normalized correlations. Bold indicates the best context within each POS group, and underlining indicates the overall best configuration. ∗ denotes significant improvements over the out-of-context baseline (one-sided Wilcoxon signed-rank test, p<0.05 ).
Word type
Context Size
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Nouns
out-of-context
0.1420
0.1083
0.1209
0.1181
0.0919
0.0883
0.0811
0.1072 ± 0.0082
window/k3
0.1537
0.1455
0.1360
0.1236
0.0880
0.0857
0.0906
0.1176 ± 0.0110 ∗
window/k5
0.1618
0.1482
0.1321
0.1304
0.1061
0.0927
0.0917
0.1233 ± 0.0103 ∗
window/k10
0.1809
0.1581
0.1449
0.1396
0.1097
0.0958
0.1036
0.1332 ± 0.0119 ∗
window/k50
0.1823
0.1363
0.1358
0.1358
0.0987
0.0878
0.0883
0.1236 ± 0.0130 ∗
up to concept
0.1606
0.1162
0.1415
0.1211
0.0917
0.0856
0.0876
0.1149 ± 0.0109 ∗
Appendix
Table 19: Brain encoding correlations across context sizes using qwen2.5-7b-instruct . We compare out-of-context annotations, fixed-size context windows ( k∈{3,5,10,50} ), and an up to concept condition including all preceding stimulus context. Results are reported for three POS subsets: Nouns , Nouns, Verbs, Adjectives , and All Semantic POS . Values correspond to noise-ceiling-normalized correlations. Bold indicates the best context within each POS group, and underlining indicates the overall best configuration. ∗ denotes significant improvements over the out-of-context baseline (one-sided Wilcoxon signed-rank test, p<0.05 ).
Word type
Context Size
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Nouns
out-of-context
0.1347
0.1239
0.1135
0.1104
0.0942
0.0943
0.0872
0.1083 ± 0.0066
window/k3
0.1454
0.1430
0.1136
0.1143
0.0832
0.0971
0.0838
0.1115 ± 0.0097
window/k5
0.1502
0.1382
0.1241
0.1247
0.0791
0.1026
0.0926
0.1160 ± 0.0096
window/k10
0.1711
0.1519
0.1295
0.1300
0.0927
0.1094
0.0961
0.1258 ± 0.0109 ∗
window/k50
0.1789
0.1372
0.1069
0.1370
0.0686
0.0948
0.0947
0.1169 ± 0.0139
up to concept
0.1509
0.1383
0.1080
0.1200
0.0524
0.0905
0.1028
0.1090 ± 0.0123
Appendix
Table 20: Brain encoding correlations across context sizes using llama3.1-70b-instruct . We compare out-of-context annotations, fixed-size context windows ( k∈{3,5,10,50} ), and an up to concept condition including all preceding stimulus context. Results are reported for three POS subsets: Nouns , Nouns, Verbs, Adjectives , and All Semantic POS . Values correspond to noise-ceiling-normalized correlations. Bold indicates the best context within each POS group, and underlining indicates the overall best configuration. ∗ denotes significant improvements over the out-of-context baseline (one-sided Wilcoxon signed-rank test, p<0.05 ).
Figure 9: Voxel-wise differences in brain-prediction accuracy between contextual and out-of-context SCRIBE representations ( SCRIBEC−SCRIBEO ) across subjects. Red regions indicate greater prediction accuracy for SCRIBEC , whereas blue regions indicate greater accuracy for SCRIBEO . Contextualization-related improvements are predominantly observed in language-related cortical regions, with stronger effects in the right hemisphere.
Figure 10: Unique variance explained by SCRIBEC and QAEmb across cortical regions. Bars show the mean unique explained variance across subjects, with error bars indicating the standard error of the mean; individual dots represent subjects. p -values correspond to one-sided paired Wilcoxon signed-rank tests testing whether the unique variance explained by SCRIBEC is greater than that explained by QAEmb. Asterisks indicate significant differences ( p<0.05 ).
Region
Method
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Whole Brain
SCRIBEC
0.2284
0.2263
0.1714
0.2023
0.1800
0.1519
0.1258
0.1837 ± 0.0144
QAEmb
0.2343
0.2115
0.1626
0.1788
0.1554
0.1544
0.1477
0.1778 ± 0.0125
QAEmb + SCRIBEC
0.2579
0.2426
0.1895
0.2140
0.1926
0.1684
0.1504
0.2022 ± 0.0146 ∗
Late Language
SCRIBEC
0.1673
0.1990
0.2112
0.1348
0.1371
0.1191
0.0536
0.1460 ± 0.0201
QAEmb
0.1715
0.1943
0.1932
0.1136
0.1214
0.1153
0.0615
0.1387 ± 0.0186
QAEmb + SCRIBEC
0.1880
0.2145
0.2253
0.1395
0.1450
0.1285
0.0632
0.1577 ± 0.0213 ∗
Appendix
Table 21: Noise-ceiling-normalized brain encoding correlations for SCRIBE ( SCRIBEC ), QAEmb, and their combined representation, evaluated across the whole brain and semantic language ROIs. Values are reported for individual subjects and as mean ± standard error across subjects. ∗ denotes that QAEmb + SCRIBEC significantly outperforms both individual representations (one-sided Wilcoxon signed-rank tests, p<0.05 ).
Figure 11: Voxel-wise unique variance explained by SCRIBE and QAEmb across subjects. Variance partitioning was performed directly between the two representations, without including the syntactic control features. Red regions indicate greater unique variance explained by SCRIBE, whereas blue regions indicate greater unique variance explained by QAEmb. Only voxels with a noise ceiling above 0.4 are shown. Across subjects, the maps predominantly show greater unique variance for SCRIBE, particularly in language-related cortical areas, although this pattern is less pronounced in Subject 8.
Region
Method
S1
S2
S3
S5
S6
S7
S8
Avg ± STE
Whole Brain
SCRIBEC (65)
0.2284
0.2263
0.1714
0.2023
0.1800
0.1519
0.1258
0.1837 ± 0.0144
SCRIBEC (33)
0.2275
0.2101
0.1848
0.2046
0.1897
0.1504
0.1340
0.1859 ± 0.0126
Late Language
SCRIBEC (65)
0.1673
0.1990
0.2112
0.1348
0.1371
0.1191
0.0536
0.1460 ± 0.0201
SCRIBEC (33)
0.1660
0.1897
0.2124
0.1351
0.1396
0.1178
0.0561
0.1452 ± 0.0197
LTC
SCRIBEC (65)
0.1991
0.2046
0.1770
0.1609
0.1819
0.1155
0.0377
0.1538 ± 0.0223
SCRIBEC (33)
0.1938
0.1974
0.1762
0.1615
0.1814
0.1146
0.0402
0.1522 ± 0.0216
Appendix
Table 22: Noise-ceiling-normalized brain encoding correlations for SCRIBEC before and after feature selection, evaluated across the whole brain and semantic language ROIs. Feature selection reduces the representation from 65 to 33 semantic dimensions. Values are reported for individual subjects and as mean ± standard error across subjects. At the whole-brain level, performance does not significantly differ after feature selection (two-sided Wilcoxon signed-rank test).
Figure 12: Variance partitioning of SCRIBEC and QAEmb with syntactic features across language-related ROIs. (a) Brain-predictive variance shared with the syntactic feature space. (b) Unique brain-predictive variance retained by each representation after accounting for the syntactic feature space. Bars show means across subjects and error bars indicate the standard error of the mean. Bold ROI labels and asterisks indicate significant paired differences between SCRIBEC and QAEmb (one-sided paired Wilcoxon signed-rank test, p<0.05 ).
Figure 13: Permutation-based significance testing of the unique neural variance explained by SCRIBEC after accounting for syntactic features. Histograms show permutation-based null distributions, and dashed lines indicate the observed unique variance.
Figure 14: Brain-encoding performance of Llama2-7b, QAEmb , and SCRIBEC . (a) Whole-brain encoding performance. (b) Encoding performance across the Late Language, lateral temporal cortex (LTC), IFG/MFG, and Angular Gyrus ROIs. Bars show mean noise-ceiling-normalized correlation across subjects, individual points represent subjects, and error bars indicate the standard error of the mean. Asterisks indicate significant differences between SCRIBEC and the corresponding baseline ( p<0.05 ; one-sided Wilcoxon signed-rank test).
Figure 15: Shared brain-predictive variance with syntactic features for contextual Llama2-7B and SCRIBE ( SCRIBEC ). Bars show the mean shared variance across subjects and error bars indicate the standard error of the mean. SCRIBE shares significantly less variance with the syntactic feature space than Llama2-7B in IFG/MFG (one-sided paired Wilcoxon signed-rank test, p<0.05 ), indicated by the bold ROI label and asterisk.
Figure 16: Permutation-based significance testing of the unique neural variance explained by SCRIBEC in variance partitioning with Llama2-7b. Histograms show permutation-based null distributions, and dashed lines indicate the observed unique variance.
Large language models (LLMs) predict neural responses during naturalistic language comprehension, but the linguistic information underlying this brain alignment remains difficult to interpret. We use SCRIBE, an interpretable contextual semantic representation spanning 65 neurocognitively grounded semantic properties, as a lens for decomposing the brain alignment of one LLM, LLaMA2-7B. Using variance partitioning between LLaMA2-7B and SCRIBE on naturalistic fMRI data from seven participants listening to narrative stories (Moth Radio Hour), we separate brain-predictive variance shared between the two representations from variance unique to each. We find that shared variance is the dominant component across the whole brain and language-related regions, while both representations also retain significant unique contributions. We then examine whether the concreteness of individual semantic properties relates to shared and SCRIBE-unique brain-predictive variance. Concrete properties show greater shared variance with LLaMA2-7B than abstract properties across the whole brain and most examined regions, whereas abstract properties do not show greater SCRIBE-unique variance. Finally, property ablations reveal that semantic contributions extend across multiple dimensions, with cumulative effects on both shared and SCRIBE-unique variance. Together, these results show that, for LLaMA2-7B on this dataset, a substantial portion of brain-LLM alignment is associated with interpretable semantic information, while revealing complementary brain-predictive semantic information beyond the LLaMA2-7B representation.
Camila Kolling, Mariya Toneva
Max Planck Institute for Software Systems (MPI-SWS)
Naturalistic language comprehension requires listeners to process both local probabilistic expectations and contextual semantic relations. This study tested whether contextual semantic relevance, measuring how strongly a target word relates to its recent semantic context, is associated with fMRI BOLD responses independently of word surprisal and lexical, timing, acoustic, and prosodic controls. We analyzed two public datasets: Alice (23 participants, one narrative) and Narratives (47 participants, 185 runs, four stories) using FIR/deconvolution and generalized additive mixed models. In Alice, semantic relevance was significant across all ROIs in FIR analyses, whereas surprisal was not. In GAMMs, both predictors showed broad significance. In Narratives, both predictors showed comparable spatial prevalence across ROIs. Semantic relevance showed robust BOLD associations across both datasets, with a particularly strong advantage over surprisal in the timing-sensitive Alice FIR analysis. The regionally heterogeneous direction of semantic relevance effects, with negative effects in posterior semantic regions and positive effects in frontal integration regions, suggests involvement of functionally distinct neural processes rather than a single uniform mechanism. These findings indicate that contextual semantic fit and local probabilistic expectation make partially distinct, dataset-dependent contributions to hemodynamic responses during naturalistic listening.
Kun Sun, Rong Wang
Department of Linguistics, Tongji University · Department of Computational Linguistics, Tuebingen University
Recent breakthroughs in language models (LMs) using neural networks have raised the question: how similar are these models' processing to human language processing? Results using a framework called Brain Score (BS) -- predicting fMRI activations during reading from LM activations -- have been used to argue for a high degree of similarity. To understand this similarity, we conduct experiments by training LMs on various types of input data and evaluate them on BS. We find that models trained on various natural languages from many different language families have very similar BS performance. LMs trained on other structured data -- the human genome, Python, and pure hierarchical structure (nested parentheses) -- also perform reasonably well and close to natural languages in some cases. These findings suggest that BS can highlight language models' ability to extract common structure across natural languages, but that the metric may not be sensitive enough to allow us to infer human-like processing from a high BS score alone.