Existing sentence representations primarily encode what a sentence says, rather than how it is expressed, even though the latter is important for many applications. In contrast, we develop sentence representations that capture style and dialect, decoupled from semantic content. We call this the task of idiolectal representation learning. We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content, to learn a continuous representation of each sentence's style and dialect. We evaluate the approach on dialects of both Arabic and Spanish. The learned representations capture meaningful variation and transfer across domains for analysis and classification. We further explore the use of these representations as training objectives for stylistically aligning language models. Our results suggest that jointly modeling individual and community-level variation provides a useful perspective for studying idiolect and supports downstream applications requiring sensitivity to stylistic differences, such as developing diverse and accessible LLMs.
Figures & tables
Figure 1: IdioleX used to compare the idiolectal alignment between user input and GPT 5.1 generations in casual Argentinian Spanish.
Figure 2: IdioleX training framework. During training, all batches are sampled such that every individual item can act as an anchor for contrastive learning, necessitating that there are 23−n samples for each proximity score n∈[0,3] .
Figure 3
Models
Accuracy
IdioleX
31%
Fine-tuned IdioleX
36%
Fine-tuned IdioleX + Lexical
38%
Fine-tuned BERT
28%
Centroid Clustering w/ BERT
16%
Centroid Clustering w/ E5
27%
Table 2: Open-set authorship attribution exact match accuracy on the PAN 19 benchmark.
Figure 5
Model
ADI2
ChrF++
mar
egy
pse
syr
sau
mar
egy
pse
syr
sau
Allam + Idiolex SFT
0.54
0.48
0.14
0.12
0.11
33.2
36.4
36.1
34.6
36.7
Allam + SFT
0.49
0.45
0.12
0.13
0.11
31.8
33.3
34.2
35.1
37.3
Allam
0.26
0.28
0.05
0.07
0.04
30.7
33.3
34.3
33.5
35.4
Llama 3.1+ IdioleX SFT
0.60
0.29
0.12
0.10
0.07
27.0
25.4
27.1
26.4
29.5
Llama 3.1 + SFT
0.57
0.27
0.11
0.08
0.05
22.0
22.4
25.4
23.9
28.2
Table 3: Dialectalness (ADI2) and translation performance (ChrF++) scores on the AMIYA shared task evaluation set for Moroccan ( mar ), Egyptian ( egy ), Palestinian ( pse ), Syrian ( syr ), and Saudi ( sau ) Arabic. Our models, particularly Allam + Idiolex SFT, balance style and translation quality across dialects, as compared to reported scores for top submissions for the AMIYA shared task ( Robinson et al., 2026 ) (bottom three rows). Best scores from our experiments are bolded, while best reported scores from the AMIYA shared task are underlined.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Subreddit
Dialect Label
# Authors
# Sentences
Pre-Train
Train
Dev
Test
Arabic
r/algeria
Algerian
3728
24003
5977
149
182
r/Egypt
Egyptian
21117
536429
16188
209
129
r/Iraq
Iraqi
2542
21595
7810
90
224
r/jordan
Jordanian
14745
427230
15159
272
204
r/Kuwait
Kuwaiti
1739
13099
6027
88
228
Appendix
Table 4: Data split sizes for Reddit data by dialect.
Spanish Binary Features
Subject Pronouns & Verbal Morphology
contains explicit subject yo
contains 2pl present suffix ais
contains explicit subject tu
contains 2pl present suffix eis
contains explicit subject usted
contains 2pl present suffix is
contains explicit subject vos
contains voseo present suffix as stressed
contains explicit subject vosotros or vosotras
contains voseo present suffix es stressed
Appendix
Table 5: Spanish linguistic features extracted via LLM for IdioleX training. All features are sentence-level binary indicators.
Arabic Binary Features
Morphosyntax & Clause Structure
contains case endings u a i
contains future prefix sa
contains tanwin un an in
contains future particle sawfa
contains dual suffix an or ayn
contains dialectal future marker rah ha ghadi bash
contains 2sg feminine suffix ki
contains progressive marker 3am qa3id jaalis ka bi
contains 2sg feminine dialect suffix sh or ish
contains imperfect prefix bi or ba
Appendix
Table 6: Arabic linguistic features extracted via LLM for IdioleX training. All features are sentence-level binary indicators.
Parameter
Value
Hidden size
768
Transformer layers
12
Maximum sequence length
512
Batch size
32
Ranking group size
16
Pretraining epochs
3
Appendix
Table 7: Training hyperparameters used for IdioleX .
Hyperparameter
Value
Learning Rate
1×10−5
Batch Size
32
Weight Decay
0.01
Warmup Ratio
0.1
Early Stopping Patience
3 epochs
Max Sequence Length
512 tokens
Appendix
Table 8: Hyperparameters used for classification fine-tuning.
Hyperparameter
Value
Learning Rate
1×10−5
Batch Size
32
Weight Decay
0.01
Warmup Ratio
0.1
Early Stopping Patience
3 epochs
Max Sequence Length
512 tokens
Appendix
Table 9: Hyperparameters used for the lexical component of the classifier.
Language
Model
KMeans
Centroid-Based
Retrieval
F1-macro
Accuracy
F1-macro
Accuracy
Accuracy*
Arabic (MADAR 26)
BERT
0.11
0.14
0.20
0.22
0.10
E5
0.04
0.07
0.15
0.17
0.06
Spanish (DSL-ML)
BERT
0.50
0.46
0.77
0.57
0.07
E5
0.44
0.45
0.74
0.54
0.06
Spanish (PAN 19)
BERT
0.07
0.32
0.14
0.16
0.07
Appendix
Table 10: Clustering and retrieval-based evaluation of dialectal structure in sentence embedding spaces.
Source
Type
Dialects
Size
Method
MADAR-26 (train) ( Bouamor et al., 2018 )
Bitext
Multi-Dialectal
2k sentences per
Template
FLORES-200 ( Goyal et al., 2022 )
Bitext
Multi-Dialectal
1k sentences per
Template
SauDial ( Alanazi et al., 2025 )
Bitext
Saudi
1k sentences
Template
PALM (train) ( Alwajih et al., 2025 )
Monolingual
Multi-Dialectal
4k sentences
LLM
HABIBI ( El-Haj, 2020 )
Monolingual
Multi-Dialectal
525k verses
LLM
EDC ( Tarmom et al., 2020 )
Monolingual
Egyptian
13k sentences
LLM
Appendix
Table 11: AMIYA data used for post-training and how it was augmented to make instruction-following data.
Systematic dialectal performance gaps in language models (LMs) are well documented, but the source of these disparities within the modern language modeling pipeline remains unclear. Our study traces this "dialect tax" across the natural language processing pipeline. Using parallel English dialect corpora that hold meaning fixed while varying surface form, we first confirm that LMs recognize matched Standard American English (SAE) and dialectal texts as semantically equivalent. However, we discover further representational gaps corresponding to downstream performance gaps. Across model families and generations, modern LMs still encode dialectal texts unequally during tokenization, pre-training, post-training, and inference. Strikingly, bypassing traditional subword segmentation via a character-level counterfactual tokenizer removes neither input and output asymmetries nor dialectal accuracy gaps. During pre-training, dialect pairs induce more divergent gradient updates than pairs of entirely unrelated SAE documents, indicating that models find semantically equivalent dialectal content harder to learn from than unrelated SAE documents. During post-training, reward models show contextual, unstable dialect preferences, assigning higher values to isolated AAVE-exclusive tokens than to SAE-exclusive tokens, while full reasoning contexts receive task- and model-dependent dialect penalties. Overall, our findings suggest that the dialect tax is encoded and accumulated not by any one step in isolation, but at every step of the language modeling process.
Elle
University of Oxford, Department of Computer Science
While large language model outputs are frequently analysed as a collective super variety termed "AI language," this chapter argues that this perspective coexists with distinct, model-specific linguistic signatures akin to human idiolects. We analyse two datasets of LLM-generated texts on societal topics: a 2024 corpus of six models (Improta et al. 2024) and a newly generated 2026 corpus using the same prompts featuring six contemporary models. Our findings, utilising computational descriptors and stylometric principal component analysis reveal a generational shift between the style of the 2024 and 2026 cohorts, while demonstrating that each individual model maintains a unique linguistic profile. This multi-layered interplay is illustrated by contraction frequencies, which vary from over 1,200 to over 30,000 per million words within the same cohort of models (2026). Ultimately, we conclude that treating LLM output as idiolectal in nature provides a valuable framework with potential implications for research on variation and change, LLM-generated text detection, forensic linguistics and usage-based approaches to language.
Karolina Rudnicka, Thomas Stephan Juzek
Faculty of Languages, University of Gdańsk, Poland · Department of Scientific Computing, Florida State University, United States
Dialectal data are scarce relative to Modern Standard Arabic (MSA), causing Arabic LLMs to overproduce MSA and struggle with dialectally accurate generation. This raises a fundamental interpretability question about where and how dialectal features are encoded within model internals and whether these representations can improve dialect generation without fine-tuning. We study two inference-time approaches as interpretability probes and control mechanisms. First, neuron-level analysis identifies sparse populations that encode dialect-specific features and tests whether amplifying or suppressing them steers model outputs toward target dialects. Second, vector steering extracts dialect-specific activation directions and injects them during inference, motivated by feature entanglement at the neuron level. We find that these neurons are real but only partially explanatory. They occupy under 1% of MLP dimensions but span only 5% to 21% of the residual dialect direction. This limited coverage is causally consequential. Neuron steering reinforces dialect in some varieties when the prompt is already dialectal but cannot induce it from MSA prompts, whereas vector steering succeeds in both settings. Arabic dialects are therefore steerable mainly through distributed rather than localized representations
Kareem Elozeiri, Mervat Abassy, Omar Kallas +4
1Mohamed bin Zayed University of Artificial Intelligence · 2Qatar Computing Research Institute, Hamad Bin Khalifa University · 3Tohoku University +1