Existing sentence representations primarily encode what a sentence says, rather than how it is expressed, even though the latter is important for many applications. In contrast, we develop sentence representations that capture style and dialect, decoupled from semantic content. We call this the task of idiolectal representation learning. We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content, to learn a continuous representation of each sentence's style and dialect. We evaluate the approach on dialects of both Arabic and Spanish. The learned representations capture meaningful variation and transfer across domains for analysis and classification. We further explore the use of these representations as training objectives for stylistically aligning language models. Our results suggest that jointly modeling individual and community-level variation provides a useful perspective for studying idiolect and supports downstream applications requiring sensitivity to stylistic differences, such as developing diverse and accessible LLMs.
Figures & tables
Figure 1: IdioleX used to compare the idiolectal alignment between user input and GPT 5.1 generations in casual Argentinian Spanish.
Figure 2: IdioleX training framework. During training, all batches are sampled such that every individual item can act as an anchor for contrastive learning, necessitating that there are 23−n samples for each proximity score n∈[0,3] .
Figure 3
Models
Accuracy
IdioleX
31%
Fine-tuned IdioleX
36%
Fine-tuned IdioleX + Lexical
38%
Fine-tuned BERT
28%
Centroid Clustering w/ BERT
16%
Centroid Clustering w/ E5
27%
Table 2: Open-set authorship attribution exact match accuracy on the PAN 19 benchmark.
Figure 5
Model
ADI2
ChrF++
mar
egy
pse
syr
sau
mar
egy
pse
syr
sau
Allam + Idiolex SFT
0.54
0.48
0.14
0.12
0.11
33.2
36.4
36.1
34.6
36.7
Allam + SFT
0.49
0.45
0.12
0.13
0.11
31.8
33.3
34.2
35.1
37.3
Allam
0.26
0.28
0.05
0.07
0.04
30.7
33.3
34.3
33.5
35.4
Llama 3.1+ IdioleX SFT
0.60
0.29
0.12
0.10
0.07
27.0
25.4
27.1
26.4
29.5
Llama 3.1 + SFT
0.57
0.27
0.11
0.08
0.05
22.0
22.4
25.4
23.9
28.2
Table 3: Dialectalness (ADI2) and translation performance (ChrF++) scores on the AMIYA shared task evaluation set for Moroccan ( mar ), Egyptian ( egy ), Palestinian ( pse ), Syrian ( syr ), and Saudi ( sau ) Arabic. Our models, particularly Allam + Idiolex SFT, balance style and translation quality across dialects, as compared to reported scores for top submissions for the AMIYA shared task ( Robinson et al., 2026 ) (bottom three rows). Best scores from our experiments are bolded, while best reported scores from the AMIYA shared task are underlined.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Subreddit
Dialect Label
# Authors
# Sentences
Pre-Train
Train
Dev
Test
Arabic
r/algeria
Algerian
3728
24003
5977
149
182
r/Egypt
Egyptian
21117
536429
16188
209
129
r/Iraq
Iraqi
2542
21595
7810
90
224
r/jordan
Jordanian
14745
427230
15159
272
204
r/Kuwait
Kuwaiti
1739
13099
6027
88
228
Appendix
Table 4: Data split sizes for Reddit data by dialect.
Spanish Binary Features
Subject Pronouns & Verbal Morphology
contains explicit subject yo
contains 2pl present suffix ais
contains explicit subject tu
contains 2pl present suffix eis
contains explicit subject usted
contains 2pl present suffix is
contains explicit subject vos
contains voseo present suffix as stressed
contains explicit subject vosotros or vosotras
contains voseo present suffix es stressed
Appendix
Table 5: Spanish linguistic features extracted via LLM for IdioleX training. All features are sentence-level binary indicators.
Arabic Binary Features
Morphosyntax & Clause Structure
contains case endings u a i
contains future prefix sa
contains tanwin un an in
contains future particle sawfa
contains dual suffix an or ayn
contains dialectal future marker rah ha ghadi bash
contains 2sg feminine suffix ki
contains progressive marker 3am qa3id jaalis ka bi
contains 2sg feminine dialect suffix sh or ish
contains imperfect prefix bi or ba
Appendix
Table 6: Arabic linguistic features extracted via LLM for IdioleX training. All features are sentence-level binary indicators.
Parameter
Value
Hidden size
768
Transformer layers
12
Maximum sequence length
512
Batch size
32
Ranking group size
16
Pretraining epochs
3
Appendix
Table 7: Training hyperparameters used for IdioleX .
Hyperparameter
Value
Learning Rate
1×10−5
Batch Size
32
Weight Decay
0.01
Warmup Ratio
0.1
Early Stopping Patience
3 epochs
Max Sequence Length
512 tokens
Appendix
Table 8: Hyperparameters used for classification fine-tuning.
Hyperparameter
Value
Learning Rate
1×10−5
Batch Size
32
Weight Decay
0.01
Warmup Ratio
0.1
Early Stopping Patience
3 epochs
Max Sequence Length
512 tokens
Appendix
Table 9: Hyperparameters used for the lexical component of the classifier.
Language
Model
KMeans
Centroid-Based
Retrieval
F1-macro
Accuracy
F1-macro
Accuracy
Accuracy*
Arabic (MADAR 26)
BERT
0.11
0.14
0.20
0.22
0.10
E5
0.04
0.07
0.15
0.17
0.06
Spanish (DSL-ML)
BERT
0.50
0.46
0.77
0.57
0.07
E5
0.44
0.45
0.74
0.54
0.06
Spanish (PAN 19)
BERT
0.07
0.32
0.14
0.16
0.07
Appendix
Table 10: Clustering and retrieval-based evaluation of dialectal structure in sentence embedding spaces.
Source
Type
Dialects
Size
Method
MADAR-26 (train) ( Bouamor et al., 2018 )
Bitext
Multi-Dialectal
2k sentences per
Template
FLORES-200 ( Goyal et al., 2022 )
Bitext
Multi-Dialectal
1k sentences per
Template
SauDial ( Alanazi et al., 2025 )
Bitext
Saudi
1k sentences
Template
PALM (train) ( Alwajih et al., 2025 )
Monolingual
Multi-Dialectal
4k sentences
LLM
HABIBI ( El-Haj, 2020 )
Monolingual
Multi-Dialectal
525k verses
LLM
EDC ( Tarmom et al., 2020 )
Monolingual
Egyptian
13k sentences
LLM
Appendix
Table 11: AMIYA data used for post-training and how it was augmented to make instruction-following data.