Organizations: Informatics Institute of Technology, Sri Lanka · Independant Researcher, Sri Lanka · National Health and Medical Research Council, Australia · Luxembourg Centre for Systems Biomedicine (LCSB), University of Luxembourg
The evolution of linguistic style in conversations is an underexplored issue in NLP. Most style-control datasets focus on sentences or assume a static style throughout, missing the dynamic shifts that occur as user preferences change during interactions. We introduce ConvoDrift, a dataset designed to model progressive stylistic conversational tone drift under fixed semantic intent. It is built on 15,727 shared multi-turn conversational structures for adaptation and persona-conditioned alignment methods. It consists of six prompt-response pairs per conversation, each with the annotation of style drift and style direction labels. These pairs cover a range of communication genres. We further derive a complementary pairwise dataset by pairing semantically equivalent but stylistically distinct responses and annotating persona-conditioned preferences using five distinct style communication personas, enabling the controlled study of personalisation and pluralistic alignment in language tone. In addition to dataset construction, we conduct a comprehensive evaluation involving human validation, LLM-as-judge assessment, and automatic lexical and semantic evaluations. Across seven Likert criteria annotated by three human annotators, the average Krippendorff's alpha is 0.88, and our lexical and semantic analyses show that drift events induce lexical changes while preserving semantic similarity.
Figures & tables
Figure 1: Overview of the ConvoDrift data construction and evaluation pipeline. 1 Data Generation illustrates the creation of structured multi-turn conversations across diverse domains and scenarios using selected language models, followed by the derivation of stylistically contrasting responses for RLHF pairwise preference data with persona-conditioned profiling. 2 Data Validation shows the quality assurance process, combining human validation via a custom annotation interface and LLM-as-a-judge validation. 3 Evaluation Experiments summarises the downstream evaluation protocols applied to the resulting datasets.
Communication genre
Count
Business email
2,997
Casual email
3,246
Quotes/Wishes
3,340
LinkedIn post
3,150
Tweet
2,994
Table 1: Conversation counts by communication genre in the ConvoDrift dataset.
Persona
Description
Task-First
Focuses on efficiency, clarity, and structured reasoning, prioritising actionable outcomes with minimal unnecessary elaboration.
Relationship-First
Emphasises warmth, empathy, and cooperative language to support positive and sustainable interpersonal interactions.
Authority-First
Communicates with confidence and decisiveness, favouring clear direction, assertive tone, and leadership-oriented language.
Calm & Careful
Uses measured, precise language to minimise risk, ensure safety, and promote stable and cautious decision-making.
Expressive & Energetic
Engages with enthusiasm and vivid language, favouring expressive phrasing that creates excitement and momentum.
Table 2: Persona profiles used for persona-conditioned preference labelling. Each persona encodes a distinct communication focus that guides stylistic preference judgements.
Criterion
α
95% CI
Tone consistency and drift annotation accuracy
0.763
[0.732, 0.794]
Tone direction label accuracy
0.777
[0.751, 0.803]
Drift distribution realism
0.891
[0.853, 0.926]
Fluency and clarity
0.946
[0.916, 0.973]
Cross-turn coherence
0.964
[0.939, 0.985]
Prompt-following accuracy
0.900
[0.863, 0.936]
Table 3: Inter-annotator agreement on the common subset measured using Krippendorff’s alpha for ordinal data. Confidence intervals correspond to 95% bootstrap intervals.
Similarity Type
n
Mean
Median
Min
Adjacent ( SBERT )
2500
0.8993
0.9207
0.3287
Anchor ( SBERT )
3000
0.8903
0.9034
0.2382
End-to-Start ( SBERT )
492
0.8519
0.8715
0.4423
Table 4: Overall turn-level semantic similarity across refinement turns.
Genre
#Convs
Mean
Median
Min
Business email
93
0.9123
0.9255
0.7594
Casual email
100
0.8551
0.8679
0.6223
LinkedIn post
99
0.8842
0.8919
0.7380
Quotes/Wishes
100
0.7739
0.7971
0.4423
Tweet
100
0.8385
0.8580
0.5517
Table 5: Domain-wise end-to-start semantic similarity measured using SBERT across different communication genres.
Genre
Type
n
ROUGE-L
Jaccard-1
Jaccard-2
Business email
Within (Drift=False)
6,242
0.606
0.543
0.374
Boundary (Drift=True)
8,742
0.538
0.469
0.281
Casual email
Within (Drift=False)
7,031
0.684
0.639
0.501
Boundary (Drift=True)
9,199
0.492
0.457
0.267
LinkedIn post
Within (Drift=False)
6,823
0.686
0.624
0.508
Boundary (Drift=True)
8,927
0.393
0.342
0.162
Table 6: Genre-wise lexical similarity between consecutive turns in conversations within non-drift regions (Within; Drift=False) and at drift boundaries (Boundary; Drift=True), reported with ROUGE-L and unigram/bigram Jaccard ( n transitions).
Model
Stability
Sensitivity
Average
Entropy
Formal Rate
Claude Haiku 4.5
0.990
0.533
0.853
0.542
Gemini 2.5 flash
0.942
0.504
0.868
0.581
GPT-5 Mini
0.932
0.483
0.886
0.585
GPT-5
0.956
0.442
0.884
0.604
Table 7: Comparison of candidate LLMs for persona-conditioned preference labelling.
Figure 3: Distribution of pairwise agreement scores for each persona using Cohen’s κ .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: t-SNE visualisation of the latent representations learned by the Llama 3.1 8B-based VPL model.
Figure 5: t-SNE visualisation of the latent representations learned by the GPT-2-based VPL model.
Figure 6: Business Email prompt template used for dataset generation.
Figure 7: Casual Email prompt template used for dataset generation.
Figure 8: LinkedIn post prompt template used for dataset generation.
Figure 9: Quotes and wishes prompt template used for dataset generation.
Figure 10: Tweet post prompt template used for dataset generation.
Figure 11: Prompt template used for LLM as a Judge.
A
B
C
D
E
A
1.00
-0.26
0.49
0.51
-0.29
B
-0.26
1.00
-0.36
-0.29
0.38
C
0.49
-0.36
1.00
0.34
-0.24
D
0.51
-0.29
0.34
1.00
-0.66
E
-0.29
0.38
-0.24
-0.66
1.00
Appendix
Table 8: Pairwise Cohen’s κ agreement between personas.
Figure 12: Distribution of drift labels across conversation turns.
Figure 13: Distribution of direction labels across conversation turns.
Figure 14: Distribution of response length by communication genre.