Approaches to authorship verification or attribution typically model the style of a document, asking whether two documents were likely written by the same author. While recent approaches have been performant, such approaches largely ignore modeling the authors themselves and how varied their style is. Here, we introduce multiple alternatives for modeling the author and demonstrate that allowing an author's representation to capture multiple consistent but distinct writing styles yields significant performance improvements. Using our new authorship model, we then examine when an author writes in a way that is most idiomatic of their style across four scenarios: (i) variation in the size of their writing audiences, (ii) temporal distance from when the model was trained, (iii) the effect of preceding context, and (iv) the effect of genre norms on style. Across all four experiments, we demonstrate that authors vary substantially in how idiolectic their writing is. Our results have clear implications for both the modeling and data used for authorship verification and attribution, showing that authors need to be represented with their multiple styles and that some settings produce text that is much more informative about who wrote that text.
Figures & tables
Figure 1: In authorship attribution, documents (above: circles) are embedded as style vectors. Most approaches attribute a new document (the empty circle) to the nearest neighbor or an author’s centroid (stars). However, these methods overlook that authors often employ multiple styles. For instance, the new document here is nearest the blue author, even though fuchsia is the likely true source. Here, we develop models designed to mitigate this stylistic complexity.
Figure 2: Performance at attribution documents of 40K authors for different representations of authors and styles, shown as Success@8 on the left and p(a∣d) on the right. Representing authors as mixtures of styles yields consistent performance gains for both metrics. Note that the probabilities/Success@8 for the Nearest Neighbor and RoBERTa model are non-zero but round to 0.00.
Figure 3: As the number of communities that an author writes in increases, we see a decrease in stylistic consistency. Authors that write in a large number of subreddits write in their own style significantly less than authors who are focused in a few.
Figure 4: In longer-form documents, authors tend to write least like themselves in the beginning and ending portions. Authors write most like their style in the middle of the book, specifically just before the middle.
Figure 5: As time progresses, authors deviate more and more from their initial style. This indicates that author style changes significantly over time, and author models trained in the past may not be as effective as authors continue to write. This effect is particularly noticeable on Reddit and GMANE, which both show a steep decrease, whereas Wikipedia drops initially but then continues with only a small decline.
Figure 6: Heatmap of average log-likelihood for pairs of model window and data window. Performance is highest along the diagonal and declines with increasing separation, indicating distributional drift over time.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Test Count
Reddit
1,154,014
GMANE
1,206,047
Wikipedia
1,344,228
books
1,421,921
Appendix
Table 1: All sources have a training set of 500K and a development set size of 250K, while test set sizes vary as shown. All sources have over 100 test documents per author on average.
Figure 7: This figure shows consistency across the four data sources. Wikipedia has the lowest consistency by far, and the opposite can be said for GMANE. We hypothesize that this is a result of Wikipedia having more rules and structure, while emails are more informal and are used for day-to-day speech.
Figure 8: GMANE and Wikipedia have more complex idiolect distributions for their authors. Books and Reddit have comparable values, making it difficult to determine what causes authors to have more components. It seems plausible that GMANE has a higher number of styles because it can represent day-to-day and formal speech. However, Wikipedia usually follows a similar norm for its pages, so we would expect a lower number of styles.
Split
Success@8
p(a∣d)
Random
0.751
0.573
Held-out
0.684
0.493
Appendix
Table 2: Performance under random and held-out-subreddit splits for the cross-topic experiment.
Figure 9: The mean optimal number of Gaussian components, k , plateaus after approximately 20 training documents, indicating diminishing returns from additional samples.
GMM Components
Percentage of Authors
1
20.94
2
33.80
3
17.49
4
9.71
5
6.70
6
5.77
Appendix
Table 3: We find that roughly 61% of authors have optimal k values between 2-4. Additionally, only about 20.94% of authors have the centroid as their best model, indicating that an overwhelming majority of authors benefit from more complex modeling. Interestingly, 18% of authors have 5+ components, indicating a much higher idiolectic complexity than the average author.
Figure 10: There does not seem to be a general trend with respect to the strength of the embedding and the optimal k value. In fact, the weakest embedding (NeuroBiber) has a higher average number of components than the strongest embedding (SADIRI).
Figure 11: There is a clear overlap in the stylistic features that the six embeddings are capturing.
Figure 12: There is a large boost to performance for all embeddings after using orthogonalization in our modeling setup. This indicates that the step is crucial for our diagonal covariance setup.
Figure 13: We need not perform full orthogonalization to gain most of the benefits. This figure shows that we can use 50% of the PCA reduced embedding and get the best performance for a much lower computational cost.
Figure 14: Authors who are consistent in one genre are not guaranteed to be consistent in another. This Wikipedia analysis shows us that stratified authors who are consistent in a formal setting actually are the least consistent authors in a more informal setting.
Figure 15: The grid tabulates different hyperparameter configurations for the time heatmap found in Figure 6 . We can see that the general trend of the diagonal performing the best holds up regardless of the time range and minimum number of documents.
Figure 16: The mean number of components per GMM across the different temporal windows used in the experiment in Section 8 ; cf. Figure 6 . Note that the same authors are present in all windows, so the changes reflect a shift in global behavior.
Figure 17: Relative increase in location drift ( Dμ ) and spread drift ( Dσ ) as a function of temporal lag ∣i−j∣ . Location drift grows substantially faster than spread drift as lag increases, indicating that temporal degradation is driven primarily by stylistic displacement rather than inconsistency. Note we use the same configurations as Figure 6 .
Figure 18: Average Δself and Δparent values for GMANE replies. Replies become slightly more idiolectically rich for the reply author, while showing no meaningful accommodation.
Model
Fitting
Evaluating
GMM
O(AkdT)
O(A2Skd)
Centroid
O(AdT)
O(A2Sd)
Nearest Neighbor
O(Ad)
O(A2Sd)
Appendix
Table 4: Worst-case fitting and evaluation complexity for the author models.
Representation
NN
Centroid
GMM
Δ
NeuroBiber
0.15
0.07
0.08
+0.01
LISA
0.17
0.23
0.26
+0.03
StyleDistance
0.23
0.29
0.34
+0.05
LUAR
0.47
0.55
0.57
+0.02
PCM
0.64
0.65
0.69
+0.04
SADIRI
0.71
0.76
0.78
+0.02
Appendix
Table 5: Success@8 values corresponding to Figure 2 . NN denotes the nearest-neighbor model, and Δ represents the absolute improvement from using the GMM model versus the Centroid model.
Representation
NN
Centroid
GMM
Δ
NeuroBiber
0.00
0.03
0.04
+0.01
LISA
0.00
0.11
0.14
+0.03
StyleDistance
0.00
0.15
0.19
+0.04
LUAR
0.00
0.34
0.36
+0.02
PCM
0.00
0.44
0.50
+0.06
SADIRI
0.00
0.61
0.64
+0.03
Appendix
Table 6: Average p(a∣d) values corresponding to Figure 2 . NN denotes the nearest-neighbor model, and Δ represents the absolute improvement from using the GMM model versus the Centroid model.