Foundational Study on Authorship Attribution of Japanese Web Reviews for Actor Analysis

Hiroshi Matsubara; Shingo Matsugaya; Taichi Aoki; Masaki Hashimoto

cs.CLMar 24, 2026

Foundational Study on Authorship Attribution of Japanese Web Reviews for Actor Analysis

Authors: Hiroshi Matsubara, Shingo Matsugaya, Taichi Aoki, Masaki Hashimoto

Organizations: Faculty of Engineering and Design, Kagawa University · Trend Micro, Inc. · JC3 Japan Cybercrime Control Center

Abstract

This study investigates the applicability of authorship attribution based on stylistic features to support actor analysis in threat intelligence. As a foundational step toward future application to dark web forums, we conducted experiments using Japanese review data from clear web sources. We constructed datasets from Rakuten Ichiba reviews and compared four methods: TF-IDF with logistic regression (TF-IDF+LR), BERT embeddings with logistic regression (BERT-Emb+LR), BERT fine-tuning (BERT-FT), and metric learning with $k$ -nearest neighbors (Metric+kNN). Results showed that BERT-FT achieved the best performance; however, training became unstable as the number of authors scaled to several hundred, where TF-IDF+LR proved superior in terms of accuracy, stability, and computational cost. Furthermore, Top- $k$ evaluation demonstrated the utility of candidate screening, and error analysis revealed that boilerplate text, topic dependency, and short text length were primary factors causing misclassification.

Figures & tables

Author (Year)	Method	Lang.	Length	Dataset	Role
Classical Stylometry
Mosteller & Wallace (1964) ( Mosteller and Wallace, 1964 )	Bayes + function words	En	Long	Political documents, few authors	$\circ$
Stamatatos (2009) ( Stamatatos, 2009 )	Feature + classifier + evaluation (Survey)	Multi	–	Various corpora (survey)	$\circledast$
Jin & Murakami (2007) ( Jin and Murakami, 2007 )	RF + stylistic features	Ja	Long	Novels/essays/diaries	$\circ$
Challenges of Conventional Methods (Scale, Topic Dependency)
Luyckx & Daelemans (2010) ( Luyckx and Daelemans, 2010 )	Scale factor analysis ( $U$ , $k$ )	En	Short	Email, etc. (variable $U$ , $k$ )	$\circledast$

Table 1: Comparison of related work (Short = short text, Long = long text,

\circledast

= core,

\circ

= supplementary,

\triangle

= background).

Item	Value
Number of authors $U$	100
Total reviews $N$	68,222
Posts/author (min–median–max)	419 – 548 – 2,489
Posts/author (mean $\pm$ SD)	682.22 $\pm$ 472.6
Chars/review (median)	60
Chars/review (mean)	118.45

Table 2: Rakuten Ichiba review dataset statistics (2019, top 100 authors).

Item	Value
Number of authors $U$	1,000
Total reviews $N$	294,443
Posts/author (min–median–max)	186 – 242 – 2,489
Posts/author (mean)	294.44
Chars/review (median)	53
Chars/review (mean)	88.76

Table 3: Rakuten Ichiba review dataset statistics (2019, top 1,000 authors).

Item	Setting (Key Parameters)
Common	Stratified 5-fold cross-validation (seed=42); Metrics: Accuracy, Macro-F1, Top- $k$ ( $k\in\{3,5,10\}$ )
TF-IDF	Character 2-gram to 3-gram, max_features=10,000
LR	$C=1.0$ , solver=lbfgs, max_iter=1,000
BERT (Emb/FT)	cl-tohoku/bert-base-japanese-v3 , max length 128
BERT-FT	AdamW, epochs=10, batch=16, weight_decay=0.01, warmup_steps=100, fp16
Metric+kNN	BatchHardTripletLoss, normalize=True, kNN: cosine, $k=3$

Table 4: Key training settings used in this study.

	$k=100$		$k=200$		$k=300$		$k=400$
Method	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
TF-IDF+LR	0.7372	0.7343	0.7903	0.7889	0.8157	0.8156	0.8314	0.8323
BERT-Emb+LR	0.7059	0.7059	0.7467	0.7464	0.7636	0.7644	0.7772	0.7781
BERT-FT	0.7663	0.7662	0.8143	0.8127	0.8420	0.8415	0.8624	0.8597
Metric+kNN	0.6902	0.6778	0.7369	0.7283	0.7595	0.7542	0.7793	0.7735
	$k=100$		$k=200$		$k=300$		$k=400$

Table 5: Experiment 1 (

U=100

): Method comparison (

k\in\{100,200,300,400\}

, 5-fold mean).

Figure 2: Experiment 1: Relationship between reviews per author $k$ and Accuracy ( $U=100$ ).

Figure 3: Experiment 1: Relationship between reviews per author $k$ and Macro-F1 ( $U=100$ ).

Panel of Figure 4 — Figure 4: Relationship between $k$ and Top- $k$ performance at $U=100$ .

Method	Acc	Macro-F1	Top-5	Train	Infer
TF-IDF+LR	0.8622	0.8459	0.9576	36.72	0.94
BERT-FT	0.8827	0.8610	0.9621	1063.99	7.70

Table 6: Experiment 2A: Performance and computational cost under imbalance (full data,

U=100

, 5-fold mean; time in seconds/fold).

$K_{\max}$	TF-IDF+LR			BERT-FT
	Acc	Macro-F1	Top-5	Acc	Macro-F1	Top-5
500	0.8426	0.8416	0.9484	0.8489	0.8454	0.9510
1000	0.8567	0.8473	0.9548	0.8694	0.8569	0.9579
1500	0.8584	0.8464	0.9561	0.8710	0.8537	0.9592

Table 7: Experiment 2B: Effect of

K_{\max}

cap (

U=100

, 5-fold mean).

$U$	$N$	TF-IDF+LR					BERT-FT
		Acc	Macro-F1	Top-5	Top-10	Time(s)	Acc	Macro-F1	Top-5	Top-10	Time(s)
2	372	0.997 $\pm$ 0.005	0.997 $\pm$ 0.005	1.000	1.000	3	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000	1.000	61
5	930	0.975 $\pm$ 0.012	0.975 $\pm$ 0.012	1.000	1.000	4	0.983 $\pm$ 0.009	0.983 $\pm$ 0.009	1.000	1.000	97
10	1,860	0.901 $\pm$ 0.012	0.900 $\pm$ 0.012	0.989	1.000	3	0.936 $\pm$ 0.015	0.936 $\pm$ 0.015	0.996	1.000	144
20	3,720	0.897 $\pm$ 0.015	0.898 $\pm$ 0.015	0.978	0.994	6	0.901 $\pm$ 0.014	0.901 $\pm$ 0.013	0.981	0.995	326
50	9,300	0.867 $\pm$ 0.010	0.867 $\pm$ 0.010	0.956	0.985	16	0.871 $\pm$ 0.011	0.870 $\pm$ 0.011	0.967	0.988	717

Table 8: Experiment 3: Scaling characteristics (

k=186

fixed, 5-fold mean; Acc/Macro-F1 as mean

\pm

SD, Top-

k

and time as mean).

Authorship Attribution

Authorship attribution identifies or verifies who wrote a text, code sample, or other creative work from patterns in its style and content.

See all 66 papers

Fusing Stylometric and Embedding Systems to Estimate Authorship Likelihood Ratios in Japanese
Jun 12, 2026Praju Ghatpande, Satoru Tsuge, Shunichi Ishihara +2Ensemble Learning Digital Forensics
AuthBench: A Large-Scale Multilingual Benchmark for Authorship Representation across Genres and Lengths
Sep 6, 2026MaoXun Huang, Zhenxing Zhang, Claire CardieAuthorship Attribution
Department of Computer Science, Cornell University Ithaca, NY, United States
Rank-Turbulence Delta and Interpretable Approaches to Stylometric Delta Metrics
Apr 21, 2026Dmitry Pronin, Evgeny KazartsevComputational Literary Studies Stylometry
HSE University

Jun 12, 2026J/K move · Enter open · S save

Fusing Stylometric and Embedding Systems to Estimate Authorship Likelihood Ratios in Japanese

Praju Ghatpande, Satoru Tsuge, Shunichi Ishihara, Wataru Zaitsu+1

Ensemble Learning Digital Forensics Stylometry Authorship Attribution

Open paper AskPDF arXiv

The likelihood ratio framework is widely recognized as the logically and legally sound basis for evidential analysis across forensic sciences, and its importance is increasingly acknowledged in analyses of authorship in textual evidence. To date, however, its application has been confined to English-language texts. Meanwhile, authorship attribution has traditionally relied on a diverse array of stylometric features, even as the rise of pre-trained large language models enables new contextual-embedding approaches. Combining these diverse approaches through fusion promises enhanced performance, yet it has not been applied to integrate stylometric-feature systems with embedding-based systems within the likelihood ratio paradigm. This study is the first to apply likelihood ratio-based forensic text comparison to Japanese digital texts, using ~1,000-character excerpts from blogs, to 1) evaluate system performance and likelihood ratio magnitudes and 2) assess the impact of fusing stylometric-feature systems with embedding-based systems. The results demonstrate that the fused system maintains excellent calibration while 1) increasing consistent-with-fact likelihood ratio magnitudes; 2) decreasing contrary-to-fact likelihood ratio magnitudes and 3) improving overall discriminability. The best-performing fusion achieved a log-likelihood-ratio cost of 0.32484, illustrating both the feasibility of likelihood ratio framework for Japanese and the benefits of fusion across heterogeneous systems.

Abstract

Authorship Attribution

Explore similar work

Fusing Stylometric and Embedding Systems to Estimate Authorship Likelihood Ratios in Japanese

AuthBench: A Large-Scale Multilingual Benchmark for Authorship Representation across Genres and Lengths

Rank-Turbulence Delta and Interpretable Approaches to Stylometric Delta Metrics