STEB: Style Text Embedding Benchmark

Rafael Rivera Soto; Anna Wegmann; Cristina Aggazzotti

cs.CLJun 30, 2026

STEB: Style Text Embedding Benchmark

Authors: Rafael Rivera Soto, Anna Wegmann, Cristina Aggazzotti

Organizations: 1Johns Hopkins University · 2Utrecht University

Abstract

While semantic embeddings are rigorously evaluated on the Massive Text Embedding Benchmark, the evaluation of style embeddings remains fragmented, with each work relying on their own set of tasks and datasets. To bridge this gap, we introduce the Style Text Embedding Benchmark, a comprehensive open-source benchmark intended to standardize the evaluation of style embeddings. STEB encompasses 96 datasets across 7 languages, spanning applications such as authorship verification, authorship retrieval, AI-text detection, probing of linguistic features, and others. We find that semantic embeddings consistently fail in stylistic tasks, and that there is no style embedding that is universally superior across all tasks evaluated. We open-source the STEB code base at: https://github.com/rrivera1849/STEB.

Explore similar work

Evaluating Style-Personalized Text Generation: Challenges and Directions
Aug 8, 2025Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2Stylistic Fidelity Writing
Measuring Embedding Sensitivity to Authorial Style in French: Comparing Literary Texts with Language Model Rewritings
May 11, 2026Benjamin Icard, Lila Sainero, Alice Breton +2Stylistic Fidelity Authorship
The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
May 27, 2026Manuel Frank, Haithem AfliDomain-Invariant Text Embeddings Hard

Aug 8, 2025 · cs.CLJ/K move · Enter open · S save

Evaluating Style-Personalized Text Generation: Challenges and Directions

Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan, Adrian de Wynter+1

Columbia University, USA · Microsoft, USA · The University of York, UK

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.

Stylistic Fidelity Writing Text Generation Evaluation Metrics

Open paper Ask about it PDF

Abstract

Explore similar work

Evaluating Style-Personalized Text Generation: Challenges and Directions

Measuring Embedding Sensitivity to Authorial Style in French: Comparing Literary Texts with Language Model Rewritings

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness