cs.CLFeb 16, 2026
SaveRT-SFT: Text Style Transfer from Non-Parallel Corpora by Roundtrip Translation
Abstract
Text style transfer (TST) is naturally a supervised task - rewrite a sentence in a target style while preserving its meaning - yet the parallel corpora that supervision requires exist for only a handful of style domains. A common workaround is to normalize an input into a style-agnostic intermediate and then stylize it into the target style, but the normalizer is typically a lightweight, task-specific paraphraser applied only at test time, feeding a correspondingly small stylizer. We observe that a style-stripping normalizer already exists at scale: neural MT systems trained on hundreds of millions of general-domain sentence pairs preserve content while regressing toward generic phrasing, so roundtrip translation through a pivot language strips stylistic signal without any task-specific training. This turns normalization from an inference-time patch into a data-generation tool. Roundtrip-translating a monolingual in-style corpus yields a pseudo-parallel corpus on which we LoRA-finetune an instruction-tuned LLM as the stylizer (RT-SFT); applying the same normalizer to test queries keeps that stylizer in-distribution. We show that across four style domains, RT-SFT outperforms state-of-the-art methods, such as few-shot in-context learning, by considerable margins. We also report on effective retrieval augmentation methods for expert style domains with strict terminology and naming conventions.
Figures & tables
Figure 1: Our proposed workflow for finetuning large language models (LLMs) for text style transfer (TST) using only non-parallel data in the target domain. A bilingual general-domain parallel dataset is used to train a pair of neural machine translation (NMT) models capable of translating between English and a pivot language. We then obtain RT-normalized texts of the original in-domain texts by roundtrip translating the in-domain set with the NMT models. This enables supervised finetuning of LLMs for TST, where we finetune LLMs for RT-normalized-domain to target-domain transfer using the synthetic parallel corpus.
Figure 2: Example sentences illustrating semantically equivalent content in various styles. Outputs of our roundtrip translation pipeline are considered RT-normalized.
Figure 3: Our proposed workflow. We train MT systems using general domain parallel data for Chinese-English and German-English two-way translations, and roundtrip-translate target style text to obtain RT-normalized counterparts. We roundtrip translate every record with both pivot languages and mix the synthetic corpus. The synthetic parallel corpus is then used for finetuning a stylizer LLM, with retrieved information injected in the prompt.
Figure 4: Retrieval augmentation workflow. Left (a): Relevant example retriever. We encode the target-side texts of the parallel synthetic datasets for nearest-neighbor search. For each query, we first do k-shot inference with the RT-SFT-LLM to obtain an “in-domain” sketch, which is used as search query in the target-side dataset to obtain the k most relevant pairs. Right (b) Terminology and name retriever: For each instance in the synthetic parallel datasets, the first LLM call extracts relevant words from the source side, then the second call matches them with their counterparts in the target side, yielding a terminology pair list for each domain.
| Dataset | Training | Dev | Test |
| WMT26-eng-zho | 241,429,382 | 3,000 | – |
| WMT26-deu-eng | 1,128,176,715 | 3,000 | – |
| MSD | 130,199 | 150 | 675 |
| Shakespeare | 24,852 | 1,313 | 1,293 |
| GYAFC | 103,562 | 1,000 | 2,748 |
| ParaDetox | 19,466 | 150 | 150 |
Table 1: Datasets. Underlined counts are parallel (aligned) sentence pairs; plain counts are monolingual sentences. The WMT26 General Machine Translation corpora ( Kocmi et al., 2025 ) are used to train the roundtrip-translation normalizers; we use only the parallel portion of the released splits and manually hold out 3,000 sentence pairs per language pair as dev set. The four style datasets are used as training data for our RT-SFT method. Note that although 3 of the 4 style domains provide parallel training data, we only use the in-domain side of them during training, as our system only requires in-domain non-parallel training data. We use the provided, human-annotated parallel sentences only for the dev and test sets.
| Method | MSD | Shakespeare | GYAFC | ParaDetox | ||||
| Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | |
| COPY | 0.107 | 14.01 | 0.009 | 15.35 | 0.149 | 16.54 | 0.012 | 20.03 |
| 5-shot In-Context Learning ( 2022 ) | 0.396 | 9.58 | 0.771 | 12.79 | 0.521 | 19.34 | 0.792 | 21.35 |
| Automatic Post-Editing ( 2022 ) | 0.672 | 12.21 | 0.693 | 11.90 | 0.435 | 17.74 | 0.720 | 17.35 |
| RT-SFT 0-shot | 0.706 | 11.32 | 0.841 | 11.47 | 0.591 | 21.33 | 0.820 | 18.40 |
| RT-SFT random-3-shot | 0.715 | 13.35 | 0.809 | 13.21 | 0.620 | 22.41 | 0.893 | 18.11 |
Table 2: Main results. TST performance of RT-SFT against baselines across the four style domains (base model: Llama-3.1-8B-Instruct; synthetic corpus pooled from German and Chinese roundtrip pivots). random and relevant indicate whether the in-context example pairs are sampled at random or retrieved for relevance; + terminology and name retrieval adds domain-specific term guidance when triggered (§ 3.3.2 ). Best result per column in bold, excluding COPY: returning the input unchanged attains very high BLEU at near-zero style accuracy, a known artefact of BLEU-based TST evaluation ( Pang, 2019 ) . RT-SFT improves style accuracy over both baselines in all four domains and in every configuration tested, and relevant-example retrieval improves accuracy over random examples in all eight paired comparisons.
| Inference methods | MSD | Shakespeare | GYAFC | ParaDetox | ||||
| Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | |
| COPY | 0.107 | 14.01 | 0.009 | 15.35 | 0.149 | 16.54 | 0.012 | 20.03 |
| 5-shot In-Context Learning ( 2022 ) | 0.396 | 9.58 | 0.771 | 12.79 | 0.521 | 19.34 | 0.792 | 21.35 |
| Automatic Post-Editing ( 2022 ) | 0.672 | 12.21 | 0.693 | 11.90 | 0.435 | 17.74 | 0.720 | 17.35 |
| 0-shot inference | 0.730 | 13.31 | 0.791 | 17.21 | 0.595 | 23.96 | 0.909 | 17.07 |
| random 5-shot inference | 0.759 | 11.53 | 0.841 | 17.01 | 0.575 | 21.17 | 0.893 | 18.63 |
Table 3: TST performance under different inference procedures. All groups use one Llama-3.1-8B-Instruct RT-SFT-LLM, finetuned with 5-shot relevant examples and terminology retrieval (§ 4.2 ); baseline rows are repeated from Table 2 for reference. All RT-SFT rows apply RT-first inference except without RT normalization , which feeds the raw query. Best result per column in bold, excluding COPY. RT-first inference raises style accuracy and lowers BLEU in all four domains; relevant-example retrieval raises BLEU in all four while leaving accuracy largely unchanged.
| Pretrained LLMs | MSD | Shakespeare | GYAFC | ParaDetox | ||||
| Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | Acc. | BLEU-ref | |
| COPY | 0.107 | 14.01 | 0.009 | 15.35 | 0.149 | 16.54 | 0.012 | 20.03 |
| meta-llama/Llama-3.1-8B-Instruct | 0.763 | 13.79 | 0.842 | 17.35 | 0.612 | 24.17 | 0.912 | 19.91 |
| gorilla-llm/gorilla-openfunctions-v2 | 0.661 | 11.07 | 0.733 | 17.11 | 0.573 | 21.15 | 0.874 | 19.93 |
| mistralai/Mistral-7B | 0.692 | 12.21 | 0.749 | 15.97 | 0.552 | 22.23 | 0.806 | 18.45 |
| facebook/opt-2.7b | 0.631 | 11.39 | 0.701 | 13.33 | 0.606 | 21.27 | 0.811 | 17.21 |
Table 4: TST finetuning performance across base LLMs (relevant 5-shot finetuning with terminology and name retrieval; synthetic corpus pooled from German and Chinese roundtrip pivots). The Llama-3.1-8B-Instruct row is the best configuration of Table 2 , repeated here for comparison. COPY returns the input unchanged and gives the untransferred floor for each domain’s style classifier. Best result per column in bold, excluding COPY. All four base models exceed that floor in every domain; Llama-3.1-8B-Instruct leads on style accuracy in all four and on BLEU in three.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Prompt Template Index | Prompt Template Text |
| I | Rewrite the following sentence into the style of [style name]. Here are [n] examples: Input: [example input i] . Output: [example output i] . Note that you may want to rewrite "[input term]" to "[output term]" for contextual consistency. Now go ahead: Input: [query input] . The [style name] output: |
| II | Rewrite the following sentence into the style of [style name]: Input: [query input] . Here are [n] examples: Input: [example input i] . Output: [example output i] . Note that word [input term] should be rewritten to [output term] for contextual consistency. |
| III | Note that word [input term] should be rewritten to [output term] for contextual consistency. General domain: [example input i] . [style name] domain: [example output i] . general domain: Input: [query input] . [style name] domain: |
Table 5: Prompts for TST finetuning
| Template | MSD | Shakespeare | ||
| Acc. | BLEU-ref | Acc. | BLEU-ref | |
| COPY | 0.107 | 14.01 | 0.009 | 15.35 |
| APE ( 2022 ) | 0.672 | 12.21 | 0.693 | 11.90 |
| Template I | 0.763 | 13.79 | 0.842 | 17.35 |
| Template II | 0.614 | 12.91 | 0.773 | 16.69 |
| Template III | 0.721 | 13.63 | 0.751 | 18.01 |
Table 6: BLEU and acc. score across the MSD and Shakespeare domains for three potential templates. Template (I) has consistently higher style accuracy score, indicating robust ability in transferring to target style. Both Template (I) and (III) have stably high BLEU score.
| Prompt type | Prompt Text |
| First round | Identify terminologies or character names in the sentence and return in comma separated format, without any additional explanation. Sentence: [source-side sentence] . Terminologies and names: |
| Second round | Find the counterpart of the word [source-side retrieved word] in the following sentence and return a single word, without any additional explanation. Sentence: [target-side sentence] : |
Table 7: Prompts for terminology RAG