cs.CLApr 2, 2026
SaveTaming CATS: Controllable Automatic Text Simplification through Instruction Fine-Tuning with Control Tokens
Organizations: Department of Computational Linguistics, University of Zurich, Switzerland · Department of Clinical Research, University of Bern, Switzerland
Abstract
Controllable Automatic Text Simplification (CATS) produces user-tailored outputs, yet controllability is often treated as a decoding problem and evaluated with metrics that are not reflective to the measure of control. We observe that controllability in ATS is significantly constrained by data and evaluation. To this end, we introduce a domain-agnostic CATS framework based on instruction fine-tuning with discrete control tokens, steering open-source models to target readability levels and compression rates. Across three model families with different model sizes (Llama, Mistral, Qwen; 1-14B) and four domains (medicine, public administration, news, encyclopedic text), we find that smaller models (1-3B) can be competitive, but reliable controllability strongly depends on whether the training data encodes sufficient variation in the target attribute. Readability control (FKGL, ARI, Dale-Chall) is learned consistently, whereas compression control underperforms due to limited signal variability in the existing corpora. We further show that standard simplification and similarity metrics are insufficient for measuring control, motivating error-based measures for target-output alignment. Finally, our sampling and stratification experiments demonstrate that naive splits can introduce distributional mismatch that undermines both training and evaluation.
Figures & tables
Figure 1: Mean control-attribute values across models and datasets. Fine-tuned models tend to generate text that is simpler than the source but more complex than the target. Sentence-aligned datasets ( Med-EASi, SimPA, WikiLarge ) show very little compression signal (around 1.0).
Figure 2: Both SimPA and Newsela show larger distribution shifts by readability than by length transformations. Text-aligned Newsela shows greater distribution shift than the sentence-aligned SimPA.
Figure 4: WikiLarge . Global sampling with stratification by readability and length, measured in terms of KS, EMD and JSD. Stratification by chars shows smallest divergence across all metrics.
| SARI | LENS | MAE | |||
| 1B | 8B | 1B | 8B | 1B | 8B |
| Med-EASi FKGL | |||||
| 43.68 | 36.20 | 59.03 | 68.95 | 3.37 | 4.89 |
| 42.64 -1.04 | 43.99 +7.77 | 57.33 -1.70 | 59.83 -9.12 | 3.96 +0.51 | 2.91 ? |
| Med-EASi char | |||||
| 32.38 | 36.76 | 21.23 | 62.89 | 1.23 | 0.37 |
Table 2: IFT of 1B and 8B Llama models improves SARI and MAE over the non-fine-tuned baseline, with stronger gains for larger models and compression-based controls, whereas LENS does not consistently improve and in some cases decreases after fine-tuning.
Table 3: Scaling experiment with a broader parameter size range. Rows in green represent results on MED-EASi , whereas rows in blue represent results on SimPA . Top value per row and metric is in bold, second-best value is underscored. (a) - (c) FKGL, (d) - (f) char compression.
Figure 6: Dataset: SimPA . Control attribute: FKGL. Scaling does not always boost performance. However, we observe strong positive correlation between SARI and COMET, and a strong negative correlation between SARI/COMET and error-based metrics across most datasets and control attributes.
| Data | Ctrl. | SARI | LENS | MAE | |||||||||||||||||||||||
| Llama | Mistral | Qwen | Llama | Mistral | Qwen | Llama | Mistral | Qwen | |||||||||||||||||||
| 1B | 8B | 3B | 7B | 1.7B | 8B | 1B | 8B | 3B | 7B | 1.7B | 8B | 1B | 8B | 3B | 7B | 1.7B | 8B | ||||||||||
| Med-EASi | |||||||||||||||||||||||||||
| base | ARI | 42.47 | 43.47 | 44.78 | 50.90 | 50.32 | 50.55 | 56.89 | 60.17 | 50.20 | 59.28 | 58.74 | 59.33 | 4.39 | 3.35 | 8.19 | 3.30 | 3.63 | 2.86 | ||||||||
| base | DC | 42.04 | 44.03 | 43.85 | 52.18 | 50.29 | 50.49 | 55.21 | 59.64 | 50.89 | 57.32 | 57.14 | 57.62 | 1.54 | 1.27 | 1.66 | 1.29 | 1.39 | 1.16 | ||||||||
| base | char | 42.43 | 45.25 | 44.86 | 52.03 | 52.09 | 51.91 | 55.74 | 59.83 | 51.25 | 57.42 | 60.54 | 59.94 | 0.39 | 0.21 | 0.70 | 0.28 | 0.18 | 0.19 | ||||||||
Table 4: Main experiment results show strong performance of the fine-tuned Qwen models closely followed by the larger Mistral. “DC” stands for Dale-Chall. “Mono” refers to the monotonically filtered subsets of the respective dataset; “base” refers to its full version. “Word” and “char” denote word- and character-level compression ratio, respectively. Higher is better for SARI and LENS; lower is better for MAE.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Model performance on Newsela .
Figure 8: Model performance on Med-EASi .
Figure 9: Model performance on SimPA .
Figure 10: Model performance on WikiLarge .
Figure 11: Dataset: SimPA
Figure 16: Pearson correlation heatmaps analyzing metric correlation, aggregated across all models. Dataset: Newsela .
Figure 17: Pearson correlation heatmaps analyzing metric correlation, aggregated across all models. Dataset: Med-EASi .
Figure 18: Pearson correlation heatmaps analyzing metric correlation, aggregated across all models. Dataset: SimPA .
Figure 19: Pearson correlation heatmaps analyzing metric correlation, aggregated across all models. Dataset: WikiLarge .
| DC | Text | |
|---|---|---|
| Source | 9.5 | “In public saunas it is more common that men and women go to the sauna separately, although people of both sexes may sometimes bathe together, for example in student clubs. Still, saunas are not associated with sex and sexuality”. |
| Reference | 5.7 | “Some saunas are only for men , or only for women”. |
| Llama-3.2-1B-Instruct | 11.1 | “The sauna is not associated with sex and sexuality”. |
| Llama-3.1-8B-Instruct | 7.3 | “The sauna is a place where people go to relax”. |
| Ministral-3b-instruct | 9.3 | “In public saunas it is more common that men and women go to the sauna separately, although people of both sexes may occasionally bathe together”. |
| Mistral-7B-Instruct-v0.1 | 9.8 | “In public saunas it is more common that men and women go to the sauna separately, although people of both sexes may sometimes bathe together, for example in student clubs”. |
Table 5: Dataset: WikiLarge . Control Attribute: Dale-Chall. Instance ID: wikilarge_ori_252807. In this example, we see how different models focus on different information in the source text, resulting in very different simplifications. Further, reference simplification is much shorter than the source text and omits essential information. This is an example of a poor reference text, with lower adequacy and fluency than the automatically generated simplifications by various models.
| ARI | Text | |
|---|---|---|
| Source | 20.1 | “Symptoms include a sense of fullness, heaviness, aching, cramps, pain, tiredness, and paresthesias in the legs; these symptoms worsen with standing or walking and are relieved by rest and elevation”. |
| Reference | 15.2 | “The swelling may not cause any symptoms, but some people feel fullness, heaviness, aching, cramps, pain, tiredness, and tingling in the legs”. |
| Llama-3.2-1B-Instruct | 11.8 | “The symptoms include a sense of fullness, heaviness, aching, cramps, pain, tiredness, and numbness in the legs. These symptoms worsen with standing or walking and are relieved by rest and elevation”. |
| Llama-3.1-8B-Instruct | 19.6 | “The symptoms include a sense of fullness, heaviness, aching, cramps, pain, tiredness, and numbness in the legs; these symptoms worsen with standing or walking and are relieved by rest and elevation”. |
| Ministral-3b-instruct | 10.7 | “Symptoms include a sense of fullness, heavyness, aching, cramps, and tiredness. They worsen with standing or walking and are relieved by rest and elevation”. |
| Mistral-7B-Instruct-v0.1 | 19.5 | “Symptoms include a sense of fullness, heaviness, aching, cramps, pain, tiredness, and tingling in the legs; these symptoms worsen with standing or walking and are relieved by rest and elevation”. |
Table 6: Dataset: Med-EASi . Control Attribute: ARI. Instance ID: medeasi_000434. Simplifications generated by the model are very similar, further highlighting the effect of sentence splitting on the ARI score. Llama-3.2-1B-Instruct and Ministral-3b-instruct split the long sentence into two and drastically reduce the ARI score without any further transformations.
Figure 20: Formatted prompt examples.
| Hyperparameters | Shared Hyperparams | |
|---|---|---|
| batch size | 4 | |
| grad. accumulation steps | 4 | |
| cumulative batch size | 16 | |
| weight decay | 0.01 | |
| warmup steps | 30 | |
| max epochs | 3 | |
Table 7: Left: training hyperparameters selected for smaller ( 4B) models, fine-tuned without PEFT. Right: training hyperparameters selected for larger ( 4B) models, fine-tuned with PEFT.
| Experiment | Seeds |
|---|---|
| Strat. Partitioning | 2746317213, 478163327, 107420369, 3184935163, 1181241943, 1051802512, 958682846, 599310825, 3163119785, 440213415 |
| Downsampling | 69, 1, 40, 7, 29, 48, 78, 34, 67, 84 |
| Robust LLM Eval. | 37, 15, 96, 2, 28 |
Table 8: Random seeds grouped by experiment, with results are aggregated across multiple.
| No PEFT | PEFT |
|---|---|
| ⬇ method : grid metric : goal : minimize name : eval_loss parameters : learning_rate : values : - 1 e -06 - 5 e -06 - 1 e -05 - 5 e -05 max_grad_norm : values : - 0.5 - 1 - 2 weight_decay : values : - 0 - 0.01 - 0.1 | ⬇ method : grid metric : goal : minimize name : eval_loss parameters : learning_rate : values : - 1 e -05 - 5 e -05 - 0.0001 - 0.0002 lora_dropout : values : - 0.05 - 0.1 lora_r : values : - 4 - 8 - 16 max_grad_norm : values : - 0.5 - 1 |
Table 9: Hyperparameter optimization on Weights&Biases platform. Left: sweep configuration for smaller ( 4B) models, finetuned without PEFT. Right: sweep configuration for larger ( 4B) models, finetuned with PEFT (LoRA).