Parameter-Efficient Conciseness Adaptation: Differentiable Length Control for Concise Clinical Reasoning in Smaller Language Models
Organizations: Department of Computer Science, COMSATS University Islamabad, Islamabad, Pakistan · School of Health and Behavioral Sciences, Bryant University, Smithfield, Rhode Island 02917, United States
Abstract
Smaller language models (SLMs) make it feasible to deploy clinical natural language processing where large proprietary models are too costly, and parameter-efficient fine-tuning allows them to be specialised on commodity hardware. One obstacle persists after training: on medical tasks these models answer at far greater length than the clinical question requires. In clinical decision support and point-of-care use, where time is short, such verbosity is hard to act on even when the underlying reasoning is sound. Existing length-control methods depend on preference optimisation, reward models, or proprietary teachers that undercut the accessibility making SLMs attractive. We introduce Parameter-Efficient Conciseness Adaptation (PECA), which adds a single differentiable conciseness term to the QLoRA supervised fine-tuning objective. Because the token count is discrete and non-differentiable, PECA instead raises the end-of-sequence probability once generation exceeds a token budget, a smooth signal optimised during ordinary training, with no preference data, no auxiliary model, and a single commodity GPU. On a 1.7-billion-parameter model fine-tuned on the medical-o1-reasoning-SFT corpus, PECA reduces mean response length by 72.8% while improving multiple-choice diagnostic accuracy from 40.8% to 44.0%. A sweep over the conciseness strength traces an interpretable accuracy-to-length frontier for balancing brevity and diagnostic completeness.
Figures & tables
| Condition | Mean len | Len red. | ROUGE-L | BERTScore | MCQ acc. |
|---|---|---|---|---|---|
| Vanilla SFT | 256.0 | 0.0% | 0.162 | 0.129 | 40.8% |
| PECA ( ) | 69.7 | 72.8% | 0.155 | 0.118 | 44.0% |
| PECA ( ) | 66.2 | 74.1% | 0.154 | 0.117 | 31.9% |
| PECA ( ) | 65.2 | 74.5% | 0.152 | 0.113 | 4.6% |
| Output | |
|---|---|
| Vanilla SFT (256 tokens, no diagnosis reached) | Alright, let’s think this through. We’ve got a 32-week pregnant woman who’s in the ER with continuous bleeding from the vagina. That’s definitely concerning. Now, she’s also saying the uterus is painful and tender. That’s a big clue. So, what could be causing this? First off, let’s consider the options. A circumvallate placenta is a type of placenta that’s usually not a problem in the first trimester, so it doesn’t seem likely here. Rupture of membranes is more about the membranes breaking, not necessarily causing bleeding like this. Placenta praevia would typically present with painless bleeding … [response truncated at generation limit; diagnosis never stated] |
| PECA | |
| ( =0.25) (68 tokens) | Alright, so we have a 32-week pregnant woman in the ER with continuous vaginal bleeding. The uterus is painful and tender, which is the key distinguishing sign. Placenta praevia typically causes painless bleeding, so that is less likely here. A painful, tender uterus with bleeding at this gestation points clearly toward accidental haemorrhage (placental abruption). |
| Method | Stage | Pref. data | Reward/teacher | Diff. term | Param. eff. |
|---|---|---|---|---|---|
| Vanilla LoRA/QLoRA [2, 12] | SFT (PEFT) | No | No | — | Yes |
| DPO, length-reg. [33] | Preference opt. | Yes | Ref. policy | No | No |
| SimPO [32] | Preference opt. | Yes | Ref.-free | No | No |
| LMPO [19] | Preference opt. | Yes | Ref.-free | No | No |
| EOS weighting [31] | SFT | No | No | Yes | No * |
| SmartThinker [37] | RL (GRPO) | No | Length reward | No | No |
| Attribute | Value |
|---|---|
| Total usable instances | 19,704 |
| Average question length (words) | 46.11 |
| Average CoT reasoning length (words) | 312.45 |
| Average response length (words) | 93.98 |
| Reasoning generation | Teacher LLM, verifier-validated |
| Source problems | MedQA-USMLE, MedMCQA |
| Split | Instances |
|---|---|
| Training | 8,000 |
| Validation | 1,000 |
| Test | 1,000 |
| Split | Mean | Median | 95th Percentile |
|---|---|---|---|
| Training | 523.7 | 507.5 | 686.1 |
| Validation | 525.9 | 509.0 | 701.1 |
| Test | 525.2 | 510.5 | 691.1 |
| Hyperparameter | Value |
|---|---|
| Rank ( ) | 16 |
| Scaling factor ( ) | 32 |
| Dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, |
| gate_proj, up_proj, down_proj | |
| Trainable parameters | 17,432,576 |
| Hyperparameter | Value |
|---|---|
| Optimizer | Paged AdamW 8-bit |
| Learning rate | |
| Weight decay | 0.01 |
| Effective batch size | 16 |
| Gradient accumulation steps | 16 |
| Max sequence length | 768 |