Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models
Organizations: Department of Information Technology, Delhi Technological University, New Delhi, India. · Indian Institute of Technology, Guwahati, Guwahati, Assam, India. · Delhi Skill and Entrepreneurship University, New Delhi, India. · Central University of Haryana, Mahendragarh, Haryana, India.
Abstract
Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models. Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales. Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference labels.We apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2,000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones. Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward. Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical content.We discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.
Figures & tables
| Component | Value |
|---|---|
| Base model | mistralai/Mistral-7B-Instruct-v0.2 Jiang et al. (2023) |
| Parameter count | 7.26B (13.6M trainable with LoRA) Hu et al. (2021) |
| Dataset | Meerkat instruction subset (medical prompts) Kim et al. (2025) |
| Number of prompts | 2,000 |
| Max input length | 256 tokens |
| Max generated tokens | 200 tokens |
| Metric | Base Model | RL-Tuned Model |
|---|---|---|
| Average tokens per response | 34.2 | 62.8 |
| Distinct-1 (lexical diversity) | 0.42 | 0.55 |
| Average sentences per response | 2.1 | 4.6 |
| Prompt (short) | Base model (Mistral-7B-Instruct) | RL-tuned model (PPO + proxy reward) |
|---|---|---|
| Headache | Provides brief reassurance and suggests over-the-counter analgesics; limited structure and few explicit warning signs. | Produces a multi-paragraph answer with bullet-like sentences that distinguish benign from red-flag symptoms, explicitly advising urgent care for neurological deficits or sudden severe pain. |
| Insulin | Gives a high-level explanation of insulin as a hormone that lowers blood glucose, with limited mechanistic detail. | Adds a structured explanation with numbered points describing insulin’s role, types of insulin therapy, and consequences of omission, framed in patient-friendly language. |
| Fever | Mentions that fever is a sign of infection and lists a few symptoms; advice on when to seek care is brief. | Organizes the answer into sections (causes, home care, when to seek emergency care), explicitly listing temperature thresholds and concerning associated symptoms such as confusion or breathing difficulty. |
| Chest pain | Provides a generic warning that chest pain can be serious and mentions heart attack as a possibility without elaborating on differential diagnoses. | Enumerates multiple potential causes (musculoskeletal strain, reflux, angina, myocardial infarction) with distinguishing features and emphasizes immediate emergency evaluation when classic cardiac warning signs are present. |
| Stroke signs | Lists some neurological symptoms in a single long paragraph. | Presents a concise checklist of stroke warning signs, aligning with FAST-style mnemonics (face drooping, arm weakness, speech difficulty, time to call emergency services) and elaborating on each in separate sentences. |