cs.AINov 18, 2026

Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models

Authors: Prakhar Mishra, Bhavya Yajush Awasthi, Tanuja, Nishant

Organizations: Department of Information Technology, Delhi Technological University, New Delhi, India. · Indian Institute of Technology, Guwahati, Guwahati, Assam, India. · Delhi Skill and Entrepreneurship University, New Delhi, India. · Central University of Haryana, Mahendragarh, Haryana, India.

Abstract

Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models. Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales. Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference labels.We apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2,000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones. Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward. Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical content.We discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.

Figures & tables

Explore similar work

CardsList
  1. Training Large Language Models for Self-Explanation Faithfulness

    Jul 23, 2026Yeoktatt Cheah, María Pérez-Ortiz, Noah Y. Siegel +1Faithfulness of Language Model ExplanationsLLM Interpretability

  2. MediX-R1: Open Ended Medical Reinforcement Learning

    Feb 26, 2026Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Jaseel Muhammad +6RL for VLMsMedical VLMs

  3. Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

    Jul 27, 2026Md Rezwanul Haque, Md. Milon Islam, Fakhri KarrayRL for Language ModelsLLM Alignment