Organizations: Department of Information Technology, Delhi Technological University, New Delhi, India. · Indian Institute of Technology, Guwahati, Guwahati, Assam, India. · Delhi Skill and Entrepreneurship University, New Delhi, India. · Central University of Haryana, Mahendragarh, Haryana, India.
Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models. Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales. Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference labels.We apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2,000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones. Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward. Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical content.We discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.
Figures & tables
Figure 1: PPO training dynamics over the Meerkat-derived medical prompt subset. The plot shows the average scalar reward across more than 160 PPO updates. Initially, rewards cluster around the minimum clamped value of 0.01, and later increase toward approximately 0.5 as the policy improves to generate longer and more varied outputs with appropriate punctuation under the proxy reward.
Figure 2: Evolution of reward components during PPO training. The increase in the length reward and punctuation bonus indicates that the model progressively learns to generate longer and more structured explanations under the proxy reward function.
Figure 3: Effect of PPO training on the average length of generated explanations. As PPO optimization progresses, the model gradually produces longer responses because the proxy reward encourages more detailed and structured outputs.
Component
Value
Base model
mistralai/Mistral-7B-Instruct-v0.2 Jiang et al. (2023)
Parameter count
7.26B (13.6M trainable with LoRA) Hu et al. (2021)
Dataset
Meerkat instruction subset (medical prompts) Kim et al. (2025)
Number of prompts
2,000
Max input length
256 tokens
Max generated tokens
200 tokens
Table 1: Training configuration for PPO fine-tuning of Mistral-7B-Instruct on the Meerkat-derived prompt subset.
Metric
Base Model
RL-Tuned Model
Average tokens per response
34.2
62.8
Distinct-1 (lexical diversity)
0.42
0.55
Average sentences per response
2.1
4.6
Table 2: Quantitative comparison of explanation style between the base model and the RL-tuned model.
Figure 4: Overview of the proposed reinforcement learning pipeline for medical SLMs. The model backbone is a Mistral-7B-Instruct model that is wrapped with a value head and adapters using LoRA. Prompts from a medical subset of the Meerkat instructions are fed into the model to generate responses that are then evaluated using a self-supervised reward function that measures the length, diversity, and punctuation of the responses. The adapter and value head parameters are updated using a KL penalty for the PPO objective.
Figure 5: Prompt-wise qualitative comparison for a persistent headache query. The RL-tuned model provides a more structured response, including symptom tracking and trigger identification in addition to general self-care recommendations.
Figure 6: Prompt-wise qualitative comparison for an insulin-related query. While both models correctly explain the role of insulin in diabetes, the RL-tuned model presents the response in a more organized way and extends it toward different insulin types and their functional use.
Figure 7: Prompt-wise qualitative comparison for a fever-related query. The RL-tuned model more clearly structures the explanation of fever causes and warning thresholds, making the response easier to interpret in a patient-facing context.
Figure 8: Prompt-wise qualitative comparison for a chest-pain-after-exercise query. The RL-tuned model expands the explanation of possible causes and more explicitly highlights symptom combinations that may indicate an urgent medical condition.
Figure 9: Prompt-wise qualitative comparison for an early-stroke-signs query. The RL-tuned model presents the warning signs in a more checklist-like and structured form, improving readability and making the response more directly actionable.
Prompt (short)
Base model (Mistral-7B-Instruct)
RL-tuned model (PPO + proxy reward)
Headache
Provides brief reassurance and suggests over-the-counter analgesics; limited structure and few explicit warning signs.
Produces a multi-paragraph answer with bullet-like sentences that distinguish benign from red-flag symptoms, explicitly advising urgent care for neurological deficits or sudden severe pain.
Insulin
Gives a high-level explanation of insulin as a hormone that lowers blood glucose, with limited mechanistic detail.
Adds a structured explanation with numbered points describing insulin’s role, types of insulin therapy, and consequences of omission, framed in patient-friendly language.
Fever
Mentions that fever is a sign of infection and lists a few symptoms; advice on when to seek care is brief.
Organizes the answer into sections (causes, home care, when to seek emergency care), explicitly listing temperature thresholds and concerning associated symptoms such as confusion or breathing difficulty.
Chest pain
Provides a generic warning that chest pain can be serious and mentions heart attack as a possibility without elaborating on differential diagnoses.
Enumerates multiple potential causes (musculoskeletal strain, reflux, angina, myocardial infarction) with distinguishing features and emphasizes immediate emergency evaluation when classic cardiac warning signs are present.
Stroke signs
Lists some neurological symptoms in a single long paragraph.
Presents a concise checklist of stroke warning signs, aligning with FAST-style mnemonics (face drooping, arm weakness, speech difficulty, time to call emergency services) and elaborating on each in separate sentences.
Table 3: Qualitative comparison of base vs RL-tuned model responses on clinical prompts. Descriptions are paraphrased summaries of the generated outputs.
Figure 10: Qualitative evaluation pipeline. A limited set of clinically relevant prompts is passed through both the base model and the RL-tuned model using the same decoding configuration. The responses are then qualitatively evaluated along dimensions such as structure, completeness, and clarity. The summarized comparison results are reported in Table 3 .
We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations - the extent to which a model's generated reasoning accurately reflects its internal decision-making process. While existing work focuses on evaluating faithfulness or using inference-time prompting frameworks to improve an LLM's self-explanation's tractability, these approaches do not provide a mechanism to directly optimize a model's parameters to generate faithful self-explanations. We bridge this gap by modifying existing faithfulness metrics into an RL training objective. We investigate (1) if models can be trained to accurately detect factors that affect their decisions, and (2) whether RL can directly optimize for the disclosure of these factors thereby improving LLM self-explanations' faithfulness. We experiment with two intervention types: random-word insertions and user-bias insertions, using a per-sample reward derived from the Phi-CCT correlation metric. RL fine-tuned Llama3.1-8B and Qwen3-8B show substantial improvements on the Phi-CCT faithfulness metric, with in-distribution scores rising from near-zero to as high as 0.664, and out-of-distribution scores reaching up to 0.691 on held-out tasks such as StrategyQA. Cross-intervention generalization is weaker but more interesting: a priori we would not expect a model trained only on random word insertions to generalize to user-bias phrases, yet Llama3.1-8B shows non-zero transfer in this direction. The reverse direction and Qwen3-8B do not replicate this, indicating model-dependent and setup-dependent effects we cannot yet explain. Lastly we analyze model behavior to rule out reward gaming behaviors that often plague RL training. Ultimately, we show that models can be trained to implicitly identify influential factors and disclose them, offering a scalable path toward reducing unfaithful reasoning in LLMs.
Yeoktatt Cheah, María Pérez-Ortiz, Noah Y. Siegel +1
Centre for AI, Department of Computer Science University College London · Imperial College London
In this work, we introduce MediX-R1, an open-ended Reinforcement Learning (RL) framework for medical vision language models (VLMs) that enables free-form answers beyond multiple-choice formats. MediX-R1 fine-tunes a baseline vision-language backbone with Group Based RL and a composite reward tailored for medical reasoning: an LLM-based accuracy reward that judges semantic correctness with a strict YES/NO decision, a medical embedding-based semantic reward to capture paraphrases and terminology variants, and lightweight format and modality rewards that enforce interpretable reasoning and modality recognition. This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable or MCQ-only rewards fall short. To evaluate performance, we employ a unified evaluation framework for both text-only and multimodal (image+text) tasks that uses a Reference-based LLM-as-judge in place of brittle string-overlap metrics, capturing semantic correctness, clinical accuracy, and contextual alignment. Our results show that open-ended RL with comprehensive reward signals can effectively strengthen medical reasoning in multimodal models. Despite using only ∼51K instruction examples, MediX-R1 achieves strong performance across standard medical LLM (text-only) and VLM (image + text) benchmarks, outperforming existing open-weight medical VLMs, with large gains particularly on open-ended clinical tasks. These improvements remain consistent across multiple RL algorithms and VLM backbones and are further supported by blinded evaluations with medical experts. Our trained models, curated datasets and source code are available at https://medix.cvmbzuai.com
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Jaseel Muhammad +6
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College
The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL<20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released§.
Md Rezwanul Haque, Md. Milon Islam, Fakhri Karray
Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, N2L 3G1, Ontario, Canada · Department of Computer Science and Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates