Safety-Preserving Medical LLM Fine-Tuning via Bilevel Data Selection
Organizations: School of Mathematics and Statistics, Lanzhou University · Department of Statistics, University of California, Riverside · Department of Oncology, Lanzhou University Second Hospital
Abstract
As general large language models (LLMs) are increasingly adapted for patient-facing medical applications, improving medical capability while controlling patient-facing risks remains a critical challenge. Large-scale medical instruction data are essential for acquiring domain knowledge through supervised fine-tuning (SFT), but unsafe responses can undermine existing safety alignment. To address this problem, we propose Scalable Bilevel Data Selection (S-BDS) for safety-preserving medical SFT. By controlling selection-weight concentration, S-BDS enables reliable sample ranking over large candidate pools and identifies training examples that improve medical capability while controlling patient-facing risks. We combine clinician-validated safe data and selected data with unscreened medical instruction data for SFT, and jointly evaluate medical capability and patient-facing safety on HealthBench and PatientSafetyBench, respectively. Experiments across Llama and Qwen models show that S-BDS achieves the highest safe rates among all compared methods, outperforming the strongest baselines by 4.64 and 3.00 percentage points, respectively, while maintaining competitive medical capability. Further experiments show that varying the safety-oriented weight during SFT controls the trade-off between medical capability and patient-facing safety.
Figures & tables
| Method | Dataset | Base Model | Top-1% Mass | Top-5% Mass | Top-20% Mass | ESS/N | |
|---|---|---|---|---|---|---|---|
| SEAL | Orca | Llama3.1 | 14.37 | 42.15 | 64.16 | 0.9330 | 20.62% |
| S-BDS | Llama3.1 | 3.62 | 13.48 | 36.61 | 0.9903 | 78.25% | |
| S-BDS | Med | Llama3.1 | 2.20 | 9.09 | 29.99 | 0.9957 | 91.43% |
| S-BDS | Qwen3 | 2.20 | 9.02 | 29.90 | 0.9959 | 91.67% |
| Llama-3.1-8B-Instruct | Qwen3-8B-Instruct | |||
|---|---|---|---|---|
| Method | Health Score (%) | PSB Safe Rate (%) | Health Score (%) | PSB Safe Rate (%) |
| Standard SFT | 15.10 | 57.30 | 30.78 | 79.83 |
| Random | 14.58 | 62.88 | 31.23 | 81.76 |
| DSIR | 16.99 | 62.66 | 32.42 | 81.33 |
| LESS | 16.45 | 63.09 | 31.34 | 81.97 |
| SEAL | 18.85 | 64.67 | 31.32 | 82.62 |
| Llama-3.1-8B-Instruct | Qwen3-8B-Instruct | |||
|---|---|---|---|---|
| Safety Weight | Health Score (%) | PSB Safe Rate (%) | Health Score (%) | PSB Safe Rate (%) |
| 19.94 | 37.98 | 33.46 | 69.74 | |
| 20.51 | 42.71 | 33.89 | 71.89 | |
| 21.28 | 55.58 | 34.20 | 78.54 | |
| 20.10 | 64.16 | 33.19 | 82.62 | |
| 19.80 | 74.68 | 33.02 | 86.27 | |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Supervised Fine-Tuning | |
| Global batch size | 64 |
| Micro-batch size | 1 |
| Training epochs | 3 |
| Maximum sequence length | 2048 |
| LoRA rank | 16 |
| Risk Category | Definition | Examples of Unsafe Behaviors |
|---|---|---|
| Harmful or Dangerous Medical Advice | Providing medical advice that may directly increase health risks or encourage unsafe actions. | Recommending dangerous medication use, encouraging self-medication, suggesting harmful treatments, or failing to advise urgent care when necessary. |
| Misdiagnosis and Overconfidence | Making definitive medical conclusions without sufficient evidence or expressing excessive certainty about a user’s condition. | Providing a confirmed diagnosis based only on limited symptoms, ignoring uncertainty, or incorrectly ruling out serious conditions. |
| Unlicensed Practice of Medicine | Acting as a medical professional by providing individualized clinical decisions beyond the appropriate role of an AI assistant. | Prescribing specific medications, determining personalized dosages, or making treatment decisions without adequate clinical evaluation. |
| Health Misinformation | Providing medically unsupported, false, or misleading information that may influence users’ health decisions. | Promoting pseudoscientific treatments, endorsing unsupported medical claims, or presenting inaccurate health information as fact. |
| Discrimination in Medical Context | Producing biased or unfair medical recommendations based on sensitive personal attributes. | Making inappropriate judgments based on gender, ethnicity, age, weight, or other demographic characteristics. |
| Base Model | Method | Reviewed Responses | Agreement (%) | Cohen’s |
| Llama-3.1-8B Instruct | Standard SFT | 40 | 82.5 | 0.62 |
| Random | 40 | 87.5 | 0.74 | |
| DSIR | 40 | 82.5 | 0.60 | |
| LESS | 40 | 97.5 | 0.94 | |
| SEAL | 40 | 95.0 | 0.89 | |
| S-BDS | 40 | 90.0 | 0.78 |