ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection

He Geng; Yangmin Huang; Lixian Lai; Qianyun Du; Hui Chu; Zhiyang He; Jiaxue Hu; Xiaodong Tao

cs.AIApr 9, 2026

ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection

Authors: He Geng, Yangmin Huang, Lixian Lai, Qianyun Du, Hui Chu, Zhiyang He, Jiaxue Hu, Xiaodong Tao

Organizations: Xunfei Healthcare Technology Co., Ltd.

Abstract

Aligning Large Language Models (LLMs) with high-stakes medical standards remains a significant challenge, primarily due to the dissonance between coarse-grained preference signals and the complex, multi-dimensional nature of clinical protocols. To bridge this gap, we introduce ProMedical, a unified alignment framework grounded in fine-grained clinical criteria. We first construct ProMedical-Preference-50k, a dataset generated via a human-in-the-loop pipeline that augments medical instructions with rigorous, physician-derived rubrics. Leveraging this corpus, we propose the Explicit Criteria Injection paradigm to train a multi-dimensional reward model. Unlike traditional scalar reward models, our approach explicitly disentangles safety constraints from general proficiency, enabling precise guidance during reinforcement learning. To rigorously validate this framework, we establish ProMedical-Bench, a held-out evaluation suite anchored by double-blind expert adjudication. Empirical evaluations demonstrate that optimizing the Qwen3-8B base model via ProMedical-RM-guided GRPO yields substantial gains, improving overall accuracy by 22.3% and safety compliance by 21.7%, effectively rivaling proprietary frontier models. Furthermore, the aligned policy generalizes robustly to external benchmarks, demonstrating performance comparable to state-of-the-art models on UltraMedical. We publicly release our datasets, reward models, and benchmarks to facilitate reproducible research in safety-aware medical alignment.

Figures & tables

Figure 2: Overview of the ProMedical framework. (Left) Construction of the ProMedical-Preference-50k dataset via a human-in-the-loop pipeline that transforms coarse medical instructions into fine-grained, rubric-enriched training samples. (Top Right) The proposed Medical-Rubric GRPO paradigm, which leverages a Rubric-Aware Reward Model to calculate hierarchical reward scalars based on Main Proficiency ( $S_{1}$ ), Excellence Bonus ( $S_{2}$ ), and Safety Veto ( $S_{3}$ ) to steer policy alignment. (Bottom) The ProMedical-Bench evaluation suite, establishing a robust clinical gold standard through double-blind expert adjudication with rubric-wise rationales.

Figure 3: An illustrative example of the ProMedical annotation paradigm. Given a clinical instruction, the framework instantiates fine-grained rubrics across Proficiency, Excellence, and Safety dimensions to guide the hierarchical preference adjudication and generate rubric-wise rationales.

Closed-Source Generative Models
Model	Pointwise			Pairwise			Binary
Model	Proficiency	Excellence	Safety	Proficiency	Excellence	Safety	Overall
GPT-5	91.50	90.88	76.45	92.06	91.94	77.39	76.42
Gemini-3-Pro	89.80	91.20	64.10	91.20	92.06	65.65	64.80
Open-Source Generative Models
Qwen3-235B-Thinking	88.40	87.90	78.10	89.10	88.50	79.20	77.45

Table 1: Performance benchmarks on ProMedical-Bench. We report evaluations across three modalities: Pointwise scores, Pairwise comparison accuracy, and Binary overall ranking accuracy. Metrics include Proficiency (

S_{1}

), Excellence (

S_{2}

), and Safety Veto (

S_{3}

). Models marked with are medical-specific. Bold and underline indicate best and second-best performance. Note that due to the Safety Veto mechanism, the Overall accuracy is strictly bounded by the Safety performance.

Closed-Source Generative
Model	Precision	Recall	F1
GPT-5	79.24	73.85	76.45
Gemini-3-Pro	68.50	60.25	64.11
Open-Source Generative
DeepSeek-R1	81.50	76.28	78.80
Qwen3-235B-Thinking	80.15	76.10	78.07

Table 2: Safety Veto detection metrics on ProMedical-Bench . Precision, Recall, and F1-score are reported for the Safety dimension (

S_{3}

\star

denotes medical-specific models.

Method	Q ( $\uparrow$ )	Q+Criteria ( $\uparrow$ )	Q+Sub ( $\uparrow$ )
Ultra-Medical	80.53	-	-
RaR	79.03	80.10	81.32
InfiMed-ORBIT	80.85	81.07	81.63
ProMedical	81.94	82.32	83.60
ProMedical-RAG	81.60	83.20	84.28

Table 3: Performance comparison of rubric construction frameworks on the UltraMedical-Preference benchmark. We evaluate three fine-tuning configurations: Q, Q+Criteria, and Q+Sub, representing standard preference optimization, holistic rubric injection, and dimensional expansion, respectively.

Figure 4: Comparative assessment of policy alignment performance. We evaluate the generation capabilities of models aligned via GRPO using distinct reward signals. The ProMedical framework demonstrates superior efficacy, consistently surpassing baselines relying on holistic or implicit supervision.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Name	Description
MedQA Jin et al. (2021)	A large-scale dataset consisting of USMLE-style multiple-choice questions designed to assess professional medical knowledge and reasoning.
Medical-Eval-Sphere Hosseini et al. (2024)	A collection of realistic medical queries paired with high-quality, physician-annotated long-form responses.
PubMedQA Jin et al. (2019)	Biomedical QA pairs derived from research paper abstracts, comprising contexts, long reasoning answers, and boolean summaries.
DAHL Seo et al. (2024a)	High-quality exam questions generated from PMC research papers via GPT-4 and subsequently manually filtered for quality assurance.
Medical-Instruction-120k Mohammed-Altaf (2023)	A comprehensive compilation of medical instructions covering a wide range of topics including pharmacology, treatments, and wellness advice.
MedInstruct-52k Zhang et al. (2023b)	A diverse, machine-generated instruction-following dataset synthesized via GPT-4/ChatGPT based on high-quality expert-curated seeds.

Appendix

Table 4: Detailed breakdown of the open-source datasets aggregated in the initial phase of ProMedical construction. The datasets cover a wide range of tasks including exam questions, clinical dialogues, and instruction following.

Figure 5: Impact of difficulty curation on dataset complexity. We compare the average difficulty scores of the 11 constituent datasets before (dark blue) and after (light blue) filtering samples to the 5–9 range. The dashed red and dotted green lines represent the global average difficulty before (

5.06

) and after (

6.13

) curation, respectively. The widespread increase in average scores demonstrates that our strategy effectively filters out trivial instances, elevating the reasoning density across all data sources.

Figure 6: Hierarchical distribution of the ProMedical instruction taxonomy. The inner ring illustrates the five primary categories, dominated by Disease and Symptoms, while the outer ring displays the breakdown into sixteen fine-grained sub-categories.

Type	Primary Source	Share
False Positive (64%)	Permissive medical risk criteria	23%
False Positive (64%)	Ambiguous opening-sentence eval	19%
False Negative (34%)	Medical definition mismatch	17%
False Negative (34%)	Inconsistent disclaimer handling	10%

Appendix

Table 5: Error mode analysis of GPT-4.1 judgments prior to HITL intervention. Percentages are relative to total errors.

Specialty	%	Specialty	%
Internal Medicine	29.9	Orthopaedic Surgery	2.0
Neurology	6.6	Diagnostic Radiology	1.7
Pathology	6.5	Anesthesiology	1.5
Medical Genetics and Genomics	6.0	Thoracic Surgery	1.4
Psychiatry	6.0	Dermatology	1.4
Obstetrics and Gynecology	4.8	Neurological Surgery	1.2

Appendix

Table 6: Departmental distribution of ProMedical-Bench samples across 26 clinical specialties.

Figure 7: Distribution of evaluation criteria counts per instance. The box plot illustrates the variance in Core Criteria counts compared to the standardized Bonus and Veto Criteria.

Figure 8: Probability density of scalar weights in Core Criteria. The distribution highlights a design preference for granular, low-magnitude weights to ensure stable scoring aggregation.

Major Category	Sub-Category	Definition / Criteria
Disease and Symptoms	Disease	Knowledge that describes, explains, or manages a definite disease, syndrome, or specific pathological state with a recognized name.
Disease and Symptoms	Symptom & Sign	Knowledge explaining the meaning and etiology of independent symptoms (e.g., fever) or signs (e.g., hepatomegaly) not explicitly tied to a specific disease entity.
Treatment and Intervention	Drug	Knowledge describing specific active substances, dosage forms, or products aimed at medical intervention, including chemical and biological properties.
	Surgery	Knowledge describing specific, named invasive or interventional operational processes, including planning, execution, and management of surgical procedures.
	Others	(Aggregated) A collective category for low-frequency interventions ( $<0.5\%$ ), including Cosmetic Medicine , Chinese Materia Medica , Acupoint & Meridian , and Formula .
Inspection and Examination	Examination	Knowledge describing diagnostic tests or techniques (e.g., X-ray, gene tests) intended to produce measurable data, images, or molecular sequences.

Appendix

Table 7: The hierarchical taxonomy of the ProMedical-Bench . The 13 specific sub-categories are grouped into 5 major categories based on clinical domains. These definitions served as the system instructions for the classification task.

Figure 9: Composition of evaluation dimensions. The chart reflects a balanced focus with a primary emphasis on Completeness and Accuracy.

Generative Baselines (Proprietary)
Model Designation	Source / Checkpoint ID
GPT-5	gpt-5
Gemini-3-Pro	gemini-3-pro
Claude-4.5-Thinking	claude-sonnet-4.5-thinking
Doubao-1.6-thinking	doubao-1.6-thinking
Gemini-3-Pro-Thinking	gemini-3-pro-thinking

Appendix

Table 8: Detailed model specifications used in experiments.

Figure 10: Pairwise Preference Accuracy across Model Tiers. ProMedical-RM-8B(Qwen3) (Red) significantly outperforms open-source baselines (Light Blue), effectively bridging the gap to proprietary frontier models (Dark Blue) despite orders of magnitude fewer parameters.

Figure 11: Safety Veto (

S_{3}

) vs. Overall Pairwise Accuracy. This scatter plot illustrates the trade-off between safety and utility within the pairwise preference ranking task. The clustering of open-source baselines in the bottom-right quadrant signifies a susceptibility to reward hacking, where general utility is prioritized at the expense of safety compliance. In contrast, ProMedical-RM-8B (Qwen3) (Red) aligns with GPT-5 along the ”High-Compliance” frontier, corroborating the robustness of our hard-constraint Veto mechanism.

Figure 12: Disaggregated performance profiles across five clinical dimensions. We benchmark ProMedical-RM-8B (Qwen3) (red) against 14 baselines, demonstrating balanced proficiency across all axes in contrast to the dimensional skew of general-purpose models. For a comparative radar chart against UltraMedical-8B , refer to Figure .

Metric	Llama-based	Qwen3-based
Pointwise Prof.	90.15	90.85
Pointwise Excel.	91.90	92.80
Pointwise Safe.	87.20	88.50
Pairwise Prof.	89.65	90.26
Pairwise Excel.	91.25	92.06
Pairwise Safe.	86.10	87.39

Appendix

Table 9: Backbone-agnostic validation on ProMedical-Bench.

Method	Q	Q+Criteria	Q+Sub
UltraMedical (Base)	80.53	–	–
RaR	80.45	80.88	81.25
InfiMed-ORBIT	80.90	81.42	81.96
ProMedical	81.86	82.50	83.14
ProMedical-RAG	81.95	83.25	84.17

Appendix

Table 10: Performance on UltraMedical with Llama-3-8B backbone. Results for the Qwen3-8B backbone are reported in Table 1 .

Figure 13: Hyperparameter sensitivity analysis on ProMedical-Bench. (a) Semantic deduplication exhibits a convex trajectory, peaking at a 10% removal rate. (b) Difficulty filtering demonstrates a similar trend, where the [5-9] interval strikes the optimal balance between reasoning density and data sufficiency. Both experiments confirm the necessity of moderate, judicious curation.

Paradigm Ablation
Model Variant	Pairwise	Prof.	Excel.	Safe.
ProMedical-RM (Full)	88.50	90.85	92.80	90.26
w/o Explicit Criteria	83.15	84.62	86.10	81.33
Data Ablation
w/o Excellence Data	87.12	90.50	85.40	89.95
w/o Safety Data	84.30	89.80	91.50	79.20

Appendix

Table 11: Ablation study of the Reward Model architecture on ProMedical-Bench . The removal of explicit criteria injection yields the most significant drop in overall pairwise accuracy. Notably, ablating the Safety Veto compromises compliance despite high utility, while removing the Bonus Margin significantly degrades excellence scores due to reward saturation.

Paradigm Ablation
Model Variant	Pairwise	Prof.	Excel.	Safe.
ProMedical-RM (Full)	88.50	90.85	92.80	90.26
w/o Explicit Criteria	83.15	84.62	86.10	81.33
Data/Dimension Ablation
w/o Excellence Data	87.12	90.50	85.40	89.95
w/o Safety Data	84.30	89.80	91.50	79.20

Appendix

Table 12: Ablation study of the Reward Model architecture on ProMedical-Bench. The removal of explicit criteria injection yields the most significant drop in overall pairwise accuracy. Notably, ablating the Safety Veto compromises compliance despite high utility, while removing the Bonus Margin significantly degrades excellence scores due to reward saturation.

Algorithm Comparison (w/ Explicit Signal)
Method	Prof. ( $S_{1}$ )	Excel. ( $S_{2}$ )	Safe. ( $S_{3}$ )	Overall
DPO	86.40	88.20	85.10	72.05
PPO	88.10	89.50	87.40	74.20
Supervision Paradigm (w/ GRPO)
Implicit (Scalar)	91.20	89.10	81.50	73.15
ProMedical (Ours)	90.85	92.80	90.26	76.39

Appendix

Table 13: Ablation analysis of Policy Optimization strategies on ProMedical-Bench . Explicit Criteria (Ours) achieves the optimal trade-off between proficiency and safety, whereas Implicit methods suffer from reward hacking.

Method	Input Tokens	Output Tokens
RaR	777.7	754.0
InfiMed-ORBIT	5,415.7	744.6
Ours	1,423.6	3,888.3

Appendix

Table 14: Average per-instance token consumption for rubric construction across methods.

Figure 14: Fine-grained performance breakdown on the MedBench subset. We evaluate the policy model on Chinese clinical sub-tasks covering diverse domains. Compared to the SFT baseline, ProMedical achieves consistent improvements, particularly in complex reasoning tasks like Psychiatric QA (+28.9%), demonstrating the cross-lingual robustness of our rubric-driven alignment.

Panel of Figure 15 — Figure 15: The instruction template used for the automated categorization task. The model is conditioned on the detailed definitions provided in Table 7 to generate the classification JSON.

Panel of Figure 16 — Figure 16: The prompt template used for the difficulty curation pipeline. We utilize this prompt to filter the dataset, retaining only samples with a complexity score between 5 and 9.

Panel of Figure 17 — Figure 17: The meta-prompt used to transform concise human expert rubrics into operationalized instructions for the ProMedical-RM. This step bridges the gap between expert intuition and machine-executable criteria.

Panel of Figure 18 — Figure 18: The instruction template used for evaluating the Main Proficiency dimension ( $S_{1}$ ). The model assesses compliance with essential clinical criteria (e.g., accuracy, completeness) derived from the expert rubrics.

Panel of Figure 19 — Figure 19: The instruction template used for evaluating the Excellence dimension. The model assesses whether the response meets specific bonus criteria (e.g., empathy, logical coherence) defined in the rubrics.

Panel of Figure 20 — Figure 20: The instruction template used for evaluating the Safety Veto dimension. Note the inverted logic: adherence to a veto rule (e.g., hallucination, toxicity) indicates a violation of safety standards, triggering a strict penalty.

Panel of Figure 21 — Figure 21: The prompt template used for pairwise preference adjudication. The model acts as an expert judge to compare two responses strictly based on the injected fine-grained criteria, mitigating the influence of length or stylistic bias.

Panel of Figure 22 — Figure 22: Case study on the iterative refinement of safety rubrics. By incorporating expert adjudication, we corrected the initial rubric that penalized necessary safety disclaimers (in Response A2) as ”persona breaks.” The refined rubric successfully disentangles professional tone from identity assertion.

Panel of Figure 23 — Figure 23: Detailed Case Study on ProMedical-Bench. Comparison of two model responses to a high-stakes prenatal query. Response A accurately addresses the complex medical history (LBWC vs. Trisomy) while maintaining appropriate boundaries. Response B , while highly empathetic and structurally superior, triggers a Safety Veto by hallucinating clinical experience (”15+ years”). This case illustrates how the Explicit Criteria Injection paradigm prevents ”reward hacking,” where models fabricate authority to appear more helpful.

Panel of Figure 24 — Figure 24: Case Study on mitigating Length Bias and Reward Hacking. Comparison of model responses to a diagnostic query (Acute Pancreatitis). While Response B (Baseline) exhibits high verbosity and detail, it fails to secure a preference advantage due to the framework’s prioritization of structural logic over text volume. Response A (ProMedical) is selected for its superior hierarchical organization ( $S_{1}$ ) and strict epistemic adherence ( $S_{3}$ ), validating that the alignment mechanism successfully differentiates clinical utility from generation length.

Panel of Figure 25 — Figure 25: Cross-lingual Generalization Case Study. Comparative analysis of responses to a Chinese query regarding Acute Angle-Closure Glaucoma (AACG). Response A (Baseline) provides generic, textbook-style advice with potential safety risks regarding self-medication. Response B (ProMedical-CN) demonstrates superior alignment by strictly enforcing specific contraindications (e.g., avoiding Valsalva maneuvers) and explaining the urgency via pathophysiological mechanisms. This validates that the rubric-driven alignment effectively transfers rigorous clinical logic across languages.

Panel of Figure 26 — Figure 26: Fine-Grained Dimension Analysis Case Study (Part I). Overview of the clinical context, model response, and detailed proficiency evaluation. The model demonstrates high competence in Accuracy and Contextual Awareness , identifying the user’s specific clinical picture (DOR + Young Age). However, it shows minor lapses in Completeness (missing formal disclaimer). Evaluation continued in Figure X+1.

Panel of Figure 27 — Figure 27: Fine-Grained Dimension Analysis Case Study (Part II). Continued from Figure X. Despite earning significant Excellence Bonuses ( $S_{2}$ ) for personalized guidance, the response triggers the Safety Veto ( $S_{3}$ ) due to Persona Impersonation . This illustrates the ”Reward Hacking” phenomenon, where high-performing models may resort to hallucinated authority to maximize utility scores, a behavior strictly penalized by our hierarchical alignment framework.

Panel of Figure 28 — Figure 28: Case Study: Granular Weighting in Context-Aware Crisis Intervention. Analysis of a response to a suicidal medical student in Thrissur. The visualization demonstrates how ProMedical’s non-uniform weighting schema prioritizes high-stakes criteria (e.g., Crisis Hotlines $w=0.15$ , Local Resources $w=0.08$ ) over lower-stakes stylistic dimensions. Despite a minor omission in national hotline names (Partial Adherence), the model’s precise legal citation and localization secure a high proficiency score.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

May 26, 2026

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood. We introduce MedHarm\footnote{Code and data will be released upon acceptance. Due to the sensitive nature of high-risk medical queries, data access will be available to qualified researchers upon request.}, a high-risk medical safety benchmark with 1,100 medically grounded queries across 10 safety-critical categories, including toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm. Unlike broad medical QA benchmarks, MedHarm targets realistic clinical, educational, and technical prompts that require refusal, caution, or safe redirection rather than direct helpfulness. We evaluate 15 LLMs spanning general-purpose, medical-purpose, closed-source, and downstream SFT models, together with 4 representative guardrail models. Results reveal a substantial gap between apparent alignment and medical safety: aligned models can still produce unsafe or actionable responses, medical fine-tuning can amplify harmful specificity, and external guardrails reduce some failures while introducing brittle blocking and weak safe helpfulness. These findings show that medical safety cannot be inferred from general alignment or medical capability alone, highlighting the need for domain-specific stress testing before deploying LLMs in safety-critical medical applications.

Yige Li, Jun Sun, Wei Zhao +5

Singapore Management University · Deakin University · The University of Melbourne +2

LLM Safety Benchmarks Healthcare LLM Guardrails LLM Safety Alignment LLM Safety Evaluation Medical LLMs

Ask PDF

May 27, 2026

SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models

Large language models(LLMs) increasingly match expert performance on licensing examinations, yet routine clinical use remains limited because governance requires auditable reasoning, safety and ethics alignment, and resilience to adversarial misuse. Here we present SafeMed-R1, trained with a traceable Clinical Trust Signals(CTS) pipeline that links each reasoning instance to clinician rubric scores and edit histories, and aligned through safety and ethics supervision and red team stress testing. SafeMed-R1 attains a macro-averaged accuracy of 79.6% across clinical benchmarks. Under adversarial safety testing, it shows the lowest aggregated risk and reduces unsafe outputs by about 3 to 5% relative to its baseline. In a paired expert study of 30 medication safety vignettes, SafeMed-R1 matches PGY1 and PGY2 residents on medical correctness and scores higher for medication safety, guideline consistency, and clinical usefulness. Collectively, these results suggest that clinician-audited supervision provenance, together with domain-tailored safety and ethics alignment, can strengthen governance-relevant evidence without relying on inference-time retrieval or citation grounding.

Chao Ding, Mouxiao Bian, Tianbin Li +12

Shanghai Artificial Intelligence Laboratory, Shanghai, China · Joint Laboratory of Biomedical Artificial Intelligence, Shanghai East Hospital, Tongji University School of Medicine, Shanghai, China · School of Computer Science and Technology, Tongji University, Shanghai, China +5

Healthcare LLM Auditing LLM Safety Alignment Adversarial Robustness of Language Models Clinical Language Model Evaluation Medical LLMs

Ask PDF

Jun 24, 2026

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.

Chang Wu, Junfeng Fang, Houcheng Jiang +5

Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy

LLM Alignment LLM Safety Alignment

Ask PDF

Abstract

LLM Alignment

Explore similar work

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models