Verifiable Logic in Medical LLMs: A Critical Review of Agentic Clinical Reasoning Optimization and Neuro-Symbolic Grounding
Organizations: Artificial Intelligence Department, University of Prince Mugrin, Medina , 12435, Saudi Arabia · Robotics and Internet of Things Laboratory, Prince Sultan University, Riyadh , 11586, Saudi Arabia
Abstract
Clinical large language models are usually judged by whether the final answer is right. Yet a model can reach a fluent, plausible output through invalid reasoning that static benchmarks do not detect, and confident hallucination, causally invalid inference, and spurious demographic shortcuts can pass conventional evaluation. This critical review covers four paradigms in turn: how clinical hallucination arises and why static metrics miss it; the agentic hallucination cascade, in which multi-agent and tool-augmented systems propagate and amplify early reasoning errors; process reward modeling as step-level supervision; and neuro-symbolic grounding, in which formal logic engines, ontology reasoners, and knowledge graphs place verifiable constraints on neural output. Within their encoded domains, systems with symbolic veto layers offer guarantees that hold relative to their encoded knowledge, and they report large gains on the safety measures they use. These results come from single studies, mostly in simulated or small-sample settings, and are not directly comparable with neural baselines. Process reward models remain bounded by the statistics of their training data. We propose a conceptual framework that maps each agentic failure mode to a symbolic intervention point, and we argue that clinical AI safety depends substantially on system architecture. Because current regulation emphasizes performance and does not require verification of reasoning steps, we argue that responsible certification should also audit reasoning trajectories alongside accuracy on static benchmarks.
Figures & tables
| Study | Method | Scope | Agentic | Neuro-Sym. | Limitations |
| Lin et al. (2025) [ 5 ] | Survey (structured search) & Taxonomy | Taxonomy of medical LLM multi-agent systems. | Core Focus. | Partial. | Does not provide a unified framework in which symbolic reasoning constrains agent outputs. |
| Vatsal et al. (2026) [ 6 ] | Survey & Taxonomy | Evaluation taxonomy for medical LLM agents. | Core Focus. | Partial. | Does not incorporate neuro-symbolic integration or mechanisms to audit dynamic reasoning. |
| Wang et al. (2025) [ 7 ] | Survey | Architecture and challenges of clinical LLM agents. | Core Focus. | Partial. | Treats retrieval and symbolic grounding as equivalent and does not include a formal constraint layer. |
| Xu et al. (2025) [ 8 ] | Survey & Framework | Unification of LLMs and Knowledge Graphs. | Partial. | Core Focus. | Does not extend knowledge graph verification to multi-agent systems. |
| Gorenshtein et al. (2026) [ 9 ] | Systematic Review (PRISMA 2020) & Taxonomy | Taxonomy of neuro-symbolic integration patterns. | Partial. | Core Focus. | Views symbolic components primarily as static verifiers rather than active participants in agent workflows. |
| He et al. (2025) [ 10 ] | Survey | Roadmap of generative LLMs in healthcare. | Mentioned. | Partial. | Does not examine how dynamic agents can be integrated with rigorous neuro-symbolic verification. |
| Paradigm | n | Dominant Regime(s) | Dominant Metric Type | Representative Systems | Key Unresolved Gap |
| RQ1: Hallucination taxonomy | 29 | Not applicable (diagnostic, not verification-layer) | Hallucination or omission rate, F1 | mARC-QA P [ 27 ] , ShatterMed-QA PP [ 28 ] , MedHallu PP [ 29 ] , MedCaseReasoning PP [ 30 ] | Few benchmarks test counterfactual demographic invariance or adversarial robustness |
| RQ2: Agentic cascade | 23 | Not applicable (architectural, not verification-layer) | Task success rate, cascade or error-propagation rate | EHRAgent P [ 31 ] , MedAgentBench P [ 32 ] , MedAgentAudit PP [ 33 ] , TrustEHRAgent PP [ 34 ] | Calibration cliff: very low correct-answer yield under stringent reliability requirements |
| RQ3: PRM verification | 24 | Probabilistic learned, Structural process-level | Accuracy gain, F1, reasoning recall | Med-PRM P [ 16 ] , VPRM PP [ 35 ] , MedCEG PP [ 22 ] , MedS 3 P [ 36 ] | Evaluation infrastructure lags system development; calibration not integrated into PRM benchmarks |
| RQ4: Neuro-symbolic grounding | 22 | Deductive, Ontology-based, Evidence-grounded | Hallucination reduction (%), protocol adherence (%) | ProCDS P [ 13 ] , Ontology-grounded GraphRAG P [ 18 ] , GraphEval P † [ 37 ] , Explanation-Refiner P † [ 14 ] | Specification soundness, veto precision/recall, and temporal specification decay are unaudited |
| Regime | Representative Systems | Guarantee Strength | Bounding Factor | Sec. |
| Deductive (runtime) | ProCDS P [ 13 ] , CLP psychiatric CDSS PP [ 108 ] , Explanation-Refiner P † [ 14 ] , autoformalization policy checker PP † [ 118 ] | Conclusion entailed by the encoded rules and the supplied facts | Correctness of the encoded rules (including any the LLM generated), of the extracted facts, and of any formalization | 6 |
| Abductive (runtime) | ALP counter-abduction P † [ 119 ] | Explanatory adequacy and consistency, conditional on the hypothesized premises and the implemented preference ordering | Background theory, set of abducibles, preference ordering, and search depth bounds | 6 |
| Deductive (training-time rewards) | VPRM PP [ 35 ] , Med-U1 PP [ 98 ] | Rule-checked rewards during training; no runtime guarantee | Coverage of the encoded rules; residual reward hacking | 5 |
| Ontology-based | HermiT/Pellet DL reasoner PP [ 15 ] | Rejections sound under OWL/DL semantics; acceptance means consistent, not entailed, unless entailment is tested | How complete and up to date the ontology is; computing cost grows with expressivity | 6 |
| Evidence-grounded | Med-PRM P [ 16 ] , ontology-grounded GraphRAG framework P [ 18 ] , GraphEval P † [ 37 ] , Pusch and Conrad’s Cypher checker P [ 112 ] , MedKGI PP [ 113 ] , LogosKG PP [ 114 ] , KG-TRACES PP [ 115 ] , Agentic Graph RAG P [ 116 ] , Haibu MMIA PP [ 120 ] , corneal knowledge graph P [ 121 ] , MedReason PP [ 21 ] , Fleming-R1 PP [ 91 ] , process reward agents PP [ 17 ] | Auditable and updatable, but no formal entailment | How well the corpus or graph covers the topic, and retrieval quality | 5 , 6 |
| Probabilistic learned | Clinical note PRM P [ 19 ] , radiology report PRM PP [ 20 ] , HuatuoGPT-o1 PP [ 92 ] , MedS 3 P [ 36 ] , MiraMind/Mindora PP [ 94 ] , bilateral RL psychological reasoning P [ 96 ] , AMD fuzzy staging P [ 123 ] , decision tree/random forest oracles PP † [ 124 ] | Statistical, and depends on the data distribution | Coverage and quality of training or reward-annotation data | 5 , 6 |
| Decision Domain | Verifier | Expected Latency | Primary Veto Condition |
| Formal protocol coverage (drug dosing, surgical eligibility) | Prolog / CLP | Medium (iterative re-extraction) | Conclusion not derivable from the protocol rules and extracted facts, or an extracted fact not traceable to the source record |
| Structured ontological domain (rare disease phenotype, infectious disease semantics) | OWL DL | Potentially high; profile- and ontology-dependent | Candidate claim inconsistent with, or (where required) not entailed by, the ontology axioms |
| Multi-hop relational claim (differential diagnosis, drug interaction) | KG Verifier | Medium | Claim triple neither recorded in the KG nor implied by composable relations, or entity absent from the graph |
| Fully enumerable safety-critical rule set (e.g., a hemodialysis dosing protocol) | Executable rule engine | Expected to be minimal (not measured) | Any output that departs from the encoded rule set (checked on every query) |
| Out-of-domain query (no formal coverage available) | Abstention | N/A | Default fallback for all unrouted queries |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Abbreviations/Acronyms | Definition |
| AI | Artificial Intelligence |
| ALP | Abductive Logic Programming |
| CLP | Constraint Logic Programming |
| DL | Deep Learning; Description Logic (in OWL ontology contexts) |
| EHR | Electronic Health Record |
| FHIR | Fast Healthcare Interoperability Resources |
LLM Safety
Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.
See all 470 papers