A Discordance-Aware Multimodal Framework with Multi-Agent Clinical Reasoning

Pegah Ahadian; Mingrui Yang; Sixu Chen; Xiaojuan Li; Qiang Guan

cs.LGMar 12, 2026

A Discordance-Aware Multimodal Framework with Multi-Agent Clinical Reasoning

Authors: Pegah Ahadian, Mingrui Yang, Sixu Chen, Xiaojuan Li, Qiang Guan

Organizations: Department of Computer Science, Kent State University, Ohio, USA · Department of Biomedical Engineering, Cleveland Clinic, Ohio, USA

Abstract

Knee osteoarthritis frequently exhibits discordance between structural damage observed in imaging and patient-reported symptoms such as pain. This mismatch complicates clinical interpretation and patient stratification and remains insufficiently modeled in existing decision support systems. We propose a discordance aware multimodal framework that combines machine learning prediction models with a tool grounded multi agent reasoning system. Using baseline data from the FNIH Osteoarthritis Biomarkers Consortium, we trained multimodal models to predict two progression tasks, joint space loss only progression versus non progression, and pain only progression versus non progression. The predictive system integrates three modality specific experts: a CatBoost tabular model using demographic, radiographic, MRI-derived scalar, and biomarker features; MRI image embeddings extracted using a ResNet18 backbone; and Xray embeddings derived from the same architecture. Expert predictions are fused using a stacking ensemble. Residual based models estimate expected pain from structural features, enabling the computation of a pain structure discordance score between observed and expected symptoms. A multi-agent reasoning layer interprets these signals to assign clinically interpretable OA phenotypes and generate phenotype specific management recommendations.

Figures & tables

Characteristic	Category / value	Frequency (%)
Knees (records)	Total knees in Clinical_FNIH	600 (100.0)
Side	Right knee	322 (53.7)
	Left knee	278 (46.3)
CASE / GROUPTYPE	JSL and pain progressor (Case = 1)	194 (32.3)
	JSL only progressor (Case = 2)	103 (17.2)
	Pain only progressor (Case = 3)	103 (17.2)

Table 1: Baseline demographic and clinical characteristics of the FNIH OA Biomarkers (Clinical_FNIH dataset [ 7 ] ).

Figure 1: Discordance modeling and phenotype assignment. Structural variables are used to estimate expected symptom burden. The difference between observed and expected pain produces a discordance score that is then interpreted by the multi-agent reasoning layer.

Figure 2: Overview of the discordance-aware multi-agent framework. Multimodal clinical, radiographic, MRI, and biomarker inputs are processed by tool-grounded predictive modules to estimate structural progression risk and expected symptom burden. A Structuralist agent interprets structural evidence, a Physiologist agent interprets symptom-structure discordance, and a Lead Consultant synthesizes both views to assign an OA phenotype and generate phenotype-specific management recommendations.

Configuration	Clinician Approval Rate	Interpretation Quality (1–5)
A0 Deterministic template	–	–
A1 Single-agent LLM	0.78	3.9
A2 Decomposed multi-agent	0.84	4.2
A3 Debate-enabled multi-agent	0.88	4.4
A4 No-signal control	0.82	4.0

Table 2: Clinician-grounded evaluation of reasoning architectures.

Case	Phenotype	$p_{struct}$	$\hat{y}_{pain}$	$D_{PS}$	Interpretation
9058692	Concordant Severe	0.79	4.01	-0.01	High structural risk with symptom concordance
9075900	Concordant Severe	0.91	0.07	-0.07	High structural risk with symptom concordance
9118061	Concordant Mild	0.03	1.02	-0.02	Low structural risk with symptom concordance

Table 3: Representative cases illustrating pain–structure concordance and discordance.

Feature set / Method	JSL Only vs Non				Pain Only vs Non
Feature set / Method	AUC	AP	[email protected]	[email protected]	AUC	AP
Demographics only (CatBoost)	0.496	0.334	0.453	0.291	0.484	0.361
Demographics + MRI embeddings	0.570	0.438	0.526	0.404	0.413	0.296
Demographics + X-ray embeddings	0.628	0.443	0.602	0.496	0.456	0.310
Demographics + image-derived scalars	0.683	0.516	0.592	0.427	0.479	0.323
Demographics + scalars + embeddings (T+M+X stacking)	0.675	0.508	0.624	0.524	0.607	0.470

Table 4: Progression prediction performance across multimodal feature configurations. Results are reported using 5-fold stratified cross-validation with out-of-fold evaluation.

Agent	Illustrative reasoning output
Structuralist	“The radiographic indicators show moderate-to-high structural burden. The knee exhibits elevated Kellgren–Lawrence grade and reduced joint space width, and the multimodal model assigns a relatively high structural progression probability ( $p_{\text{struct}}$ ). From a structural standpoint, this knee appears at elevated risk for future joint-space loss. Even though current symptoms are not severe, the imaging profile suggests that structural degeneration is already present and may continue to progress.”
Physiologist	“The patient’s reported pain score is lower than the level predicted from the structural profile. When comparing observed pain $y_{\text{pain}}$ with the predicted pain estimate $\hat{y}_{\text{pain}}$ , the resulting discordance score $D_{PS}=y_{\text{pain}}-\hat{y}_{\text{pain}}$ is negative. This indicates that the patient currently experiences less pain than expected given the structural severity. Such cases may reflect pain resilience, early structural disease without symptomatic manifestation, or variability in individual pain perception. I therefore interpret this case as structure-driven rather than pain-driven.”
Lead Consultant	“The structural evidence and symptom evidence point in different directions. Structural risk is elevated according to imaging features and the structural progression model, whereas the patient’s pain burden remains comparatively low relative to expectation. Because the discordance score indicates structure $>$ pain, I classify this knee as a structure-dominant phenotype rather than a symptom-dominant phenotype. The management implication is to prioritize monitoring of structural progression and preventive strategies, such as weight management, joint-protective exercise, and imaging follow-up, rather than escalating symptom-focused interventions alone.”

Table 5: Illustrative specialist dialogue around pain–structure discordance.

Feature	Mean Importance
V00AGE	8.779630
Boneshape_V00nPatellaOAVector	5.624315
Labcorp_V00Urine_Col21N2SD	5.207179
Labcorp_V00Serum_CPII_lc	4.546927
Biomediq_V00LateralTibialCartilage	3.804936
Labcorp_V00Urine_C2C_lc	3.511122

Table 6: Top-20 tabular features ranked by mean CatBoost importance (across folds). Pain vs Non

Feature	Mean Importance
Biomediq_V00LateralMeniscus	13.617486
Biomediq_V00MedialMeniscus	8.246588
Labcorp_V00Serum_Comp_lc	6.332388
Boneshape_V00nFemurOAVector	4.334102
Labcorp_V00Urine_CTXII_lc	3.796752
Labcorp_V00Serum_CTXI_lc	3.769110

Table 7: Top-20 tabular features ranked by mean CatBoost importance (across folds). JSL vs Non

Figure 3: Multimodal progression prediction architecture. MRI and X-ray images are encoded into deep embeddings using ResNet18, reduced with PCA, and classified by logistic regression. Structured demographic, scalar imaging, and biomarker features are modeled by CatBoost. The final progression estimate is produced by a stacking ensemble that combines modality-specific predictions

Explore similar work

OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression
Oct 8, 2026Sixu Chen, Mingrui Yang, Qiang Guan +1Clinical Outcome Prediction Multimodal Agents
Cleveland Clinic, Cleveland, OH, USA
An interpretable and trustworthy AI framework for large-scale longitudinal structure-pain association studies using data from the Osteoarthritis Initiative (OAI)
Jun 3, 2026Jincheng Yu, Haoyang Li, Yiwen Liu +5Longitudinal Medical Image Analysis Clinical Outcome Prediction
Statistics & Data Science GIDP, University of Arizona · Department of Epidemiology and Biostatistics, University of Arizona · R. Kent Coit College of Pharmacy, University of Arizona +2
From Kellgren-Lawrence to Calcium Pyrophosphate Crystal Deposition: A Soft-Labelling Framework for Knee Osteoarthritis Assessmen
May 27, 2026Francisco Bérchez-Moreno, Riccardo Rosati, Maria Chiara Fiorentino +6Label Distribution Learning Medical Image Classification
Departamento de Ciencia de la Computación e Inteligencia Artificial, Universidad de Córdoba, Córdoba, España · Department of Political Science, Communication, and International Relations, University of Macerata, Macerata, Italy · Department of Innovative Technologies in Medicine & Dentistry, Università degli Studi "G. D’Annunzio" Chieti - Pescara, Chieti, Italy +5

Oct 8, 2026J/K move · Enter open · S save

OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression

Sixu Chen, Mingrui Yang, Qiang Guan, Xiaojuan Li

Cleveland Clinic, Cleveland, OH, USA

Clinical Outcome Prediction Multimodal Agents Multimodal Model Interpretability Uncertainty-Aware Multimodal Fusion

Open paper AskPDF arXiv

Knee osteoarthritis (KOA) progression prediction can support patient monitoring, requiring the integration of multimodal data and multidomain expertise. Moreover, isolated risk estimates provide limited insight underlying a prediction. To automate the progression assessment workflow and reduce manual effort while providing interpretable findings and supporting evidence, we present OA-MAP, an autonomous multi-agent framework for evidence-grounded assessment of structural and pain progression in KOA. The system incorporates modality-specific agents including MRI, X-ray, and clinical agents, together with a coordinator agent. This framework can autonomously recruit specialist agents, select tools for prediction and analysis, and retrieve literature as external evidence based on user request and available patient information. An uncertainty-informed human-in-the-loop mechanism enables clinicians to review and correct intermediate findings, triggering recomputation of affected results. We evaluate the prediction models using 600 participants from the FNIH Osteoarthritis Biomarkers Consortium cohort. On the test set of 100 participants, the fusion models achieve AUROCs of 0.80 for structural progression and 0.68 for pain progression. A case study illustrates how OA-MAP combines risk estimates with intermediate findings, cross-modal conflicts, literature support, and uncertainty indicators to support interactive review.

Abstract

Explore similar work

OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression

An interpretable and trustworthy AI framework for large-scale longitudinal structure-pain association studies using data from the Osteoarthritis Initiative (OAI)

From Kellgren-Lawrence to Calcium Pyrophosphate Crystal Deposition: A Soft-Labelling Framework for Knee Osteoarthritis Assessmen