Shared Canonical Component Removal Beyond Expert Views

Jose Roberto Tello Ayala; Akl C. Fahed; Roy Perlis; Finale Doshi-Velez

cs.LGOct 9, 2026

Shared Canonical Component Removal Beyond Expert Views

Authors: Jose Roberto Tello Ayala, Akl C. Fahed, Roy Perlis, Finale Doshi-Velez

Organizations: School of Engineering and Applied Sciences Harvard University · Heart and Vascular Institute Mass General Brigham · Center for Quantitative Health Mass General Brigham

Abstract

In clinical practice, experts often begin by converting raw inputs into a set of expert features before making a decision. While black-box models trained on raw input can outperform models trained on expert views, it remains unclear what predictive signal the raw model exploits beyond what the expert view captures. We introduce an audit framework to quantify this gap using Shared Canonical Component Removal (SCCR). Unlike standard residualization or concept erasure, SCCR selectively removes only those directions strongly shared between learned raw and expert embeddings. Evaluations on synthetic data, coronary artery disease (CAD) detection, antidepressant treatment dropout prediction, and skin lesion classification using SkinCon annotations show that SCCR identifies the ways in which expert views are insufficient and recovers residual predictive signal while minimizing redundancy with the expert view.

Figures & tables

Panel of Figure 1 — Figure 1: Dual-branch SCCR architecture. The raw input $x$ and expert view $\tilde{x}=g(x)$ are encoded into raw and expert embeddings. The expert branch serves as both a predictive input and a reference for constructing the residual raw branch. During training, the SCCR operator $T$ is estimated from minibatch statistics to compute $Z_{\perp}=Z_{r,c}T$ . After training, a calibration pass produces a frozen SCCR operator $T^{\star}$ , yielding deterministic residuals $z_{\perp}=(z_{r}-\mu_{r}^{\star})T^{\star}$ .

(a)	Raw	OLS	D-CCA	JIVE 1	JIVE 2	AJIVE 4,4	SCCR
$R_{A}^{2}\downarrow$	1	.510	.857	1	1	1	0
$R_{F_{1}}^{2}\uparrow$	1	.910	.977	1	0	1	1
$R_{F_{2}}^{2}\uparrow$	1	.910	.977	1	1	1	1

Table 1: Synthetic Linear Factor Recovery. (a) Analytic values. (b) Image experiments: mean

\pm

sample SD across five seeds.

Figure 2: Synthetic paired-view examples. The four raw patches encode

S,A,F_{1},F_{2}

, from upper left to lower right. The expert view contains

S

and noisy proxies of

A,F_{1},F_{2}

, with correlations

.95,.30,.30

, respectively, before independent pixel noise.

(a) Seed	Diagnosis gain $G$	Gain over OLS	Fixed residual $\Delta G_{C}$	Retraining $\Delta D_{C}$	External DDI gain $G$
909	$.073\,[.027,.114]$	$.105\,[.039,.174]$	$.020\,[.008,.034]$	$.016\,[.002,.029]$	$-.187\,[-.266,-.109]$
910	$.051\,[.003,.093]$	$.050\,[.005,.103]$	$.009\,[-.001,.020]$	$.045\,[.003,.088]$	$-.229\,[-.309,-.147]$

Table 2: Clinical prediction and residual audits. (a) skin-lesion classification, SCCR

\tau=.40

; (b) antidepressant treatment dropout, held-out-site AUROC mean

\pm

sample SD over five seeds; (c) coronary artery disease, one seed. Brackets: pointwise 95% paired-bootstrap intervals conditional on fitted models.

Panel of Figure 3 — Figure 3: Coronary angiogram examples. From left: raw image, vessel mask, and Grad $\times$ Input overlays for the raw and mask inputs, respectively.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Method / output	Transformation	Population output
Raw embedding	$Z_{r,c}$	$(\sqrt{2}S,A,.1F_{1},\sqrt{.02}F_{2})$
Feature-space projection	$Z_{r,c}(I-Z_{e,c}^{\dagger}Z_{e,c})$	$(0,0,0,0)$
OLS residual	$Z_{r,c}-Z_{e,c}B_{\mathrm{OLS}}$	$\begin{aligned} (0,\,&A-\rho Q,\\ &.1(F_{1}-\gamma P_{1}),\\ &\sqrt{.02}(F_{2}-\gamma P_{2}))\end{aligned}$
Exact LEACE	$Z_{r,c}T_{\mathrm{LEACE}}$	$(0,0,0,0)$
D-CCA distinctive	$Z_{r,c}-C_{r}$	$(0,D_{A},.1D_{1},\sqrt{.02}D_{2})$
JIVE, $q=1$ , individual	$\mathcal{T}_{3}((I-H_{J})Z_{r,c})$	$(0,A,.1F_{1},\sqrt{.02}F_{2})$

Appendix

Table 3: Residualization transformations and population outputs. JIVE uses global optima with joint rank

q

and individual ranks

(4-q,4-q)

. AJIVE uses initial ranks

(4,4)

and the zero-perturbation boundary convention in Appendix A.8 . SCCR uses zero ridge and

\gamma<\tau<\rho

S_{rr}\leftarrow\tfrac{1}{B}Z_{r,c}^{\top}Z_{r,c},\quad S_{ee}\leftarrow\tfrac{1}{B}Z_{e,c}^{\top}Z_{e,c},\quad S_{re}\leftarrow\tfrac{1}{B}Z_{r,c}^{\top}Z_{e,c}

Appendix

Algorithm 1 Shared Canonical Component Removal with OAS-Selected Branchwise Ridge

Model or diagnostic	907	908	909	910	911
Dual baseline	86	99	98	91	100
Raw only	83	83	82	75	83
SCCR	100	100	100	99	100
OLS	100	84	91	90	90
Regularized LEACE-style	76	94	78	42	84
VCCA-private	100	100	100	100	100

Appendix

Table 4: Selected checkpoint epochs after completing 100 epochs in every run. Epochs are numbered from 1. The final row gives the number of directions removed by each end-to-end SCCR model’s calibrated operator at its selected checkpoint; these differ from the ranks fitted to common embeddings.

Method	Code	Probe	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$
Dual baseline	Raw	L	$.053\!\pm\!.023$	$.826\!\pm\!.021$	$.693\!\pm\!.028$	$.634\!\pm\!.032$	$.726\!\pm\!.022$
Dual baseline	Own expert	L	$.166\!\pm\!.059$	$.798\!\pm\!.025$	$-.001\!\pm\!.001$	$-.000\!\pm\!.002$	$-.001\!\pm\!.001$
Raw only	Raw	L	$.049\!\pm\!.023$	$.802\!\pm\!.012$	$.717\!\pm\!.020$	$.638\!\pm\!.017$	$.721\!\pm\!.015$
SCCR	Raw	L	$.122\!\pm\!.041$	$.890\!\pm\!.023$	$.710\!\pm\!.043$	$.647\!\pm\!.084$	$.702\!\pm\!.031$
SCCR	Own expert	L	$.399\!\pm\!.051$	$.809\!\pm\!.128$	$-.000\!\pm\!.002$	$.002\!\pm\!.012$	$-.000\!\pm\!.000$
SCCR	Residual	L	$.115\!\pm\!.044$	$.051\!\pm\!.092$	$.708\!\pm\!.044$	$.644\!\pm\!.082$	$.699\!\pm\!.033$

Appendix

Table 5: Factor recovery from separately trained representations. Test-set

R^{2}

is mean

\pm

sample SD over seeds 907–911; each code and target has its own development-selected readout. L/Q denote linear/quadratic probes. Full/shared/private are raw-view codes; own expert refers to the same supervised arm.

		Linear probes					Quadratic probes
Method/seed	Code	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$
Dual 907	Raw	$.0528$	$.8247$	$.7020$	$.6302$	$.7442$	$.0810$	$.9108$	$.8635$	$.7901$	$.8996$
Dual 907	Own expert	$.2349$	$.8014$	$.0013$	$-.0014$	$-.0002$	$.2767$	$.8195$	$.0006$	$-.0014$	$-.0007$
Dual 908	Raw	$.0244$	$.8471$	$.6451$	$.6573$	$.7385$	$.0495$	$.9291$	$.8850$	$.8528$	$.9096$
Dual 908	Own expert	$.1100$	$.7642$	$-.0026$	$.0028$	$-.0032$	$.1186$	$.7885$	$-.0006$	$-.0001$	$-.0021$
Dual 909	Raw	$.0568$	$.8387$	$.7168$	$.6581$	$.7266$	$.0952$	$.9271$	$.9178$	$.8559$	$.9074$

Appendix

Table 6: Per-seed factor recovery: supervised dual and raw-only models. Entries are test-set

R^{2}

; linear and quadratic readouts for each target were selected separately. Raw is before removal; own expert is learned by the same arm. Full concatenates raw-view shared and private codes. Duplicate raw/residual controls are shown once; shared/private own-expert codes were not exported. Dual, Raw, VCCA, and DSN denote Dual baseline, Raw only, VCCA-private, and DSN-MMD.

		Linear probes					Quadratic probes
Method/seed	Code	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$
R-LEACE 907	Raw	$.4642$	$.0792$	$.1990$	$.0546$	$-.0002$	$.5119$	$.1248$	$.1876$	$.0368$	$-.0003$
R-LEACE 907	Own expert	$.1234$	$.8686$	$-.0004$	$-.0020$	$-.0003$	$.2229$	$.8816$	$-.0006$	$-.0014$	$-.0003$
R-LEACE 907	Residual	$-.0013$	$-.0002$	$-.0006$	$-.0000$	$-.0003$	$-.0013$	$-.0002$	$-.0006$	$-.0000$	$-.0003$
R-LEACE 908	Raw	$.5817$	$.2656$	$.3546$	$.0887$	$-.0002$	$.6860$	$.2609$	$.5373$	$.0596$	$-.0003$
R-LEACE 908	Own expert	$.1935$	$.8716$	$-.0006$	$-.0026$	$-.0069$	$.3155$	$.8827$	$-.0006$	$-.0017$	$-.0003$

Appendix

Table 7: Per-seed factor recovery: regularized erasure and shared/private models. Entries are test-set

R^{2}

Method	Metric	907	908	909	910	911	Mean $\pm$ SD
Dual baseline	Accuracy	$.9839$	$.9822$	$.9839$	$.9846$	$.9841$	$.984\!\pm\!.001$
	AUROC	$.9992$	$.9992$	$.9991$	$.9992$	$.9992$	$.9992\!\pm\!.0000$
	Log loss	$.0365$	$.0380$	$.0391$	$.0361$	$.0375$	$.037\!\pm\!.001$
Raw only	Accuracy	$.9873$	$.9854$	$.9846$	$.9832$	$.9841$	$.985\!\pm\!.002$
	AUROC	$.9994$	$.9994$	$.9993$	$.9989$	$.9993$	$.9993\!\pm\!.0002$
	Log loss	$.0325$	$.0327$	$.0335$	$.0424$	$.0357$	$.035\!\pm\!.004$

Appendix

Table 8: Native task-head prediction for each seed and its five-seed mean

\pm

sample SD. Accuracy uses probability threshold

.5

; AUROC uses continuous scores; log loss is cross-entropy. All scores use the same fixed assessment cohort. These are the trained heads, not post-hoc readouts.

		Linear readouts			Quadratic readouts
Method	Code	Accuracy	AUROC	Log loss	Accuracy	AUROC	Log loss
Dual baseline	Raw	$.984\!\pm\!.001$	$.9991\!\pm\!.0002$	$.041\!\pm\!.003$	$.983\!\pm\!.002$	$.9991\!\pm\!.0001$	$.040\!\pm\!.003$
Dual baseline	$E$	$.747\!\pm\!.013$	$.8382\!\pm\!.0087$	$.494\!\pm\!.010$	$.744\!\pm\!.008$	$.8366\!\pm\!.0084$	$.499\!\pm\!.012$
Dual baseline	$(E,R)=(E_{0},R)$	$.984\!\pm\!.001$	$.9991\!\pm\!.0001$	$.040\!\pm\!.002$	$.981\!\pm\!.001$	$.9985\!\pm\!.0002$	$.051\!\pm\!.001$
Raw only	Raw	$.986\!\pm\!.001$	$.9994\!\pm\!.0001$	$.032\!\pm\!.003$	$.986\!\pm\!.001$	$.9994\!\pm\!.0001$	$.033\!\pm\!.002$
Raw only	$(E_{0},R)$	$.985\!\pm\!.003$	$.9993\!\pm\!.0002$	$.036\!\pm\!.005$	$.983\!\pm\!.002$	$.9989\!\pm\!.0002$	$.043\!\pm\!.004$

Appendix

Table 9: Post-hoc task readouts on learned codes: mean

\pm

sample SD over seeds 907–911.

E

is the own expert and

E_{0}

the same-seed dual baseline expert, so

(E,R)

and

(E_{0},R)

remain distinct. Accuracy uses threshold

.5

. Each representation and readout family is selected separately on development data; these are not native heads.

Linear probes
Output	$R_{S}^{2}$	$R_{A}^{2}$	$R_{F_{1}}^{2}$	$R_{F_{2}}^{2}$	$R_{F_{1}F_{2}}^{2}$
Raw	$.053\!\pm\!.023$	$.826\!\pm\!.021$	$.693\!\pm\!.028$	$.634\!\pm\!.032$	$.726\!\pm\!.022$
Zero	$-.001\!\pm\!.000$	$-.000\!\pm\!.000$	$-.001\!\pm\!.000$	$-.000\!\pm\!.000$	$-.000\!\pm\!.000$
SCCR	$.055\!\pm\!.020$	$.010\!\pm\!.004$	$.691\!\pm\!.027$	$.633\!\pm\!.033$	$.724\!\pm\!.022$
Exact LEACE	$-.001\!\pm\!.000$	$-.000\!\pm\!.000$	$.001\!\pm\!.003$	$.000\!\pm\!.000$	$.001\!\pm\!.002$
D-CCA	$.043\!\pm\!.019$	$.451\!\pm\!.039$	$.690\!\pm\!.029$	$.632\!\pm\!.030$	$.723\!\pm\!.023$

Appendix

Table 10: Factor recovery on common dual-baseline embeddings: mean

\pm

sample SD over seeds 907–911. AJIVE uses paired individual outputs; Gap is the primary singular-value-ratio initial-rank rule. JIVE

q=0

tests no joint removal. D-CCA is the empirical covariance plug-in without S-POET.

Output	Linear Res	Linear Comb	Quadratic Res	Quadratic Comb
Raw	$.9991\!\pm\!.0002$	$.9991\!\pm\!.0001$	$.9991\!\pm\!.0001$	$.9985\!\pm\!.0002$
Zero	$.5000\!\pm\!.0000$	$.8382\!\pm\!.0087$	$.5000\!\pm\!.0000$	$.8366\!\pm\!.0084$
SCCR	$.9132\!\pm\!.0142$	$.9916\!\pm\!.0018$	$.9814\!\pm\!.0049$	$.9952\!\pm\!.0011$
Exact LEACE	$.5000\!\pm\!.0000$	$.8382\!\pm\!.0087$	$.5000\!\pm\!.0000$	$.8366\!\pm\!.0084$
D-CCA	$.9948\!\pm\!.0012$	$.9988\!\pm\!.0002$	$.9979\!\pm\!.0004$	$.9981\!\pm\!.0002$
AJIVE Gap individual	$.9991\!\pm\!.0001$	$.9990\!\pm\!.0001$	$.9992\!\pm\!.0001$	$.9986\!\pm\!.0002$

Appendix

Table 11: Task AUROC on common embeddings: mean

\pm

sample SD over seeds 907–911. Res uses the listed output alone; Comb adds the common expert code. Paired AJIVE outputs and D-CCA already use both views before readout. These are frozen readouts, not native heads.

Linear readouts
Output	Accuracy Res	Accuracy Comb	Log loss Res	Log loss Comb
Raw	$.984\!\pm\!.001$	$.984\!\pm\!.001$	$.041\!\pm\!.003$	$.040\!\pm\!.002$
Zero	$.513\!\pm\!.000$	$.747\!\pm\!.013$	$.693\!\pm\!.000$	$.494\!\pm\!.010$
SCCR	$.824\!\pm\!.023$	$.968\!\pm\!.004$	$.380\!\pm\!.022$	$.105\!\pm\!.012$
Exact LEACE	$.513\!\pm\!.000$	$.747\!\pm\!.013$	$.693\!\pm\!.000$	$.494\!\pm\!.010$
D-CCA	$.982\!\pm\!.003$	$.983\!\pm\!.001$	$.072\!\pm\!.009$	$.045\!\pm\!.002$

Appendix

Table 12: Accuracy and log loss on common embeddings: mean

\pm

sample SD over seeds 907–911. Res/Comb denote output-only/expert-plus-output. Accuracy uses threshold

.5

; validation log loss selects each frozen readout.

Rank setting	907	908	909	910	911
$k=4,q=1$	$1.999\times 10^{-6}$	$2.366\times 10^{-6}$	$1.325\times 10^{-6}$	$1.932\times 10^{-6}$	$1.071\times 10^{-6}$
$k=4,q=2$	$4.410\times 10^{-5}$	$1.485\times 10^{-6}$	$5.008\times 10^{-8}$	$1.788\times 10^{-5}$	$4.893\times 10^{-8}$
$k=8,q=1$	$3.054\times 10^{-7}$	$3.164\times 10^{-7}$	$1.142\times 10^{-6}$	$7.696\times 10^{-7}$	$3.311\times 10^{-7}$
$k=8,q=2$	$9.055\times 10^{-6}$	$3.289\times 10^{-5}$	$4.887\times 10^{-7}$	$1.032\times 10^{-6}$	$8.283\times 10^{-7}$
$k=16,q=1$	$4.030\times 10^{-7}$	$1.725\times 10^{-6}$	$6.336\times 10^{-7}$	$9.460\times 10^{-7}$	$2.266\times 10^{-6}$
$k=16,q=2$	$1.584\times 10^{-6}$	$6.083\times 10^{-6}$	$9.851\times 10^{-8}$	$2.062\times 10^{-7}$	$1.007\times 10^{-7}$

Appendix

Table 14: Original positive-joint-rank JIVE outcomes: all entries are NC and unscored. Numbers give final relative fitted-matrix change at the 5,000-iteration cap; eligibility requires a value below

10^{-9}

. Each

(k,q)

has four variants (P/I, P/Rem, R/I, R/Rem): 24 ineligible variants per seed, 120 across five seeds, arising from 30 distinct rank fits. NC is never assigned chance or zero recovery. Individual rank budgets are

(k-q,k-q)

, with block Frobenius scaling and no rank expansion.

Cohort / partition	Images	Diagnosis $+$	Diagnosis $-$	Crust	Erosion	Scale
Fitzpatrick17k training	1,935	286	1,649	294	116	393
Operator calibration	965	129	836	128	67	199
Audit-readout fitting	970	157	813	166	49	194
Fitzpatrick17k validation	645	96	549	101	43	143
Fitzpatrick17k test	645	95	550	102	40	148
DDI external evaluation	636	171	465	53	14	103

Appendix

Table 16: SkinCon cohort counts. Positive denotes the study’s binary diagnosis label; concept columns count positive annotations. Calibration and readout rows partition the training row, rather than adding further images.

Model	Expert condition	Seed 909	Seed 910	$q_{909}$	$q_{910}$
Dual baseline	Omitted	23 / 18	19 / 14	—	—
SCCR	Omitted	29 / 24	21 / 16	—	—
OLS	Omitted	22 / 17	23 / 18	—	—
Regularized LEACE-style	Omitted	65 / 60	54 / 49	—	—
VCCA-private	Omitted	70 / 65	44 / 39	—	—
DSN-MMD	Omitted	16 / 11	21 / 16	—	—

Appendix

Table 17: Completed SkinCon training. Epoch cells give completed / selected checkpoint, both one-based; all runs reached patience-five early stopping. True and shuffled conditions restore Crust only.

Model	Seed	RGB MSE	Centered image error	Annotation BCE
VCCA-private	909	.025619	.024970	.149454
VCCA-private	910	.026722	.025869	.139893
DSN-MMD	909	.235905	.026299	.616325
DSN-MMD	910	.274671	.026180	.507646
Training-mean reference	—	.032361	.025816	.132480

Appendix

Table 18: SkinCon reconstruction controls on the 645 validation images. Lower is better. The reference uses the training mean RGB image and per-concept training prevalence. VCCA uses posterior means. Centered image error is the DSN reconstruction criterion; ordinary RGB MSE is also shown.

Model	Seed	F17k AUROC	F17k log loss	DDI AUROC	DDI log loss
Dual baseline	909	.873 [.824, .915]	.263 [.217, .316]	.559 [.510, .607]	.900 [.821, .975]
Dual baseline	910	.875 [.823, .918]	.265 [.214, .325]	.592 [.541, .641]	.791 [.720, .863]
SCCR ( $\tau=.40$ )	909	.845 [.796, .892]	.318 [.281, .360]	.541 [.489, .588]	.648 [.592, .707]
SCCR ( $\tau=.40$ )	910	.805 [.756, .853]	.349 [.315, .387]	.533 [.481, .581]	.646 [.596, .703]
OLS	909	.858 [.811, .902]	.314 [.282, .350]	.547 [.500, .594]	.689 [.646, .733]
OLS	910	.823 [.768, .869]	.321 [.275, .379]	.562 [.508, .611]	.714 [.645, .792]

Appendix

Table 19: Native diagnosis prediction from separately trained models on Fitzpatrick17k test (F17k) and DDI.

Fitzpatrick17k test
Method	Seed	Crust AUROC $(R)$ linear	Crust AUROC $(R)$ nonlinear	$G$ linear	$G$ nonlinear
Dual baseline	909	.588 [.529, .650]	.615 [.553, .677]	.031 [-.029, .088]	-.014 [-.098, .061]
Dual baseline	910	.628 [.569, .689]	.606 [.545, .665]	.040 [-.030, .102]	.001 [-.082, .068]
SCCR ( $\tau=.40$ )	909	.635 [.578, .692]	.626 [.571, .681]	.074 [.027, .121]	.073 [.027, .114]
SCCR ( $\tau=.40$ )	910	.568 [.505, .631]	.579 [.522, .641]	.042 [-.004, .087]	.051 [.003, .093]
OLS	909	.604 [.538, .670]	.639 [.579, .703]	-.121 [-.239, -.020]	-.032 [-.112, .041]

Appendix

Table 20: Residual audits of separately trained models.

Fitzpatrick17k test
Method	Seed	Crust AUROC raw/full	Diagnosis AUROC raw/full	Diagnosis AUROC residual/private	$G$ without fallback
Dual baseline	909	.615 [.553, .677]	.865 [.815, .907]	.865 [.815, .907]	-.014 [-.098, .061]
Dual baseline	910	.606 [.545, .665]	.856 [.806, .903]	.856 [.806, .903]	.001 [-.082, .068]
SCCR ( $\tau=.40$ )	909	.620 [.565, .672]	.862 [.813, .902]	.868 [.824, .905]	.073 [.027, .114]
SCCR ( $\tau=.40$ )	910	.664 [.603, .722]	.853 [.803, .896]	.831 [.782, .876]	.051 [.003, .093]
OLS	909	.586 [.524, .643]	.822 [.766, .875]	.847 [.797, .894]	-.032 [-.112, .041]

Appendix

Table 21: Raw-code learning controls and nonlinear audit prediction without expert-only fallback.

Method	Seed	Fitzpatrick17k test $G$	DDI $G$
Dual baseline	909	.031 [-.029, .088]	-.501 [-.623, -.387]
Dual baseline	910	.040 [-.030, .102]	-.465 [-.576, -.353]
SCCR ( $\tau=.40$ )	909	.074 [.027, .121]	-.182 [-.260, -.103]
SCCR ( $\tau=.40$ )	910	.042 [-.004, .087]	-.205 [-.283, -.127]
OLS	909	-.121 [-.239, -.020]	-.728 [-.874, -.592]
OLS	910	.014 [-.051, .073]	-.281 [-.370, -.191]

Appendix

Table 22: Linear diagnosis gains with the combined predictor forced at assessment.

Fitzpatrick17k test
Method	Seed	Crust AUROC $(R)$ linear	Crust AUROC $(R)$ nonlinear	$G$ linear	$G$ nonlinear
Dual baseline	909	.588 [.529, .650]	.615 [.553, .677]	.031 [-.029, .088]	-.014 [-.098, .061]
Dual baseline	910	.628 [.569, .689]	.606 [.545, .665]	.040 [-.030, .102]	.001 [-.082, .068]
SCCR ( $\tau=.40$ )	909	.554 [.493, .614]	.642 [.584, .698]	-.090 [-.185, -.003]	.000 [-.075, .068]
SCCR ( $\tau=.40$ )	910	.624 [.565, .684]	.637 [.574, .691]	.003 [-.034, .038]	.018 [-.040, .068]
Empirical LEACE	909	.500 P [.500, .500]	.500 P [.500, .500]	.000 F [.000, .000]	.000 F [.000, .000]

Appendix

Table 23: Residual audits on common frozen dual-baseline embeddings.

Method	Seed	Initial / total ranks	Joint rank	Individual ranks
JIVE	909	(1, 1)	1	(0, 0)
AJIVE	909	(1, 1)	0	(1, 1)
JIVE	910	(1, 1)	1	(0, 0)
AJIVE	910	(1, 1)	0	(1, 1)

Appendix

Table 24: Primary common-embedding JIVE and AJIVE rank settings.

Baseline	Seed	F17k linear	F17k nonlinear	DDI linear	DDI nonlinear
Dual baseline	909	-.121 [-.190, -.052]	.014 [-.028, .058]	-.269 [-.420, -.129]	.389 [.291, .483]
Dual baseline	910	-.037 [-.092, .021]	.018 [-.031, .071]	.398 [.297, .503]	.499 [.396, .596]
Empirical LEACE	909	-.090 [-.185, -.003]	.000 [-.075, .068]	-.770 [-.937, -.609]	-.548 [-.664, -.438]
Empirical LEACE	910	.003 [-.034, .038]	.018 [-.040, .068]	-.067 [-.125, -.006]	-.240 [-.322, -.163]
JIVE	909	-.090 [-.185, -.003]	.000 [-.075, .068]	-.770 [-.937, -.609]	-.548 [-.664, -.438]
JIVE	910	.003 [-.034, .038]	.018 [-.040, .068]	-.067 [-.125, -.006]	-.240 [-.322, -.163]

Appendix

Table 25: Paired SCCR-minus-baseline diagnosis gain differences on common embeddings.

Baseline	Seed	F17k linear	F17k nonlinear	DDI linear	DDI nonlinear
Dual baseline	909	.043 [-.009, .098]	.087 [.022, .157]	.319 [.225, .422]	.751 [.627, .872]
Dual baseline	910	.002 [-.052, .062]	.050 [-.010, .122]	.260 [.169, .350]	.509 [.407, .622]
OLS	909	.194 [.106, .291]	.105 [.039, .174]	.547 [.426, .673]	.456 [.355, .572]
OLS	910	.028 [-.015, .080]	.050 [.005, .103]	.076 [.003, .153]	.050 [-.017, .127]
Reg. LEACE-style	909	.074 [.027, .121]	.073 [.027, .114]	-.182 [-.260, -.103]	-.187 [-.266, -.109]
Reg. LEACE-style	910	.042 [-.004, .087]	.051 [.003, .093]	-.205 [-.283, -.127]	-.229 [-.309, -.147]

Appendix

Table 26: Paired SCCR-minus-baseline diagnosis gain differences for separately trained models.

Predictive overlap: initial gain and attenuation
Cohort	Seed / probe	$G(\emptyset)$	$A_{C}$	$A_{C}-A_{\widetilde{C}}$
F17k	909 / linear	.074 [.027, .121]	.003 [-.006, .012]	.005 [-.004, .014]
F17k	909 / nonlinear	.073 [.027, .114]	.015 [.003, .027]	.020 [.008, .034]
F17k	910 / linear	.042 [-.004, .087]	.000 [-.006, .008]	.001 [-.006, .008]
F17k	910 / nonlinear	.051 [.003, .093]	.008 [-.002, .018]	.009 [-.001, .020]
DDI	909 / linear	-.182 [-.260, -.103]	-.014 [-.023, -.005]	-.015 [-.024, -.006]

Appendix

Table 27: Fixed SCCR omission residual: predictive overlap with Crust and the two contributions to its attenuation.

$\tau$	Seed / probe	F17k $D_{T}-D_{S}$	F17k $D_{T}-D_{O}$	DDI $D_{T}-D_{S}$	DDI $D_{T}-D_{O}$
.20	909 / linear	.064 [.002, .133]	.055 [-.032, .138]	.070 [-.039, .170]	.028 [-.088, .144]
.20	909 / nonlinear	-.012 [-.109, .093]	-.074 [-.169, .026]	-.025 [-.157, .108]	.044 [-.098, .194]
.20	910 / linear	.085 [.012, .161]	.089 [.013, .164]	.018 [-.069, .109]	-.024 [-.108, .075]
.20	910 / nonlinear	.001 [-.091, .093]	.018 [-.077, .114]	-.077 [-.188, .048]	.000 [-.100, .112]
.40	909 / linear	.016 [.002, .029]	.008 [-.023, .039]	.003 [-.013, .019]	.025 [-.005, .053]
.40	909 / nonlinear	.045 [-.016, .111]	.009 [-.062, .083]	-.013 [-.101, .087]	.017 [-.070, .111]

Appendix

Table 28: Crust-only retraining: restoration contrasts at both training thresholds.

Fitzpatrick17k test
Seed / condition	Probe	Raw AUROC	Residual AUROC	Expert-code AUROC
909 / Omitted	linear	.640 [.583, .695]	.635 [.578, .692]	.662 [.597, .725]
909 / Omitted	nonlinear	.620 [.565, .672]	.626 [.571, .681]	.674 [.610, .736]
909 / True Crust	linear	.605 [.546, .668]	.592 [.530, .657]	1.000 [1.000, 1.000]
909 / True Crust	nonlinear	.583 [.524, .643]	.581 [.520, .646]	.999 [.999, 1.000]
909 / Shuffled Crust	linear	.654 [.593, .712]	.658 [.597, .717]	.671 [.606, .732]

Appendix

Table 29: SCCR (

\tau=.40

): Crust accessibility after retraining.

Seed / condition	Probe	Raw / residual AUROC	Expert-code AUROC
Fitzpatrick17k test
909 / Omitted	linear	.588 [.529, .650]	.674 [.606, .736]
909 / Omitted	nonlinear	.615 [.553, .677]	.685 [.624, .743]
909 / True Crust	linear	.570 [.503, .634]	1.000 [.999, 1.000]
909 / True Crust	nonlinear	.591 [.534, .651]	.999 [.997, 1.000]
909 / Shuffled Crust	linear	.644 [.587, .710]	.676 [.611, .737]

Appendix

Table 30: Dual baseline: Crust accessibility after retraining.

Seed / condition	Probe	Native AUROC	$G$ : common $\tilde{x}_{45}$	$G$ : matched expert
Fitzpatrick17k test
909 / Omitted	linear	.845 [.796, .892]	.074 [.027, .121]	.074 [.027, .121]
909 / Omitted	nonlinear		.073 [.027, .114]	.073 [.027, .114]
909 / True Crust	linear	.842 [.795, .884]	.044 [-.006, .094]	.041 [-.009, .092]
909 / True Crust	nonlinear		.029 [-.023, .078]	.021 [-.028, .068]
909 / Shuffled Crust	linear	.872 [.825, .911]	.010 [-.058, .073]	.002 [-.065, .065]

Appendix

Table 31: Diagnosis after SCCR Crust-only retraining (

\tau=.40

Method / seed	Code	Crust AUROC linear	Diagnosis AUROC linear	Crust AUROC nonlinear	Diagnosis AUROC nonlinear
VCCA-private / 909	Full	.506 [.446, .563]	.590 [.521, .653]	.458 [.396, .516]	.595 [.526, .656]
VCCA-private / 909	Shared	.564 [.505, .631]	.500 P [.500, .500]	.434 [.382, .496]	.495 [.432, .556]
VCCA-private / 909	Private	.483 [.422, .536]	.581 [.513, .645]	.467 [.407, .526]	.579 [.512, .638]
VCCA-private / 910	Full	.526 [.464, .590]	.649 [.587, .714]	.538 [.477, .603]	.548 [.488, .606]
VCCA-private / 910	Shared	.522 [.462, .584]	.648 [.588, .714]	.538 [.473, .600]	.558 [.496, .618]
VCCA-private / 910	Private	.571 [.509, .640]	.500 P [.500, .500]	.479 [.418, .540]	.500 P [.500, .500]

Appendix

Table 32: Fitzpatrick17k test: recovery from full, shared, and private codes.

Method / seed	Code	Crust AUROC linear	Diagnosis AUROC linear	Crust AUROC nonlinear	Diagnosis AUROC nonlinear
VCCA-private / 909	Full	.509 [.422, .594]	.560 [.510, .610]	.516 [.434, .604]	.511 [.457, .566]
VCCA-private / 909	Shared	.540 [.458, .615]	.500 P [.500, .500]	.456 [.370, .539]	.509 [.457, .561]
VCCA-private / 909	Private	.482 [.405, .557]	.557 [.506, .606]	.525 [.454, .609]	.513 [.463, .566]
VCCA-private / 910	Full	.591 [.508, .671]	.555 [.504, .605]	.480 [.401, .554]	.510 [.460, .558]
VCCA-private / 910	Shared	.568 [.492, .647]	.559 [.508, .612]	.456 [.371, .536]	.515 [.467, .564]
VCCA-private / 910	Private	.550 [.467, .623]	.500 P [.500, .500]	.485 [.402, .562]	.500 P [.500, .500]

Appendix

Table 33: DDI: recovery from full, shared, and private codes.

Encoder	Seed / $\tau$ / $q$	Crust AUROC linear	Crust AUROC nonlinear	$G$ linear	$G$ nonlinear
Fitzpatrick17k test
Dual baseline	909 / .20 / 5	.541 [.483, .603]	.566 [.510, .624]	-.050 [-.111, .006]	.020 [-.034, .075]
Dual baseline	910 / .20 / 8	.539 [.478, .596]	.548 [.488, .611]	.002 [-.022, .024]	.010 [-.030, .051]
Dual baseline	910 / .50 / 0	.628 [.569, .689]	.606 [.545, .665]	.040 [-.030, .102]	.001 [-.082, .068]
SCCR ( $.40$ )	909 / .20 / 10	.512 [.457, .569]	.579 [.516, .637]	-.028 [-.071, .017]	-.016 [-.057, .025]
SCCR ( $.40$ )	909 / .50 / 1	.651 [.596, .706]	.649 [.593, .701]	.074 [.028, .122]	.075 [.030, .117]

Appendix

Table 34: Threshold sensitivity on frozen omission encoders: additional settings around the primary

.40

fit.

Cohort	Seed	$\tau$	Native AUROC	$G$ linear	$G$ nonlinear
Fitzpatrick17k	909	.20	.708 [.649, .767]	-.012 [-.050, .021]	-.002 [-.040, .031]
Fitzpatrick17k	910	.20	.785 [.729, .836]	.016 [-.022, .053]	.033 [.003, .064]
DDI	909	.20	.509 [.457, .560]	-.128 [-.181, -.073]	-.103 [-.152, -.051]
DDI	910	.20	.553 [.501, .598]	-.032 [-.080, .013]	-.051 [-.100, -.004]

Appendix

Table 35: Additional omission models trained end to end at

\tau=.20

		Site B		Site C		Site D
Method	$n$	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC
Expert-only	5	$.565\pm.001$	$.415\pm.001$	$.574\pm.002$	$.476\pm.001$	$.555\pm.002$	$.359\pm.002$
Raw-only	5	$.619\pm.001$	$.475\pm.001$	$.611\pm.003$	$.509\pm.002$	$.585\pm.006$	$.389\pm.002$
Identity	5	$.615\pm.003$	$.471\pm.004$	$.608\pm.005$	$.508\pm.004$	$.583\pm.008$	$.386\pm.008$
OLS	5	$.613\pm.004$	$.470\pm.004$	$.605\pm.006$	$.505\pm.004$	$.573\pm.012$	$.380\pm.007$
LEACE	4	$.564\pm.001$	$.414\pm.001$	$.574\pm.002$	$.468\pm.001$	$.554\pm.004$	$.358\pm.006$

Appendix

Table 36: Dropout prediction on held-out sites. Values are mean

\pm

SD across

n

completed seeds; AUPRC is computed as average precision. Identity uses the dual architecture without removal.

Rank	Code	Rank	Code	Rank	Code	Rank	Code
1	cpt4:99213	65	icd10:f33.9	129	cpt4:76856	193	cpt4:82247
2	cpt4:99214	66	cpt4:99284	130	cpt4:88142	194	rxnorm:36567
3	icd10:i10	67	rxnorm:1191	131	icd10:r06.02	195	rxnorm:4603
4	cpt4:99211	68	cpt4:97110	132	rxnorm:8640	196	rxnorm:214182
5	cpt4:85027	69	cpt4:71010	133	cpt4:85007	197	icd10:z04
6	cpt4:36415	70	cpt4:82310	134	cpt4:83550	198	icd10:n95.1

Appendix

Table 37: Fixed code panel for treatment-dropout prediction.

$\boldsymbol{\tau}$	AUROC [95% CI]	AUPRC [95% CI]	Mask drop	Raw drop	Mask/raw attr.	Evidence
			zero / perm.	zero / perm.
0.1	0.792 [0.772, 0.809]	0.766 [0.738, 0.792]	0.283 / 0.303	0.000 / 0.003	31.25	Mask
0.5	0.790 [0.771, 0.807]	0.770 [0.745, 0.792]	0.261 / 0.278	0.000 / 0.000	16.17	Mask
0.7	0.835 [0.819, 0.851]	0.820 [0.795, 0.842]	0.000 / 0.000	0.387 / 0.348	0.029	Raw
0.9	0.831 [0.816, 0.846]	0.820 [0.797, 0.842]	-0.001 / 0.000	0.342 / 0.318	0.015	Raw

Appendix

Table 38: Historical CAD angiogram evidence-source audit. AUROC and AUPRC include bootstrap 95% confidence intervals. Drops are point-estimate decreases in test AUROC after zeroing or permuting the corresponding input branch. Attribution is the mask/raw Grad

\times

Input mass ratio for the full logit.

(a) Separately trained native heads
Model	AUROC	AUPRC	Log loss
Dual baseline	.822 [.806, .839]	.815 [.794, .837]	.526 [.499, .550]
Epoch OLS	.774 [.754, .793]	.722 [.691, .752]	.594 [.570, .621]
Epoch LEACE	.793 [.775, .810]	.773 [.748, .801]	.560 [.537, .585]
SCCR $\tau=.10$	.792 [.775, .811]	.766 [.739, .793]	.552 [.535, .569]
SCCR $\tau=.30$	.788 [.770, .807]	.767 [.742, .792]	.553 [.535, .569]

Appendix

Table 39: Follow-up CAD prediction, with 95% bootstrap intervals. Native heads use their own expert and retained embeddings; common-embedding readouts use

E_{0}

. SCCR

.90

removes no common directions and agrees with the dual baseline. Common SCCR

.10

removes three directions and equals matched top-3 CCA; the identical control is not repeated. Larger

G_{0}

means lower diagnosis log loss after adding the residual. These rows do not establish a consistent SCCR advantage.

(a) Diagnosis gain beyond each SCCR model’s own expert
$\tau$	Probe	$G_{X}$	$G_{R}$	$G_{X}-G_{R}$
.10	Linear	-.0022 [-.0044, .0001]	-.0002 [-.0022, .0017]	-.0020 [-.0040, .0001]
.10	RBF	-.0019 [-.0060, .0025]	-.0009 [-.0055, .0040]	-.0010 [-.0042, .0025]
.30	Linear	.0009 [-.0006, .0024]	.0009 [-.0006, .0024]	.0000 [.0000, .0000]
.30	RBF	-.0005 [-.0057, .0049]	-.0005 [-.0057, .0049]	.0000 [.0000, .0000]
(b) Descriptor recovery in the saved SCCR $\tau=.10$ model

Appendix

Table 40: Follow-up CAD audit with 95% bootstrap intervals. (a) Positive

G_{X}

G_{R}

indicates improvement beyond the same model’s own expert; positive

G_{X}-G_{R}

indicates loss of gain after removal. Near-zero differences at

.30

are rounded, and its frozen map is the identity. (b) Values are test

R^{2}

before and after removal. RBF denotes the bounded nonlinear probe, not a quadratic probe. The geometric and appearance targets are computed from the masks and images.

Explore similar work

MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
May 15, 2026Seungik Cho, Anqi Li, Wei QiuMissing-Modality Learning Multimodal Robustness
Department of Electrical and Computer Engineering Rice University Houston, TX 77005
Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles
Aug 16, 2026Srikumar KrishnamoorthyInterpretable ML Ensemble Learning
Information Systems Area, Indian Institute of Management Ahmedabad, India
ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs
Jul 30, 2026Ruman Wang, Hangting YeLLM-Based Program Synthesis Medical Image Classification
Liaoning University of Traditional Chinese Medicine · School of Artificial Intelligence, Jilin University

May 15, 2026, cs.LGJ/K move · Enter open · S save

MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis

Seungik Cho, Anqi Li, Wei Qiu

Department of Electrical and Computer Engineering Rice University Houston, TX 77005

Missing-Modality Learning Multimodal Robustness Healthcare Medical Diagnosis Multimodal Fusion

Open paper AskPDF arXiv

Multimodal clinical prediction faces three challenges: multiple foundation models (FMs) with complementary strengths per modality, pervasive missing modalities at training and test time, and sample-specific variation in modality contributions. We introduce MedMIX, a multimodal framework that combines intra-modality expert fusion, learned inter-modality fusion, and training-only large--small model collaboration for robust medical prediction under incomplete modalities. Within each modality, MedMIX aggregates complementary embeddings from multiple small expert models; across modalities, it performs learned fusion over available modalities; and during training, it leverages large teacher models to improve deployed representations without additional inference cost. Across three heterogeneous benchmarks (OpenI, MIMIC-IV-MM, and MMIST-ccRCC), MedMIX achieves consistently strong performance while remaining robust under controlled missing-modality perturbations, and further demonstrates sustained robustness under cross-cohort shift on MIMIC-III. These results highlight MedMIX as a practical framework that unifies within-modality expert collaboration, sample-specific cross-modality fusion, and efficient large--small model collaboration while remaining robust to incomplete modalities.

Abstract

Explore similar work

MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs