Learning Across Heterogeneous Feature Spaces with Group Robustness
Organizations: Duke University
Abstract
In clinical applications, patients undergo different combinations of tests, and machine learning models must therefore ensure accurate predictions across heterogeneous feature spaces. Current approaches upweight the patients with the highest loss, without accounting for the limited predictive information in their features preventing their loss from going any lower. Moreover, when patients' feature spaces are fully disjoint, current methods reduce to training independent models, preventing knowledge transfer across them. We propose a framework for group robust learning across heterogeneous feature spaces, where groups are patients sharing a feature space. Since a high loss may simply reflect uninformative features, we instead optimize for worst-group excess loss, the gap between a group's loss and an estimate of its achievable optimum. To allow collaboration between groups, we use group-specific encoders mapping into a shared latent space in which learnable Gaussian anchors cluster patients by outcome. On tabular and imaging medical datasets, including fully disjoint feature settings, our framework outperforms state-of-the-art methods when groups differ in their optimal losses or cannot train a good model on their own, reducing worst-group loss by up to 79% and excess loss by up to 95% with up to 18 fewer parameters.
Figures & tables
| Original groups | Disjoint feature spaces |
|---|---|
| Method WG acc acc WG loss loss WG exc. exc. Params (K) NHANES ERM (common) 65.0 5.9 0.551 0.077 0.295 0.063 11.4 GroupDRO (common) 66.5 5.8 0.523 0.075 0.262 0.057 11.4 Dedicated 60.8 11.0 0.596 0.132 0.317 0.095 35.1 Reweigh 62.1 ∗ 10.0 0.606 ∗ 0.130 0.327 ∗ 0.093 52.5 Flex-MoE 62.0 ∗ 9.3 0.597 ∗ 0.125 0.318 ∗ 0.088 536.4 REMIND 65.2 11.1 0.561 0.139 0.282 0.103 53.2 Ours 65.7 8.5 0.548 0.109 0.280 0.073 30.8 Fed-Heart ERM (common) 64.4 12.9 0.628 0.149 0.518 0.415 11.6 GroupDRO (common) 65.9 11.4 0.619 0.110 0.452 0.319 11.6 Dedicated 74.8 18.7 0.572 0.239 0.257 0.156 45.4 Reweigh 75.7 † 17.8 0.562 0.277 0.210 0.141 53.5 Flex-MoE 76.1 † 17.2 0.591 ∗ 0.118 0.415 ∗ 0.324 537.2 REMIND 75.5 † 18.0 0.570 0.256 0.238 ∗ 0.169 54.5 Ours 74.4 19.1 0.561 0.280 0.206 0.103 38.9 EMBED Dedicated 41.4 36.7 1.287 0.777 0.390 0.488 275.7 Reweigh 64.5 12.5 3.685 ∗ 3.157 2.982 ∗ 3.071 1288.6 Flex-MoE 61.4 16.0 2.442 ∗ 1.916 1.740 ∗ 1.814 927.5 REMIND 64.2 12.7 3.042 ∗ 2.517 2.339 ∗ 2.433 1337.7 Ours 62.2 14.0 1.052 0.534 0.240 0.318 275.7 | Method WG acc acc WG loss loss WG exc. exc. Params (K) NHANES (designed) Dedicated 56.0 16.0 0.679 0.202 0.356 0.130 33.6 Reweigh 56.3 ∗ 14.2 0.696 ∗ 0.213 0.373 ∗ 0.141 52.5 Flex-MoE 59.8 ∗ 10.8 0.687 ∗ 0.183 0.364 ∗ 0.111 536.4 REMIND 62.8 12.3 0.622 ∗ 0.186 0.299 ∗ 0.114 53.2 Ours 64.5 7.4 0.590 0.123 0.278 0.050 29.3 Fed-Heart Dedicated 64.0 28.8 0.626 0.398 0.201 0.146 44.0 Reweigh 63.3 29.4 0.628 † 0.423 0.178 † 0.120 52.5 Flex-MoE 63.2 29.3 0.633 0.422 0.183 † 0.122 535.7 REMIND 63.6 29.1 0.627 † 0.416 0.185 † 0.129 53.6 Ours 63.4 29.4 0.635 0.389 0.219 0.154 37.5 EMBED Dedicated 53.6 19.6 1.153 0.559 0.323 0.490 230.4 Reweigh 58.8 14.4 4.984 ∗ 4.392 4.152 ∗ 4.325 1288.1 Flex-MoE 60.8 12.1 2.311 ∗ 1.661 1.498 ∗ 1.593 927.5 REMIND 58.5 14.8 4.297 ∗ 3.708 3.465 ∗ 3.640 1320.8 Ours 58.0 15.3 1.051 0.442 0.219 0.375 230.4 |
| Dataset | Model | groups | classes | Nearest- anchor acc. |
|---|---|---|---|---|
| NHANES | No anchors | 0.53 | 0.52 | 68.4 |
| Anchors | 0.03 | 0.53 | 68.4 | |
| Fed-Heart | No anchors | 1.08 | 0.29 | 79.5 |
| Anchors | 0.33 | 0.44 | 77.3 | |
| EMBED | No anchors | 0.78 | 0.66 | 67.0 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| NHANES | Fed-Heart | EMBED | |
| Optimizer | Adam | Adam | AdamW |
| Learning rate | to , cosine | , every 5 epochs | |
| Weight decay | |||
| Gradient clipping | none | ||
| Batch size | 128 | 64 | 32 per group |
| Epochs (reported) | 30 (10) | 30 (10) | 20 (10) |
| Shared tests | Disjoint tests | |||
| Method | main | imbalance | rare | rare |
| Dedicated | 0.435 0.085 | 0.831 0.069 | 0.124 0.034 | 0.196 0.038 |
| ERM | 0.531 0.166 | 0.929 0.040 | 0.289 0.123 | 0.620 0.197 |
| GroupDRO | 0.508 0.161 | 0.672 0.115 | 0.270 0.096 | 0.460 0.129 |
| Excess DRO | 0.387 0.088 | 0.653 0.162 | 0.124 0.049 | 0.195 0.062 |
| Anchors + ERM | 0.639 0.066 | 0.924 0.053 | 0.434 0.164 | 0.677 0.186 |