cs.CVNov 3, 2026

Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease

Authors: Ankana Banerjee, Mark van der Giezen, Ole Gunnar Aasprong, Emiel A. M. Janssen, Øyvind Steinsbø, Tore Grimstad, Kjersti Engan

Organizations: University of Stavanger, Norway · Stavanger University Hospital, Norway · University of Greenwich, UK · University of Bergen, Norway

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal AI fusion of histology and clinical data is an active research area in computational pathology, but fusion methods are rarely compared under identical conditions and across different clinical endpoints. We present a modular framework in which slide-level embeddings, obtained by training an attention-based aggregator over patch embeddings from a frozen foundation model, and per-visit clinical embeddings are fed into 15 interchangeable fusion methods with a shared output dimension and prediction head (except decision-level methods). Using a prospective cohort of newly diagnosed inflammatory bowel disease patients, we predicted STRIDE-II-aligned remission outcomes from baseline or follow-up visits. The best fusion method reached macro-F1 of 0.57-0.75, outperforming both single-modality models in four of six scenarios, and differed by outcome and input. Image reliance, the drop in balanced accuracy when the slide embedding is removed, tended to follow the remission type - highest for histological and lowest for endoscopic remission; methods with similar accuracy relied on the slide to very different degrees.

Figures & tables

Figure 1: Whole Slide image and clinical data embedding generation framework.
TypeMethodDescriptionEquation
ConcatenationconcatEmbeddings are joined side by side.z=[x;c]∈R512\mathbf{z}=[\mathbf{x};\mathbf{c}]\in\mathbb{R}^{512}
 tabAESame as concat , but the clinical encoder EϕE_{\phi} is first pretrained as an autoencoder, where a decoder DψD_{\psi} reconstructs the continuous and ordinal features r\mathbf{r} from c\mathbf{c} .z=[x;c]\mathbf{z}=[\mathbf{x};\mathbf{c}] , c=Eϕ(⋅)\mathbf{c}=E_{\phi}(\cdot) , LAE=∥Dψ(c)−r∥22\mathcal{L}_{\mathrm{AE}}=\|D_{\psi}(\mathbf{c})-\mathbf{r}\|_{2}^{2}
 early concatc\mathbf{c} is appended to every slide embedding, projected, then attention-pooled.s~j′=ReLU(W[s~j;c])\tilde{\mathbf{s}}^{\prime}_{j}=\mathrm{ReLU}(\mathbf{W}[\tilde{\mathbf{s}}_{j};\mathbf{c}]) , z=pool({s~j′})\mathbf{z}=\mathrm{pool}(\{\tilde{\mathbf{s}}^{\prime}_{j}\})
Modulationfilmc\mathbf{c} produces a scale γ\boldsymbol{\gamma} and shift β\boldsymbol{\beta} that modulate x\mathbf{x} [ 16 ] .z=(1+γ(c))⊙x+β(c)\mathbf{z}=(\mathbf{1}+\boldsymbol{\gamma}(\mathbf{c}))\odot\mathbf{x}+\boldsymbol{\beta}(\mathbf{c})
 early filmSame modulation as film , but applied to every slide embedding s~j\tilde{\mathbf{s}}_{j} before pooling.s~j′=(1+γ(c))⊙s~j+β(c)\tilde{\mathbf{s}}^{\prime}_{j}=(\mathbf{1}+\boldsymbol{\gamma}(\mathbf{c}))\odot\tilde{\mathbf{s}}_{j}+\boldsymbol{\beta}(\mathbf{c}) , z=pool({s~j′})\mathbf{z}=\mathrm{pool}(\{\tilde{\mathbf{s}}^{\prime}_{j}\})
GatinggmuEach modality is projected, then the tanh⁡\tanh -activated outputs ( hx\mathbf{h}_{x} , hc\mathbf{h}_{c} ) are mixed by a sigmoid gate g\mathbf{g} that weights the two vectors [ 17 ] .hx=tanh⁡(Wxx)\mathbf{h}_{x}=\tanh(\mathbf{W}_{x}\mathbf{x}) , hc=tanh⁡(Wcc)\mathbf{h}_{c}=\tanh(\mathbf{W}_{c}\mathbf{c}) , g=σ(Wg[x;c])\mathbf{g}=\sigma(\mathbf{W}_{g}[\mathbf{x};\mathbf{c}]) , z=g⊙hx+(1−g)⊙hc\mathbf{z}=\mathbf{g}\odot\mathbf{h}_{x}+(\mathbf{1}-\mathbf{g})\odot\mathbf{h}_{c}
Table 1: Fusion methods. x∈R256\mathbf{x}\in\mathbb{R}^{256} is the pooled image embedding, c∈R256\mathbf{c}\in\mathbb{R}^{256} the clinical embedding, and s~j\tilde{\mathbf{s}}_{j} the projected slide embeddings of the input bag(s) before pooling. z\mathbf{z} is the fused vector passed to the prediction head (512-D for concat, tabAE and cross-attention, 256-D otherwise); decision-level methods output logits ℓ\boldsymbol{\ell} or class probabilities p\mathbf{p} per modality.
Figure 2: Fusion pipeline. Slide image embeddings and clinical embeddings enter one of 15 interchangeable fusion modules.
Figure 3: Best fusion method in each of the six scenarios.
Figure 4: Change in balanced accuracy when the slide embedding is replaced by zeros, by outcome and input visit (Absolute cohort). Bars show the median over the fusion methods above chance, and whiskers show the 25th to 75th percentile.
More in this topic

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 800 papers

Explore similar work

CardsList

  1. Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

    Oct 27, 2025Hongyi Wang, Zhengjie Zhu, Junlin Hou +13Multimodal IRComputational Pathology

    Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China. · School of Medicine, Wake Forest University, Winston-Salem, NC, USA. +11

  2. Plug-and-Play Logit Fusion for Heterogeneous Pathology Foundation Models

    Apr 9, 2026Gexin Huang, Anqi Li, Yusheng Tan +4Computational PathologyEnsemble Learning

    University of British Columbia, Vancouver, BC, Canada · Vector Institute, Toronto, ON, Canada · Washington University in St. Louis, St. Louis, MO, USA

  3. Multimodal Model for Computational Pathology:Representation Learning and Image Compression

    Mar 19, 2026Peihang Wu, Zehong Chen, Lijian XuComputational PathologyWhole-Slide Image Analysis

    Shenzhen University of Advanced Technology