Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer-U-Net
Authors: Yunfei Huang, Elena Van der Vorst, Alexander Richard, Benedikt Sabass
Organizations: Fakultät Physik, Technische Universität Dortmund, Dortmund, 44227, Germany. · Faculty of Physics and Center for NanoScience, Ludwig-Maximilians-Universität München, Munich, 80752, Germany. · Department of Veterinary Sciences, Ludwig-Maximilians-Universität München, Munich, 80752, Germany.
Traction force microscopy (TFM) is a widely used technique for quantifying the forces that cells exert on their surrounding extracellular matrix. Although deep learning methods have recently been applied to TFM data analysis, several challenges remain-particularly achieving reliable inference across multiple spatial scales and integrating additional contextual information such as cell type to improve accuracy. In this study, we propose ViT+UNet, a robust deep learning architecture that integrates a U-Net with a Vision Transformer. Our results demonstrate that this hybrid model outperforms both standalone U-Net and Vision Transformer architectures in predicting traction force fields. Furthermore, ViT+UNet exhibits superior generalization across diverse spatial scales and varying noise levels, enabling its application to TFM datasets obtained from different experimental setups and imaging systems. By appropriately structuring the input data, our approach also allows the inclusion of metadata, in our case cell-type information, to enhance prediction specificity and accuracy.
Figures & tables
Fig. 1: Schematic diagram of traction force microscopy and the architectures of deep learning models used for traction force analysis. (A) Experimental setup: a single cell adheres to the surface of an elastic gel, and the traction force field is calculated from the measured displacement field. (B) Architecture of U-Net, consisting of downsampling, a middle block, and upsampling stages. The black arrows indicate 2D convolutions; red arrows represent downsampling through 2D convolutions; blue arrows denote 2D transposed convolutions used for upsampling; and purple arrows indicate skip connections (copy operations). The input and output of the model correspond to the displacement field and traction force field, respectively. (C) Architecture of Vision Transformer (ViT), which includes image pre-processing, a transformer encoder, and a convolutional decoder. Note that the transformer encoder contains L layers. (D) Architecture of the hybrid deep learning model ViT+UNet , which combines a Vision Transformer with a U-Net framework. In this design, the middle block of U-Net is replaced by the Vision Transformer module.
Fig. 2: The hybrid ViT+UNet architecture consistently outperforms both standalone U-Net and Vision Transformer models in predicting traction force fields. (A) Representative cell image and its corresponding displacement field used as input (i). The reference traction force field, calculated using Bayesian Fourier-transform traction cytometry (BFTTC), is shown in (ii). Note that the vector field is plotted only for regions where the traction magnitude exceeds 50Pa , skipping every 15 indices for clarity. (B, i–iii) Comparison of predicted traction force fields obtained from U-Net, Vision Transformer (ViT), and ViT+UNet models. (C) Zoomed-in views corresponding to the colored boxes in panel (A-ii). (D, i–iii) Joint probability distributions of force magnitudes for the predictions and references, p(FRef,FNN) , obtained from U-Net, ViT, and ViT+UNet models. Forces larger than 150Pa are considered; better predictions with higher probability density values lie along the diagonal dotted line. (E) Statistical comparison of normalized root mean square error (NRMSE; i) and correlation coefficients (ii) across 34 test cases. These two metrics are defined in Eqs. ( 9 ) and ( 10 ). Lower NRMSE values indicate more accurate predictions, while correlation coefficients closer to 1 reflect stronger agreement with the reference data. Error bars represent standard deviations.
Model
Parameters of network
Initial training rate
Training time per epoch
U-Net
26536178
0.0002
7.056s
ViT
24490148
0.0002
7.485s
ViT+UNet
24357546
0.0002
9.679s
TABLE I: Overview of the architecture and training parameters for the U-Net, Vision Transformer (ViT), and ViT+UNet.
Fig. 3: The ViT+UNet architecture exhibits superior generalization across diverse spatial scales. (A) Schematic diagram illustrating the inference procedure for different spatial scales. The scale ratio s is defined as s=l/L , where l is the scaled length used during inference and L is the original length used during training, as shown in (i). For inference cases with s<1 and s>1 , these are referred to as zoom-in and zoom-out scenarios, respectively, illustrated in (ii) and (iii). (B) Example of an original cell image with its corresponding displacement field (i) and traction force field obtained using Bayesian Fourier-transform traction cytometry (BFTTC) in (ii). (C) Generalization inference results for the zoom-in ( s=0.6 ; ii–iv) and zoom-out ( s=1.67 ; vi–viii) scenarios obtained from U-Net, ViT, and ViT+UNet models compared with the reference data shown in panels (i, v). (D) Statistical summary showing normalized root mean square error (NRMSE; i) and correlation coefficients (ii) for ViT+UNet , U-Net, and ViT across 34 test cases over scale ratios ranging from s=0.25 to 2.3 . Error bars represent standard deviations.
Fig. 4: The ViT+UNet model generalizes well in predicting traction forces from displacement data under varying noise levels. (A-i) Dimensionless displacement field showing both magnitude (color map) and vector directions. (A-ii) Histogram of variance values and their average computed from dimensionless displacement fields in the training dataset. (A-iii) Gaussian noise magnitude and corresponding vector field at an 8% noise level. (A-iv) Dimensionless displacement field after adding 8% Gaussian noise, illustrating both magnitude (color map) and vector directions. (B) Reference traction force field obtained using Bayesian Fourier-transform traction cytometry (BFTTC); colors indicate traction force magnitudes. (C) Zoomed-in views of traction force fields under noise-free conditions [(ii–iv)] and at an 8% noise level [(v–vii)] predicted by three models—U-Net, ViT, and ViT+UNet —corresponding to the green box in panel (B), compared with the reference region shown in (i). (D) Plots of normalized root mean square error (NRMSE; i) and correlation coefficients (ii) for all three models at different noise levels ranging from 0 to 9%. Error bars represent standard deviations.
Fig. 5: Incorporating cell-type information improves prediction specificity and accuracy. (A) Architecture of the hybrid deep learning model ViT+UNet , which can also embed cell-type text information as an additional input. (B) Representative cell image with its corresponding displacement field and annotated cell type (C2C12) in panel (i). The reference traction force field obtained using Bayesian Fourier-transform traction cytometry (BFTTC) is shown in panel (ii). (C, i–iv) Zoomed-in views of predicted traction force fields obtained from ViT, ViT+cell-type, ViT+UNet , and ViT+UNet +cell-type models within the blue box region indicated in panel (B-ii). (D) Statistical comparison of normalized root mean square error (NRMSE; i) and correlation coefficients (ii) among the four methods: ViT, ViT+cell-type, ViT+UNet , and ViT+UNet +cell-type. Error bars represent standard deviations. (E, i–iv) Joint distributions of force magnitudes p(FRef,FNN) binned for the deep learning models ViT, ViT with added cell-type input (ViT+cell-type), ViT+UNet , and ViT+UNet with added cell-type input ( ViT+UNet +cell-type).
Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.
Albert Dominguez Mantes, Gioele La Manno, Martin Weigert
Swiss Federal Institute of Technology Lausanne, Switzerland · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Technische Universität Dresden, Germany
Vision Transformers (ViTs) have demonstrated strong performance in many computer vision tasks but do not naturally extend beyond the natural image domain. One important example is Fourier-space data, which is used to represent a rich and diverse set of modalities, including medical imaging and microscopy. Standard ViT tokenization partitions an image into square patches on a Cartesian grid, which poorly fit the frequency and phase structure of Fourier-space data. We introduce Non-Uniform ViT, a simple, general-purpose vision transformer that redesigns image tokenization, data representation, and positional encoding to respect this structure. Experiments on Fourier-domain image classification, 2D crystal lattice identification, MRI annotation, and cryo-EM protein heterogeneity analysis show that NUViT consistently captures the structure of Fourier data better than standard architectures.
Understanding tissue organisation in multiplexed imaging requires modelling both cellular phenotypes and their spatial context. Existing approaches typically rely on handcrafted features, such as marker intensity statistics or cell-type proportions, which often fail to scale or generalise across cohorts with heterogeneous marker panels. We introduce MUL-T, a lightweight transformer framework that reframes tissue architecture as a masked contextual prediction task over discrete cell tokens. By learning contextualised [CLS] embeddings without task-specific supervision, the model captures higher-order cellular interactions while remaining computationally efficient. We evaluate MUL-T on several clinically relevant downstream tasks, including core-level tumour pattern classification, patient-level grading, PD-L1 positivity prediction, and cross-dataset treatment response prediction. Across tasks, MUL-T consistently outperforms classical feature-based baselines and achieves performance comparable to a foundation ViT model, despite substantially fewer parameters and lower training cost.
Farzaneh Seyedshahi, Kai Rakovic, Adalberto Claudio Quiros +2
Cancer Research UK Scotland Institute, Glasgow, UK · University of Glasgow, Glasgow, UK