cs.CVNov 6, 2026

FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics

Authors: Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang

Organizations: School of Computer Science and Technology, Anhui University, Hefei 230031, China · Institute of Plasma Physics, Chinese Academy of Sciences, Hefei 230031, China

Abstract

Visible-light videos and multivariate diagnostic signals provide complementary observations of tokamak discharges. However, existing fusion pretraining methods, primarily learn representations by reconstructing or predicting observations, leaving the correspondence between these modalities underexplored. Representation learning is further challenged by the fact that plasma-related regions occupy only part of each video frame and diagnostic variables have substantially different sampling rates. We propose FusionCLIP, a multimodal contrastive pretraining framework. Spatiotemporal Stability Selection retains informative visual tokens, while a Temporal Query Encoder integrates asynchronous diagnostic signals using their observation timestamps. Contrastive learning then aligns the two modality representations. We construct EAST Multimodal Diagnostics (EAST-MMD), comprising 1307 EAST discharges, and evaluate the model on three independently fine-tuned tasks: disruption prediction, edge-localized mode (ELM) classification, and H98H_{98} regression. FusionCLIP achieves an aggregate score of 89.07%, exceeding the best general-purpose pretraining configuration and an in-domain contrastive pretraining configuration without Spatiotemporal Stability Selection by 6.27 and 2.12 percentage points, respectively. Ablations and attention visualizations further support the contributions of its key components.

Figures & tables

Explore similar work

CardsList
  1. Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST

    Jul 5, 2026Qiang Chen, Xiao Wang, Hao Si +9Cross-Modal Knowledge DistillationEfficient Multimodal Inference

  2. TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics

    Feb 16, 2026Tobia Boschi, Andrea Loreti, Nicola C. Amorisco +13Multimodal Foundation ModelsMultimodal Pretraining

  3. SpectralTF-MAE: A Pre-trained Spectral Foundation Model for Tokamak Fusion Plasma Diagnostics

    Oct 28, 2026Xiao Wang, Zhijie Lu, Duoqing Yang +7Representation LearningSelf-Supervised Pre-Training