FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics
Organizations: School of Computer Science and Technology, Anhui University, Hefei 230031, China · Institute of Plasma Physics, Chinese Academy of Sciences, Hefei 230031, China
Abstract
Visible-light videos and multivariate diagnostic signals provide complementary observations of tokamak discharges. However, existing fusion pretraining methods, primarily learn representations by reconstructing or predicting observations, leaving the correspondence between these modalities underexplored. Representation learning is further challenged by the fact that plasma-related regions occupy only part of each video frame and diagnostic variables have substantially different sampling rates. We propose FusionCLIP, a multimodal contrastive pretraining framework. Spatiotemporal Stability Selection retains informative visual tokens, while a Temporal Query Encoder integrates asynchronous diagnostic signals using their observation timestamps. Contrastive learning then aligns the two modality representations. We construct EAST Multimodal Diagnostics (EAST-MMD), comprising 1307 EAST discharges, and evaluate the model on three independently fine-tuned tasks: disruption prediction, edge-localized mode (ELM) classification, and regression. FusionCLIP achieves an aggregate score of 89.07%, exceeding the best general-purpose pretraining configuration and an in-domain contrastive pretraining configuration without Spatiotemporal Stability Selection by 6.27 and 2.12 percentage points, respectively. Ablations and attention visualizations further support the contributions of its key components.
Figures & tables
| Method | Pre-training | Disruption | ELM | H98 | Joint | |||||||
| TPR | FPR | F1 | AUC | AWT (ms) | ACC | F1 | AUC | MAE | ||||
| Scratch | – | 91.67% | 16.44% | 75.86% | 94.29% | 1293.95 | 89.43% | 73.46% | 96.46% | 0.034669 | 90.10% | 79.81% |
| BEiT | ImageNet-1K | 79.17% | 15.07% | 70.37% | 89.61% | 383.79 | 88.95% | 71.62% | 96.17% | 0.025314 | 93.18% | 78.39% |
| DINO | ImageNet-1K | 75.00% | 2.74% | 81.82% | 93.44% | 310.06 | 89.86% | 73.55% | 96.45% | 0.026186 | 93.04% | 82.80% |
| MoCo v3 | ImageNet-1K | 75.00% | 12.33% | 70.59% | 89.55% | 402.89 | 89.22% | 73.41% | 96.67% | 0.028204 | 92.40% | 78.80% |
| iBOT | ImageNet-1K | 83.33% | 9.59% | 78.43% | 92.92% | 385.15 | 89.18% | 73.29% | 96.45% | 0.030527 | 91.88% | 81.20% |
| Pre-training | SSS | TQE | Disruption | ELM | H98 | Joint | |||||||
| TPR | FPR | F1 | AUC | AWT (ms) | ACC | F1 | AUC | MAE | |||||
| ✗ | ✗ | ✗ | 95.83% | 5.48% | 90.20% | 98.00% | 326.83 | 88.75% | 69.88% | 96.22% | 0.032112 | 91.60% | 83.89% |
| ✗ | ✗ | ✓ | 83.33% | 2.74% | 86.96% | 98.06% | 73.85 | 90.95% | 76.25% | 97.81% | 0.026874 | 93.13% | 85.44% |
| ✗ | ✓ | ✓ | 95.83% | 4.11% | 92.00% | 97.43% | 96.39 | 90.57% | 77.21% | 97.86% | 0.030168 | 92.23% | 87.15% |
| ✓ | ✓ | ✓ | 95.83% | 2.74% | 93.88% | 99.54% | 222.17 | 90.92% | 79.82% | 97.61% | 0.025362 | 93.50% | 89.07% |
| Mask Ratio | Disruption | ELM | H98 | Joint | |||||||
| TPR | FPR | F1 | AUC | AWT (ms) | ACC | F1 | AUC | MAE | |||
| Time Series | 87.50% | 9.59% | 80.77% | 94.09% | 396.00 | 90.76% | 75.63% | 97.42% | 0.026838 | 92.66% | 83.02% |
| Full Token | 91.67% | 5.48% | 88.00% | 95.49% | 149.41 | 91.18% | 79.18% | 97.82% | 0.025787 | 93.68% | 86.95% |
| Random 50% | 91.67% | 5.48% | 88.00% | 96.00% | 60.05 | 90.36% | 77.67% | 97.46% | 0.034071 | 91.09% | 85.59% |
| SSS 25% | 79.17% | 2.74% | 84.44% | 96.52% | 156.11 | 90.99% | 79.08% | 97.76% | 0.023024 | 94.11% | 85.88% |
| SSS 50% | 95.83% | 2.74% | 93.88% | 99.54% | 222.17 | 90.92% | 79.82% | 97.61% | 0.025362 | 93.50% | 89.07% |
Multimodal Pretraining
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
See all 193 papers