Multi-channel images encode heterogeneous channel semantics aligned over a shared spatial structure, which requires models to jointly capture globally shared structure and channel-specific variations. Recent multi-channel Vision Transformers attempt to address this challenge by augmenting the [CLS] token with memory tokens to organize multi-channel representations under varying channel compositions. However, we find that these context tokens collapse onto a few dominant channels under high inter-channel redundancy, producing redundant representations and a biased global summary that fails to reliably support composition-aware channel interpretation. We propose MuCa-ViT (Multi-Channel Composition-Aware Vision Transformer), a unified framework that addresses these limitations through two coupled mechanisms. Factorial Token Learning (FTL) enforces token-level decorrelation among context tokens, encouraging the [CLS] token to capture globally shared semantics while memory tokens preserve complementary channel-specific cues. Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] as a semantic anchor to generate sample-wise modulation signals, which enable composition-aware interpretation of channels within a unified backbone rather than relying on static channel embeddings. Experiments on JUMP-CP, CHAMMI, and So2Sat show that MuCa-ViT outperforms prior multi-channel ViTs across all three benchmarks, and the magnitude of improvement aligns with the inter-channel redundancy of each dataset (ρ∈[0.19,0.77]), validating the diagnostic analysis that motivates our design. Our code is available at https://github.com/kimdanni/MuCa-ViT .
Figures & tables
Figure 1: Representation collapse and its mitigation across datasets. Δ values are computed against our reproduced ChA-MAEViT [ 18 ] . (a) CLS channel coverage across datasets ( HCLS% , y-axis). Near-max on CHAMMI but drops sharply on JUMP-CP and So2Sat, indicating that CLS coverage concentrates on few channels as C grows. (b) Consistent recovery. Our method yields positive ΔHCLS% and ΔAcc on all three datasets. (c) Larger gains in higher-redundancy regimes. ΔAcc is larger on benchmarks with higher inter-channel redundancy ρ , consistent with redundancy being a major structural factor that promotes CST collapse.
Figure 3: Overall architecture of MuCa-ViT. Multi-channel images are patchified using a shared projection and serialized with independent channel embeddings. Factorial Token Learning (FTL) enforces structured decorrelation among Context Semantic Tokens (CSTs), consisting of the global [CLS] token and memory tokens, via a token-wise Gram matrix constraint in ( 3 ). Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] token as a semantic anchor to generate sample-wise modulation parameters (γ,β) , adaptively aligning patch representations under varying channel configurations.
Model
CHAMMI [ 7 ]
JUMP-CP [ 6 ]
So2Sat [ 32 ]
Avg score
Full
Partial
Full
Partial
HyperNetViT [ 10 ]
56.08 ± 0.41
47.07 ± 0.47
42.43 ± 0.65
60.73 ± 0.24
41.88 ± 0.85
DepthwiseViT [ 7 ]
61.80 ± 0.43
49.86 ± 0.45
44.98 ± 0.71
60.41 ± 0.22
43.41 ± 1.10
TemplateMixingViT [ 20 , 21 , 23 ]
58.16 ± 0.42
52.48 ± 0.27
43.85 ± 0.73
55.86 ± 0.10
37.28 ± 0.34
ChAda-ViT [ 4 ]
63.93 ± 0.42
65.03 ± 0.98
42.15 ± 2.33
56.98 ± 0.46
12.38 ± 2.03
ChannelViT [ 2 ]
64.97 ± 0.58
67.51 ± 0.35
56.49 ± 0.53
61.03 ± 0.17
46.16 ± 0.40
Table 1: Comparison of test accuracy (%) of channel-adaptive models. Full denotes inference using all channels, while Partial evaluates on a subset of channels. All results are reported as the mean ± standard deviation of three independent runs ( n=3 ). Values marked with ∗ were reproduced using the official implementation.
Model
Full
Partial
ChA-MAEViT
90.35
68.10
+ FTL
91.58
68.23
+ CAM
92.46
70.96
+ FTL + CAM
94.35
71.81
Table 2: Component ablation on JUMP-CP. CAM is applied to the final encoder block only.
Figure 4: SVD-based subspace analysis of multi-channel representations. The 384-dimensional singular spectrum is partitioned into 55 groups of 7 consecutive singular vectors each. (a) t-SNE of ZtopN (Group 1, dtop=7 ), capturing globally shared channel semantics. (b) t-SNE of ZmidN (Groups 3–35, dmid1=15,dmid2=245 ), corresponding to residual semantic components. In the middle spectral space, the baseline collapses into an entangled representation, whereas MuCa-ViT preserves clear channel-wise clustering.
Model
Full
Partial
Model
Full
Partial
IC-ViT mem
86.68
66.41
DiChaViT mem
64.83
54.69
+ FTL + CAM
87.83
68.50
+ FTL + CAM
68.09
57.37
Table 3: Plug-in compatibility of FTL and CAM on IC-ViT [ 13 ] and DiChaViT [ 19 ] . JUMP-CP is reported under full and partial channel settings. Models with superscript mem denote architectures augmented with memory tokens using the same initialization as MuCa-ViT.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
JUMP-CP
CHAMMI
So2Sat
Hardware and batch
GPUs (NVIDIA A6000)
2
1
1
Train batch size
32
80
128
Eval batch size
300
256
512
Num workers
1
3
3
Optimization
Appendix
Table 4: Training and evaluation configurations used to reproduce ChA-MAEViT [ 18 ] (denoted ChA-MAEViT ∗ in Table 1 ). MuCa-ViT is trained under identical configurations except for the memory token count and the additional FTL/CAM hyperparameters specified in Appendix B.3.3 , B.2.2 .
Figure 5: Effect of the number of memory tokens on JUMP-CP, So2Sat, and CHAMMI. Our model consistently achieves optimal performance with fewer memory tokens compared to the default settings of ChA-MAEViT.
Variant
Full
Partial
Equal-weight ablation
94.09
71.81
MuCa-ViT (asymmetric)
95.41
72.26
Appendix
Table 5: Ablation on asymmetric weighting in FTL. JUMP-CP accuracy.
Figure 6: Sensitivity analysis of the FTL decorrelation weight λFTL on JUMP-CP under full and partial channel settings. Performance peaks at a moderate regularization strength and remains stable across a broad range of λFTL .
Figure 7: t-SNE visualization of the learned CAM modulation parameters ( γ,β ) across different channel configurations. Each point is obtained by feeding the same underlying samples with a different channel subset. Distinct clusters emerge for different channel subsets, organized by configuration rather than sample identity, indicating that CAM generates configuration-aware modulation signals rather than static embeddings.
Conditioning Source
Full
Partial
External conditioning
92.57
70.47
Intrinsic conditioning (w/o FTL refinement)
93.18
71.12
FTL-refined intrinsic conditioning (MuCa-ViT)
95.41
72.26
Appendix
Table 6: Ablation on the conditioning source in CAM. JUMP-CP accuracy.
Figure 8: Effect of CAM insertion depth across different transformer blocks. Each x-axis label denotes the starting block from which CAM is applied to all subsequent blocks (e.g., Block 9 indicates CAM applied to blocks 9–11).
Variant
Full
Partial
ChA-MAEViT (Baseline)
90.35
68.10
+ FTL (Ours w/o CAM)
91.58
68.23
+ FTL + CAM (Top, first 2 blocks)
92.18
70.66
+ FTL + CAM (Bottom, last 2 blocks)
95.41
72.26
Appendix
Table 7: Ablation on CAM placement across network depth. JUMP-CP accuracy.
WTC [ 27 ]
HPA [ 24 ]
CP [ 28 ]
Model
Task1
Task2
Task1
Task2
Task3
Task1
Task2
Task3
Task4
HyperNetViT [ 10 ]
58.83
45.78
88.78
82.70
52.52
82.13
53.74
23.16
7.42
DepthwiseViT [ 7 ]
69.81
52.19
91.65
88.04
54.78
81.24
54.08
23.21
5.87
TemplateMixingViT [ 20 , 21 , 23 ]
61.66
51.52
85.01
79.91
49.69
77.45
48.83
22.56
5.60
ChAda-ViT [ 4 ]
77.58
67.18
87.49
75.94
45.41
83.92
45.58
21.94
6.28
ChannelViT [ 2 ]
78.36
67.58
83.93
76.73
47.97
77.70
55.16
21.89
6.38
Appendix
Table 8: F1 scores of multi-channel ViT models on the CHAMMI benchmark. ChA-MAEViT ∗ denotes our reproduction under the same training environment as MuCa-ViT; both are single-run results. Bold indicates the best result within each group, and underline indicates the best result overall.
Number of channels at inference
Method
8
7
6
5
4
3
2
1
DiChaViT [ 19 ]
69.19
61.91 ± 9.3
54.49 ± 12.4
46.35 ± 13.4
38.00 ± 12.4
30.09 ± 9.3
23.97 ± 4.9
20.90 ± 1.6
ChA-MAEViT [ 18 ]
90.73
83.36 ± 8.3
74.55 ± 11.7
63.46 ± 13.8
50.85 ± 13.9
38.13 ± 11.2
27.62 ± 6.4
21.59 ± 2.1
MuCa-ViT
95.41
89.46 ± 6.28
80.06 ± 11.32
69.75 ± 12.84
55.84 ± 13.80
41.18 ± 11.65
28.79 ± 6.84
21.54 ± 2.25
Appendix
Table 9: Performance comparison across different numbers and combinations of channels at inference. Mean ± std over all channel combinations at each budget.
Model
JUMP-CP (Full)
JUMP-CP (Partial)
CHAMMI
ChA-MAEViT*
92.22
70.24
76.20
MuCa-ViT (Ours)
94.28
74.31
76.72
Appendix
Table 10: Performance at ViT-Base scale.
Model
Inference Latency
FLOPs
ChA-MAEViT
1.084s
249.48
MuCa-ViT (Ours)
1.235s
249.33
Appendix
Table 11: Computational complexity and inference latency at ViT-Small (batch size 32).
Model
mem
JUMP-CP
CHAMMI
Full
Partial
ChA-MAEViT ∗
3
90.24
67.35
72.66
ChA-MAEViT ∗ (best)
4
90.35
68.10
74.04
MuCa-ViT (Ours)
3
95.41
72.26
76.46
Appendix
Table 12: Performance comparison under matched and best-performing memory-token budgets. MuCa-ViT outperforms ChA-MAEViT ∗ at the same token budget and exceeds its best-performing configuration with fewer memory tokens. ChA-MAEViT ∗ denotes our reproduced baseline. Results are from single runs.
Figure 9: Inter-channel diversity Div(g) of reconstructed spectral components across spectral groups g on JUMP-CP. MuCa-ViT (orange) maintains consistently higher diversity than the baseline (blue), especially in the middle spectral bands.
Method
Brightfield-only Acc. (%)
After reinjection Acc. (%)
Gain (pp)
MSE ↓
Fourier error ↓
ChA-MAEViT
22.77
34.12
+11.35
0.028
0.343
MuCa-ViT (Ours)
23.54
37.39
+13.85
0.027
0.321
Appendix
Table 13: Missing-channel reconstruction and reinjection on JUMP-CP. Five fluorescence channels are reconstructed from three observed brightfield channels and reinjected into the frozen classifier. Reinjection gain is measured relative to brightfield-only accuracy.
Figure 10: Per-image channel-channel cosine similarity averaged over 2,048 samples for each evaluation group. Darker blue indicates higher similarity. JUMP-CP and CHAMMI/WTC exhibit strong redundancy across channels, whereas So2Sat is largely uncorrelated except for a few pairs
Dataset / chunk
C
Redundancy
ChA-MAEViT*
Δ Full Acc.
So2Sat [ 32 ]
18
0.19
66.44
+1.53
CHAMMI (avg) [ 7 ]
3-4
0.46
73.95
+2.24
JUMP-CP [ 6 ]
8
0.77
90.52
+4.62
Appendix
Table 14: Channel redundancy and corresponding performance gain. Redundancy is the off-diagonal mean of the channel-channel cosine similarity matrix (Fig. 10 ). Δ Full Acc. is computed against our reproduced ChA-MAEViT* under the same training environment (cf. Table 1 ).
Figure 11: Comparison of Context Semantic Token (CST) behaviors between baseline ChA-MAEViT [ 18 ] and our method. (a) Gram-matrix-based redundancy analysis among CSTs. (b) Attention heatmaps showing spatial focus for each token.
Figure 12: Dot-product similarity between channel-wise aggregated patch representations (x-axis) and the Context Semantic Tokens, including the [CLS] token and memory tokens Mi(i=1,2,3) . In the baseline, both [CLS] and memory tokens exhibit strong alignment toward dominant channels, indicating severe semantic collapse and redundancy, whereas the proposed method yields well-disentangled similarity distributions.
HCLS%
HCST%
τ
ChA-MAEViT
MuCa-ViT
Δ
ChA-MAEViT
MuCa-ViT
Δ
JUMP-CP
0.01
10.23
42.31
+32.08
39.74
76.65
+36.91
0.02
28.79
63.96
+35.17
57.07
89.10
+32.03
0.04
60.66
88.74
+28.08
78.64
98.54
+19.90
0.06
81.93
96.61
+14.68
90.17
99.63
+9.46
Appendix
Table 15: Cosine-based channel coverage across softmax temperatures. Coverage is reported as a percentage of maximum entropy, and Δ denotes the improvement of MuCa-ViT over ChA-MAEViT in percentage points. MuCa-ViT maintains higher coverage at every tested temperature.
Condition
Setting
ChA-MAEViT
MuCa-ViT
Gap
Clean
Full
90.35
95.41
+5.06
Partial
68.10
72.26
+4.16
η=0.5
Full
83.24 ( −7.11 )
89.69 ( −5.72 )
+6.45
Partial
58.67 ( −9.43 )
67.78 ( −4.48 )
+9.11
η=0.7
Full
71.64 ( −18.71 )
80.14 ( −15.27 )
+8.50
Partial
48.68 ( −19.42 )
58.06 ( −14.20 )
+9.38
Appendix
Table 16: Prediction under final [CLS] corruption on JUMP-CP. Accuracy is reported in percent; the gap and parenthesized changes from clean accuracy are reported in percentage points.