Multi-channel images encode heterogeneous channel semantics aligned over a shared spatial structure, which requires models to jointly capture globally shared structure and channel-specific variations. Recent multi-channel Vision Transformers attempt to address this challenge by augmenting the [CLS] token with memory tokens to organize multi-channel representations under varying channel compositions. However, we find that these context tokens collapse onto a few dominant channels under high inter-channel redundancy, producing redundant representations and a biased global summary that fails to reliably support composition-aware channel interpretation. We propose MuCa-ViT (Multi-Channel Composition-Aware Vision Transformer), a unified framework that addresses these limitations through two coupled mechanisms. Factorial Token Learning (FTL) enforces token-level decorrelation among context tokens, encouraging the [CLS] token to capture globally shared semantics while memory tokens preserve complementary channel-specific cues. Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] as a semantic anchor to generate sample-wise modulation signals, which enable composition-aware interpretation of channels within a unified backbone rather than relying on static channel embeddings. Experiments on JUMP-CP, CHAMMI, and So2Sat show that MuCa-ViT outperforms prior multi-channel ViTs across all three benchmarks, and the magnitude of improvement aligns with the inter-channel redundancy of each dataset (ρ∈[0.19,0.77]), validating the diagnostic analysis that motivates our design. Our code is available at https://github.com/kimdanni/MuCa-ViT .
Figures & tables
Figure 1: Representation collapse and its mitigation across datasets. Δ values are computed against our reproduced ChA-MAEViT [ 18 ] . (a) CLS channel coverage across datasets ( HCLS% , y-axis). Near-max on CHAMMI but drops sharply on JUMP-CP and So2Sat, indicating that CLS coverage concentrates on few channels as C grows. (b) Consistent recovery. Our method yields positive ΔHCLS% and ΔAcc on all three datasets. (c) Larger gains in higher-redundancy regimes. ΔAcc is larger on benchmarks with higher inter-channel redundancy ρ , consistent with redundancy being a major structural factor that promotes CST collapse.
Figure 3: Overall architecture of MuCa-ViT. Multi-channel images are patchified using a shared projection and serialized with independent channel embeddings. Factorial Token Learning (FTL) enforces structured decorrelation among Context Semantic Tokens (CSTs), consisting of the global [CLS] token and memory tokens, via a token-wise Gram matrix constraint in ( 3 ). Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] token as a semantic anchor to generate sample-wise modulation parameters (γ,β) , adaptively aligning patch representations under varying channel configurations.
Model
CHAMMI [ 7 ]
JUMP-CP [ 6 ]
So2Sat [ 32 ]
Avg score
Full
Partial
Full
Partial
HyperNetViT [ 10 ]
56.08 ± 0.41
47.07 ± 0.47
42.43 ± 0.65
60.73 ± 0.24
41.88 ± 0.85
DepthwiseViT [ 7 ]
61.80 ± 0.43
49.86 ± 0.45
44.98 ± 0.71
60.41 ± 0.22
43.41 ± 1.10
TemplateMixingViT [ 20 , 21 , 23 ]
58.16 ± 0.42
52.48 ± 0.27
43.85 ± 0.73
55.86 ± 0.10
37.28 ± 0.34
ChAda-ViT [ 4 ]
63.93 ± 0.42
65.03 ± 0.98
42.15 ± 2.33
56.98 ± 0.46
12.38 ± 2.03
ChannelViT [ 2 ]
64.97 ± 0.58
67.51 ± 0.35
56.49 ± 0.53
61.03 ± 0.17
46.16 ± 0.40
Table 1: Comparison of test accuracy (%) of channel-adaptive models. Full denotes inference using all channels, while Partial evaluates on a subset of channels. All results are reported as the mean ± standard deviation of three independent runs ( n=3 ). Values marked with ∗ were reproduced using the official implementation.
Model
Full
Partial
ChA-MAEViT
90.35
68.10
+ FTL
91.58
68.23
+ CAM
92.46
70.96
+ FTL + CAM
94.35
71.81
Table 2: Component ablation on JUMP-CP. CAM is applied to the final encoder block only.
Figure 4: SVD-based subspace analysis of multi-channel representations. The 384-dimensional singular spectrum is partitioned into 55 groups of 7 consecutive singular vectors each. (a) t-SNE of ZtopN (Group 1, dtop=7 ), capturing globally shared channel semantics. (b) t-SNE of ZmidN (Groups 3–35, dmid1=15,dmid2=245 ), corresponding to residual semantic components. In the middle spectral space, the baseline collapses into an entangled representation, whereas MuCa-ViT preserves clear channel-wise clustering.
Model
Full
Partial
Model
Full
Partial
IC-ViT mem
86.68
66.41
DiChaViT mem
64.83
54.69
+ FTL + CAM
87.83
68.50
+ FTL + CAM
68.09
57.37
Table 3: Plug-in compatibility of FTL and CAM on IC-ViT [ 13 ] and DiChaViT [ 19 ] . JUMP-CP is reported under full and partial channel settings. Models with superscript mem denote architectures augmented with memory tokens using the same initialization as MuCa-ViT.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
JUMP-CP
CHAMMI
So2Sat
Hardware and batch
GPUs (NVIDIA A6000)
2
1
1
Train batch size
32
80
128
Eval batch size
300
256
512
Num workers
1
3
3
Optimization
Appendix
Table 4: Training and evaluation configurations used to reproduce ChA-MAEViT [ 18 ] (denoted ChA-MAEViT ∗ in Table 1 ). MuCa-ViT is trained under identical configurations except for the memory token count and the additional FTL/CAM hyperparameters specified in Appendix B.3.3 , B.2.2 .
Figure 5: Effect of the number of memory tokens on JUMP-CP, So2Sat, and CHAMMI. Our model consistently achieves optimal performance with fewer memory tokens compared to the default settings of ChA-MAEViT.
Variant
Full
Partial
Equal-weight ablation
94.09
71.81
MuCa-ViT (asymmetric)
95.41
72.26
Appendix
Table 5: Ablation on asymmetric weighting in FTL. JUMP-CP accuracy.
Figure 6: Sensitivity analysis of the FTL decorrelation weight λFTL on JUMP-CP under full and partial channel settings. Performance peaks at a moderate regularization strength and remains stable across a broad range of λFTL .
Figure 7: t-SNE visualization of the learned CAM modulation parameters ( γ,β ) across different channel configurations. Each point is obtained by feeding the same underlying samples with a different channel subset. Distinct clusters emerge for different channel subsets, organized by configuration rather than sample identity, indicating that CAM generates configuration-aware modulation signals rather than static embeddings.
Conditioning Source
Full
Partial
External conditioning
92.57
70.47
Intrinsic conditioning (w/o FTL refinement)
93.18
71.12
FTL-refined intrinsic conditioning (MuCa-ViT)
95.41
72.26
Appendix
Table 6: Ablation on the conditioning source in CAM. JUMP-CP accuracy.
Figure 8: Effect of CAM insertion depth across different transformer blocks. Each x-axis label denotes the starting block from which CAM is applied to all subsequent blocks (e.g., Block 9 indicates CAM applied to blocks 9–11).
Variant
Full
Partial
ChA-MAEViT (Baseline)
90.35
68.10
+ FTL (Ours w/o CAM)
91.58
68.23
+ FTL + CAM (Top, first 2 blocks)
92.18
70.66
+ FTL + CAM (Bottom, last 2 blocks)
95.41
72.26
Appendix
Table 7: Ablation on CAM placement across network depth. JUMP-CP accuracy.
WTC [ 27 ]
HPA [ 24 ]
CP [ 28 ]
Model
Task1
Task2
Task1
Task2
Task3
Task1
Task2
Task3
Task4
HyperNetViT [ 10 ]
58.83
45.78
88.78
82.70
52.52
82.13
53.74
23.16
7.42
DepthwiseViT [ 7 ]
69.81
52.19
91.65
88.04
54.78
81.24
54.08
23.21
5.87
TemplateMixingViT [ 20 , 21 , 23 ]
61.66
51.52
85.01
79.91
49.69
77.45
48.83
22.56
5.60
ChAda-ViT [ 4 ]
77.58
67.18
87.49
75.94
45.41
83.92
45.58
21.94
6.28
ChannelViT [ 2 ]
78.36
67.58
83.93
76.73
47.97
77.70
55.16
21.89
6.38
Appendix
Table 8: F1 scores of multi-channel ViT models on the CHAMMI benchmark. ChA-MAEViT ∗ denotes our reproduction under the same training environment as MuCa-ViT; both are single-run results. Bold indicates the best result within each group, and underline indicates the best result overall.
Number of channels at inference
Method
8
7
6
5
4
3
2
1
DiChaViT [ 19 ]
69.19
61.91 ± 9.3
54.49 ± 12.4
46.35 ± 13.4
38.00 ± 12.4
30.09 ± 9.3
23.97 ± 4.9
20.90 ± 1.6
ChA-MAEViT [ 18 ]
90.73
83.36 ± 8.3
74.55 ± 11.7
63.46 ± 13.8
50.85 ± 13.9
38.13 ± 11.2
27.62 ± 6.4
21.59 ± 2.1
MuCa-ViT
95.41
89.46 ± 6.28
80.06 ± 11.32
69.75 ± 12.84
55.84 ± 13.80
41.18 ± 11.65
28.79 ± 6.84
21.54 ± 2.25
Appendix
Table 9: Performance comparison across different numbers and combinations of channels at inference. Mean ± std over all channel combinations at each budget.
Model
JUMP-CP (Full)
JUMP-CP (Partial)
CHAMMI
ChA-MAEViT*
92.22
70.24
76.20
MuCa-ViT (Ours)
94.28
74.31
76.72
Appendix
Table 10: Performance at ViT-Base scale.
Model
Inference Latency
FLOPs
ChA-MAEViT
1.084s
249.48
MuCa-ViT (Ours)
1.235s
249.33
Appendix
Table 11: Computational complexity and inference latency at ViT-Small (batch size 32).
Model
mem
JUMP-CP
CHAMMI
Full
Partial
ChA-MAEViT ∗
3
90.24
67.35
72.66
ChA-MAEViT ∗ (best)
4
90.35
68.10
74.04
MuCa-ViT (Ours)
3
95.41
72.26
76.46
Appendix
Table 12: Performance comparison under matched and best-performing memory-token budgets. MuCa-ViT outperforms ChA-MAEViT ∗ at the same token budget and exceeds its best-performing configuration with fewer memory tokens. ChA-MAEViT ∗ denotes our reproduced baseline. Results are from single runs.
Figure 9: Inter-channel diversity Div(g) of reconstructed spectral components across spectral groups g on JUMP-CP. MuCa-ViT (orange) maintains consistently higher diversity than the baseline (blue), especially in the middle spectral bands.
Method
Brightfield-only Acc. (%)
After reinjection Acc. (%)
Gain (pp)
MSE ↓
Fourier error ↓
ChA-MAEViT
22.77
34.12
+11.35
0.028
0.343
MuCa-ViT (Ours)
23.54
37.39
+13.85
0.027
0.321
Appendix
Table 13: Missing-channel reconstruction and reinjection on JUMP-CP. Five fluorescence channels are reconstructed from three observed brightfield channels and reinjected into the frozen classifier. Reinjection gain is measured relative to brightfield-only accuracy.
Figure 10: Per-image channel-channel cosine similarity averaged over 2,048 samples for each evaluation group. Darker blue indicates higher similarity. JUMP-CP and CHAMMI/WTC exhibit strong redundancy across channels, whereas So2Sat is largely uncorrelated except for a few pairs
Dataset / chunk
C
Redundancy
ChA-MAEViT*
Δ Full Acc.
So2Sat [ 32 ]
18
0.19
66.44
+1.53
CHAMMI (avg) [ 7 ]
3-4
0.46
73.95
+2.24
JUMP-CP [ 6 ]
8
0.77
90.52
+4.62
Appendix
Table 14: Channel redundancy and corresponding performance gain. Redundancy is the off-diagonal mean of the channel-channel cosine similarity matrix (Fig. 10 ). Δ Full Acc. is computed against our reproduced ChA-MAEViT* under the same training environment (cf. Table 1 ).
Figure 11: Comparison of Context Semantic Token (CST) behaviors between baseline ChA-MAEViT [ 18 ] and our method. (a) Gram-matrix-based redundancy analysis among CSTs. (b) Attention heatmaps showing spatial focus for each token.
Figure 12: Dot-product similarity between channel-wise aggregated patch representations (x-axis) and the Context Semantic Tokens, including the [CLS] token and memory tokens Mi(i=1,2,3) . In the baseline, both [CLS] and memory tokens exhibit strong alignment toward dominant channels, indicating severe semantic collapse and redundancy, whereas the proposed method yields well-disentangled similarity distributions.
HCLS%
HCST%
τ
ChA-MAEViT
MuCa-ViT
Δ
ChA-MAEViT
MuCa-ViT
Δ
JUMP-CP
0.01
10.23
42.31
+32.08
39.74
76.65
+36.91
0.02
28.79
63.96
+35.17
57.07
89.10
+32.03
0.04
60.66
88.74
+28.08
78.64
98.54
+19.90
0.06
81.93
96.61
+14.68
90.17
99.63
+9.46
Appendix
Table 15: Cosine-based channel coverage across softmax temperatures. Coverage is reported as a percentage of maximum entropy, and Δ denotes the improvement of MuCa-ViT over ChA-MAEViT in percentage points. MuCa-ViT maintains higher coverage at every tested temperature.
Condition
Setting
ChA-MAEViT
MuCa-ViT
Gap
Clean
Full
90.35
95.41
+5.06
Partial
68.10
72.26
+4.16
η=0.5
Full
83.24 ( −7.11 )
89.69 ( −5.72 )
+6.45
Partial
58.67 ( −9.43 )
67.78 ( −4.48 )
+9.11
η=0.7
Full
71.64 ( −18.71 )
80.14 ( −15.27 )
+8.50
Partial
48.68 ( −19.42 )
58.06 ( −14.20 )
+9.38
Appendix
Table 16: Prediction under final [CLS] corruption on JUMP-CP. Accuracy is reported in percent; the gap and parenthesized changes from clean accuracy are reported in percentage points.
Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations arising from varying staining protocols, sensor types, and acquisition settings. This heterogeneity limits the applicability of fixed-channel encoders commonly used in general computer vision. Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space. However, unrestricted token interactions across channels can lead to feature dilution, reducing the ability to preserve channel-specific semantics that are critical in MCI data. To address this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA), which decomposes token updates into two complementary pathways: spatial updates that model intra-channel structure, and channel-wise updates that adaptively integrate cross-channel information. This decoupling mitigates informational collapse while allowing selective inter-channel interaction. To further exploit these enhanced channel-specific representations, we introduce Decoupled Aggregation (DAG), which allows the model to learn task-specific channel importances. Extensive experiments across three MCI benchmarks demonstrate consistent improvements over existing MC-ViT approaches. Code is available at https://github.com/umarikkar/DCViT.
Umar Marikkar, Syed Sameed Husain, Muhammad Awais +1
Multi-Channel imaging (MCI) data differs fundamentally from natural images, as each channel records a semantically distinct signal rather than a colour band. To adapt vision encoders to MCI data, Multi-Channel Vision Transformers (MC-ViTs) tokenize each channel independently and concatenate the resulting tokens into one sequence, and the channel count is no longer fixed by the architecture. Self-attention is then computed across all channel-patch tokens with no restriction on which channels attend to which, which dilutes the features of individual channels. The Decoupled Vision Transformer (DC-ViT) regulates this by separating updates computed within a channel from updates computed across channels, and by forming a representation per channel before the channels are combined. Its formulation, however, pairs tokens by spatial position, and thus requires the same visible tokens in every channel. Correspondence under independent per-channel masking is recovered by solving a linear assignment between the retained patches of each channel, which allows decoupled attention to be combined with current masked multi-channel training in its standard configuration rather than a restricted one. Across three classification and three segmentation benchmarks spanning fluorescence microscopy, imaging mass cytometry and satellite imaging, including dense prediction at high channel counts, the resulting formulation outperforms the strongest MC-ViT baseline.
High-content multi-channel imaging, from Cell Painting assays to remote sensing, is central to modern scientific pipelines. Because these channels are highly heterogeneous and experimental configurations frequently evolve, a practical vision backbone must be fundamentally channel-agnostic. Current channel-adaptive vision transformers rely on global self-attention with learnable channel embeddings, which can be computed only for previously known channels. Attempts to overcome this limitation of new channels either discard the embeddings, losing channel identity, or process each channel independently, thereby eliminating critical cross-channel interactions. To address this, we introduce ChanSFormer, an efficient Vision Transformer that replaces rigid embeddings with disentangled spatial-channel attention and individual channel CLS tokens. This architecture natively preserves both distinct channel identities and cross-channel information flow. Furthermore, the individual channel class token design unlocks more informative feature representations for self-supervised learning and enables representation-based channel sampling while remaining channel-agnostic. We evaluate ChanSFormer on two biology multi-channel datasets, CHAMMI, JUMP-CP and a satellite dataset, So2Sat. Experimental results show that ChanSFormer outperforms state-of-the-art methods by 3.93%-4.27% on JUMP-CP in classification accuracy. It also demonstrates exceptional cross-dataset transferability under a self-supervised setting, outperforming previous methods by up to 2.72% in accuracy in the same classification tasks. Furthermore, the disentangled attention reduces the quadratic complexity of channel-by-spatial sequence length of the global attention baseline, thus improving throughput by 55%-313%. Code is available at https://github.com/cvlab-stonybrook/ChanSFormer.