High-content multi-channel imaging, from Cell Painting assays to remote sensing, is central to modern scientific pipelines. Because these channels are highly heterogeneous and experimental configurations frequently evolve, a practical vision backbone must be fundamentally channel-agnostic. Current channel-adaptive vision transformers rely on global self-attention with learnable channel embeddings, which can be computed only for previously known channels. Attempts to overcome this limitation of new channels either discard the embeddings, losing channel identity, or process each channel independently, thereby eliminating critical cross-channel interactions. To address this, we introduce ChanSFormer, an efficient Vision Transformer that replaces rigid embeddings with disentangled spatial-channel attention and individual channel CLS tokens. This architecture natively preserves both distinct channel identities and cross-channel information flow. Furthermore, the individual channel class token design unlocks more informative feature representations for self-supervised learning and enables representation-based channel sampling while remaining channel-agnostic. We evaluate ChanSFormer on two biology multi-channel datasets, CHAMMI, JUMP-CP and a satellite dataset, So2Sat. Experimental results show that ChanSFormer outperforms state-of-the-art methods by 3.93%-4.27% on JUMP-CP in classification accuracy. It also demonstrates exceptional cross-dataset transferability under a self-supervised setting, outperforming previous methods by up to 2.72% in accuracy in the same classification tasks. Furthermore, the disentangled attention reduces the quadratic complexity of channel-by-spatial sequence length of the global attention baseline, thus improving throughput by 55%-313%. Code is available at https://github.com/cvlab-stonybrook/ChanSFormer.
Figures & tables
Figure 1: (a) Overall framework of ChanSFormer . Each channel of an input image is independently projected by a shared patch embedding; a shared spatial positional embedding is added, and individual channel class (CLS) tokens, copied from a single learnable CLS token, are concatenated. The resulting tokens pass through L ChanSFormer encoders. (b) Supervised classification . The channel CLS tokens are aggregated via attention into an image-level representation. (c) Self-supervised DINOv2 pretraining . The channel CLS tokens are treated as distinct samples to generate the DINO loss [ 11 ] , while the remaining patch tokens generate the iBOT [ 17 ] loss. (d) Architecture of the ChanSFormer encoder . A single global attention block is replaced by channel attention (across channels, per spatial position) followed by spatial attention (across spatial positions, per channel).
CHAMMI
JUMP-CP
So2Sat
Method
#Params
Avg
Full
Flu.
Full
Sen-1
HyperViT [ 15 ]
24.7M
54.54
47.07
42.43
60.73
41.88
DepthwiseViT [ 12 ]
22.6M
60.94
49.86
44.98
60.41
43.41
TemplateViT [ 13 , 14 ]
21.5M
57.02
52.48
43.85
55.86
37.28
ChAda-ViT [ 7 ]
21.5M
63.88
65.03
42.15
56.98
12.38
ChannelViT [ 6 ]
21.5M
64.90
67.51
56.49
62.75
47.70
Table 1: Supervised classification on three multichannel benchmarks. For JUMP-CP and So2Sat, Full evaluates on all channels, Flu. evaluates on the fluorescence channels for JUMP-CP and Sen-1 evaluates on the Sentinel-1 channels for So2Sat. The highest number is bold, the second highest is underlined. #Params is the number of parameters. The reported numbers are the mean of 3 runs.
CHAMMI
JUMP-CP
So2Sat
Pretrain
Method
Avg
Full
Flu.
Full
Sen-1
CHAMMI
BoC [ 10 ]
48.92
39.36 †
36.42 †
67.50 †
46.46 †
ChannelViT [ 6 ]
44.29
23.94 †
24.71 †
66.65 †
44.54 †
ChanSFormer
50.24
40.82 †
39.14 †
68.08 †
46.23 †
JUMP-CP Flu.
BoC [ 10 ]
40.85 †
46.00 †
42.24
65.47 †
45.28 †
ChannelViT [ 6 ]
32.88 †
30.72 †
32.97
65.00 †
44.47 †
Table 2: Self-supervised linear-probe transfer across multichannel datasets. Models are pretrained with DINOv2 [ 11 ] on the dataset listed in the Pretrain column, then evaluated by linear probe on the target dataset. Flu. and Sen-1 represent the fluorescence and Sentinel-1 channels respectively. For ChannelViT, the channel embeddings for unseen channels are zero-padded. † marks evaluations involving channels absent from SSL pretraining.
Table 4
Number of channels for evaluation ( k )
Method
8
7
6
5
4
3
2
1
ChannelViT [ 6 ]
67.51
60.36±9.1
52.74±12.2
44.89±13.2
36.88±12.3
29.36±9.3
23.70±5.0
20.78±1.6
DiChaViT [ 8 ]
69.19
61.91±9.3
54.49±12.4
46.35±13.4
38.00±12.4
30.09±9.3
23.97±4.9
20.90±1.6
ChanSFormer
73.46
65.61±10.0
57.32±12.8
48.52±13.7
39.50±12.5
30.92±9.2
24.17±4.6
20.65±1.3
Table 5: Test accuracy under partial channels of JUMP-CP. Each column reports the mean ± std over all (k8) combinations of k available channels out of 8 [ 6 , 8 ] . ChanSFormer achieves comparable or better performance in all 8 cases.
CHAMMI
JUMP-CP
Pretrain
Method
Avg
Full
Flu.
CHAMMI: Allen + CP (HPA held out)
BoC [ 10 ]
35.72
19.63
19.65
ChannelViT [ 6 ]
33.92
19.54
19.53
ChanSFormer
46.00
38.38
37.25
CHAMMI: HPA + CP (Allen held out)
BoC [ 10 ]
46.76
37.75
35.67
ChannelViT [ 6 ]
40.93
19.62
19.62
Table 6: Leave-one-source-out self-supervised transfer. Models are pretrained with DINOv2 [ 11 ] on CHAMMI with one source held out, then evaluated by linear probe (JUMP-CP) or the official CHAMMI protocol. All evaluations in this table involve channels absent from SSL pretraining. Bold denotes the best result under each pretraining source.
CHAMMI
JUMP-CP
So2Sat
Method
Avg
Full
Flu.
Full
Sen-1
ChanSFormer
68.83
73.46
61.91
64.76
49.75
w. channel embedding
69.03
73.78
62.36
64.78
49.55
ChannelViT [ 6 ]
64.90
67.51
56.49
62.75
47.70
w/o channel embedding (CA-MAE [ 9 ] )
50.37
61.93
51.56
62.58
46.73
Spatial-only (BoC [ 10 ] + CLS channel attn)
61.73
41.56
39.10
63.41
47.70
Table 7: Ablation on channel embeddings and cross-channel interactions. Channel embeddings are not critical for ChanSFormer compared with ChannelViT with global attention, and cross - channel interactions prove essential. Flu. and Sen-1 represents the fluorescence and Sentinel-1 channels respectively.
Setting
Method
CHAMMI
JUMP-CP Full
JUMP-CP Flu.
Supervised
ChanSFormer
68.83
73.46
61.91
w. HCS [ 6 ] channel sampling
66.17
72.81
61.78
w. HCS [ 6 ] w/o channel CLS token [ 22 ]
64.58
69.77
58.00
DINOv2
ChanSFormer (channel-level)
50.24
40.82
39.14
(PT: CHAMMI)
w/ attention pooling (image-level)
48.61
38.97
38.52
DINOv2
ChanSFormer (channel-level)
42.86
46.13
45.15
Table 8: Ablation of ChanSFormer’s design choices. The Setting column distinguishes between supervised classification and DINOv2 linear probes. For the DINOv2 ablation, models are pretrained on either CHAMMI or JUMP-CP (indicated as PT) to compare feature granularity. Flu. represents the fluorescence channels.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
CHAMMI
JUMP-CP Full
JUMP-CP Flu.
So2Sat Full
So2Sat Sen-1
ChanSFormer
68.83 ± 0.51
73.46 ± 0.14
61.91 ± 0.62
64.76 ± 0.30
49.75 ± 0.33
Appendix
Table 9: Standard deviation of ChanSFormer on 3 datasets.
Method
#Params
Allen
HPA
CP
Avg
HyperViT [ 15 ]
24.7M
45.17
63.90
26.23
54.54
DepthwiseViT [ 12 ]
22.6M
50.35
71.52
27.74
60.94
TemplateViT [ 13 ]
21.5M
49.51
64.52
25.65
57.02
ChAda-ViT [ 7 ]
21.5M
67.08
60.67
24.60
63.88
ChannelViT [ 6 ]
21.5M
67.66
62.14
27.62
64.90
DiChaViT [ 8 ]
21.5M
75.69
63.67
28.98
69.68
Appendix
Table 10: Detailed breakdown of performance on CHAMMI subsets (Allen, HPA, and CP). The Avg column represents the mean of Allen and HPA. Baseline results are reported from DiChaViT [ 8 ] .
Pretrain
Method
Allen
HPA
CP
Avg
CHAMMI
BoC [ 10 ]
49.88
47.96
27.91
48.92
ChannelViT [ 6 ]
46.60
41.98
26.93
44.29
ChanSFormer
48.96
51.51
25.10
50.24
JUMP-CP Flu.
BoC [ 10 ]
43.31
38.40
27.83
40.85
ChannelViT [ 6 ]
32.48
33.29
28.77
32.88
ChanSFormer
44.48
41.25
27.99
42.86
Appendix
Table 11: Detailed breakdown of performance on CHAMMI subsets (Allen, HPA, and CP) for DINOv2 linear-probe. The Avg column represents the mean of Allen and HPA.
Method
#Params
CHAMMI
JUMP-CP
So2Sat
HyperViT [ 15 ]
24.7M
62.28
39.64
61.22
DepthwiseViT [ 12 ]
22.6M
76.52
46.97
61.55
TemplateViT [ 13 , 14 ]
21.5M
70.30
35.01
60.84
ChannelViT [ 6 ]
21.5M
80.68
80.81
68.01
DiChaViT [ 8 ]
21.5M
78.83
82.99
67.18
ChanSFormer
28.9M
82.18
84.11
68.77
Appendix
Table 12: Supervised classification with ImageNet ViT-S/16 pretraining. Top-1 accuracy on the full channel set of each dataset. The highest number per column is bold and the second highest is underlined.
T
So2Sat Full
Sen-1
0.01
64.85
48.35
0.1
64.76
49.75
1.0
64.99
49.46
Appendix
Table 13: Sensitivity to the sampling temperature T in supervised training. The selected value for each dataset is shown in bold in the T column; bold numbers denote the best result in each column.
CHAMMI
JUMP-CP
Method
# Channels per view
Avg
Full
Flu.
ChannelViT [ 6 ]
2
32.88
30.72
32.97
4
34.09
31.97
35.53
ChanSFormer
2
42.86
46.13
45.15
4
45.04
48.18
46.49
Appendix
Table 14: Number of channels per view in DINOv2 pretraining on JUMP-CP Flu., evaluated as in Table 2 . The image batch size is halved for four channels to keep the number of channel views per iteration fixed.
SSL source
Target
ChannelViT (zero-pad)
ChannelViT (CE adapt.)
ChanSFormer
CHAMMI
JUMP-CP Full
23.94
25.54
40.82
JUMP-CP Flu.
24.71
24.99
39.14
So2Sat Full
66.65
66.91
68.08
So2Sat Sen-1
44.54
45.14
46.23
JUMP-CP Flu.
JUMP-CP Full
30.72
31.10
46.13
So2Sat Full
65.00
65.19
66.33
Appendix
Table 15: Zero-padded vs. adapted channel embeddings for ChannelViT [ 6 ] on unseen-channel transfer. Models are pretrained with DINOv2 on the SSL source and evaluated by linear probe on the Target .
Figure 2: Learned channel relations on JUMP-CP. Left: the EMA channel-similarity matrix S used by our sampler. Right: channel-attention weights (row: query, column: key) at patch positions, averaged over all heads, all 12 layers, and the full test set with all 8 channels present; each row sums to 1 (uniform =0.125 ). White lines separate the fluorescence (Mito, AGP, RNA, ER, DNA) and brightfield (HighZBF, LowZBF, Brightfield) channels.
Figure 3: Learned channel relations on So2Sat. Left: the EMA channel-similarity matrix S used by our sampler. Right: channel-attention weights (row: query, column: key) at patch positions, averaged over all heads, all 12 layers, and the full test set with all 18 channels present; each row sums to 1. White lines separate the 8 Sentinel-1 channels from the 10 Sentinel-2 bands. Sentinel-1: real (re) and imaginary (im) parts of the unfiltered VH and VV channels, intensity (int) of the refined-Lee-filtered VH and VV channels, and real and imaginary parts of the refined-Lee-filtered covariance-matrix off-diagonal element (Cov).
Multi-channel images encode heterogeneous channel semantics aligned over a shared spatial structure, which requires models to jointly capture globally shared structure and channel-specific variations. Recent multi-channel Vision Transformers attempt to address this challenge by augmenting the [CLS] token with memory tokens to organize multi-channel representations under varying channel compositions. However, we find that these context tokens collapse onto a few dominant channels under high inter-channel redundancy, producing redundant representations and a biased global summary that fails to reliably support composition-aware channel interpretation. We propose MuCa-ViT (Multi-Channel Composition-Aware Vision Transformer), a unified framework that addresses these limitations through two coupled mechanisms. Factorial Token Learning (FTL) enforces token-level decorrelation among context tokens, encouraging the [CLS] token to capture globally shared semantics while memory tokens preserve complementary channel-specific cues. Channel Composition-Aware Modulation (CAM) uses the FTL-refined [CLS] as a semantic anchor to generate sample-wise modulation signals, which enable composition-aware interpretation of channels within a unified backbone rather than relying on static channel embeddings. Experiments on JUMP-CP, CHAMMI, and So2Sat show that MuCa-ViT outperforms prior multi-channel ViTs across all three benchmarks, and the magnitude of improvement aligns with the inter-channel redundancy of each dataset (ρ∈[0.19,0.77]), validating the diagnostic analysis that motivates our design. Our code is available at https://github.com/kimdanni/MuCa-ViT .
Daeun Kim, Hyesong Choi, Hyejin Park +1
Division of Artificial Intelligence & Software, Ewha Womans University · Soongsil University
Multi-Channel imaging (MCI) data differs fundamentally from natural images, as each channel records a semantically distinct signal rather than a colour band. To adapt vision encoders to MCI data, Multi-Channel Vision Transformers (MC-ViTs) tokenize each channel independently and concatenate the resulting tokens into one sequence, and the channel count is no longer fixed by the architecture. Self-attention is then computed across all channel-patch tokens with no restriction on which channels attend to which, which dilutes the features of individual channels. The Decoupled Vision Transformer (DC-ViT) regulates this by separating updates computed within a channel from updates computed across channels, and by forming a representation per channel before the channels are combined. Its formulation, however, pairs tokens by spatial position, and thus requires the same visible tokens in every channel. Correspondence under independent per-channel masking is recovered by solving a linear assignment between the retained patches of each channel, which allows decoupled attention to be combined with current masked multi-channel training in its standard configuration rather than a restricted one. Across three classification and three segmentation benchmarks spanning fluorescence microscopy, imaging mass cytometry and satellite imaging, including dense prediction at high channel counts, the resulting formulation outperforms the strongest MC-ViT baseline.
Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations arising from varying staining protocols, sensor types, and acquisition settings. This heterogeneity limits the applicability of fixed-channel encoders commonly used in general computer vision. Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space. However, unrestricted token interactions across channels can lead to feature dilution, reducing the ability to preserve channel-specific semantics that are critical in MCI data. To address this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA), which decomposes token updates into two complementary pathways: spatial updates that model intra-channel structure, and channel-wise updates that adaptively integrate cross-channel information. This decoupling mitigates informational collapse while allowing selective inter-channel interaction. To further exploit these enhanced channel-specific representations, we introduce Decoupled Aggregation (DAG), which allows the model to learn task-specific channel importances. Extensive experiments across three MCI benchmarks demonstrate consistent improvements over existing MC-ViT approaches. Code is available at https://github.com/umarikkar/DCViT.
Umar Marikkar, Syed Sameed Husain, Muhammad Awais +1