The interpretation of chest X-rays poses significant challenges in achieving accurate multi-label pathology classification and spatial localization. These tasks demand different annotation granularities but are constrained by the scarcity of region-level (dense) annotations. We introduce CLARiTy (Class Localizing and Attention Refining Image Transformer), a vision transformer-based model for joint multi-label classification and weakly-supervised localization of thoracic pathologies. CLARiTy employs multiple class-specific tokens to generate discriminative attention maps and a SegmentCAM module for foreground segmentation and background suppression using explicit anatomical priors. Trained on image-level labels from the NIH ChestXray14 dataset, it leverages distillation from a ConvNeXtV2 teacher for efficiency. Localization heatmaps are converted to bounding boxes with OneBox, a hysteresis-gated extractor whose parameters are calibrated per finding on held-out validation data. Evaluated on the official NIH split, CLARiTy achieves competitive classification across 14 pathologies and strong weakly-supervised localization on 8 pathologies, outperforming prior methods by up to 50.7% relative Macro Intersection over Union (IoU) Accuracy at stringent thresholds. Gains are most pronounced for small pathologies: at T(IoU) = 0.5, the 224 variant reaches an IoU Accuracy of 0.154 on nodules, where the compared methods remain at or below 0.013. The lower-resolution CLARiTy-S-16-224 offers high efficiency while surpassing baselines, and could potentially be applied in low-resource settings. An ablation study confirms the contributions of SegmentCAM, DINO pretraining, and attention pooling. CLARiTy advances beyond CNN-ViT hybrids by harnessing ViT self-attention for global context and class-specific localization, refined through convolutional background suppression for precise, noise-reduced heatmaps.
Figures & tables
Figure 1: Illustration of the proposed CLARiTy model. An input chest X-ray image is split into N2 patches and embedded into P patch tokens, which are concatenated with C class tokens. Learned positional embeddings are added to produce C+P input tokens to the transformer. A series of d transformer blocks extracts relevant information for classification and weakly-supervised localization. At the output, the C class tokens are passed through an attention pooling module to produce class token logits. The output P patch tokens are passed through the SegmentCAM module, where foreground logits and a segmentation map are produced. The self-attention maps from the final p transformer blocks are fused together to produce class-specific attention maps.
Figure 2: Transformer self-attention matrix with multiple class tokens. Each class token is denoted c1,c2,…,cC , and each patch token is denoted n1,n2,…,nP . The class-to-patch attention in the upper-right sub-matrix is extracted for the class-specific attention maps.
Figure 3: Weakly-supervised localization method of CLARiTy. During inference, a foreground mask and attention map are produced for each class. Element-wise multiplication yields a class-specific localization heatmap that is both highly precise and confined to the class’s ground-truth region. In this chest X-ray, a mass (round opacity) is found in the upper-middle left lung of the patient. The attention map has high intensity directly over the mass, and the heatmap intensity is confined to the lung lobes.
Figure 4: Illustration of the CLARiTy losses. Following the attention pooling layer, the class token logits are used to calculate the class token classification loss LCLSC . The outputs of the SegmentCAM module are used to calculate the combined CAM loss LCAM .
Figure 5: Illustration of the attention pooling module. Row-wise softmax activation is applied to a set of learned weights, which are invariant to the input image. This matrix is element-wise multiplied with the output class tokens after reshaping to a C×D matrix. Thereafter, a row-wise sum produces class-specific logits. Each dimension in D is able to attend to particular class features.
Figure 6: Illustration of the progressive fusion block used in the heatmap branch of SegmentCAM. The N×N×D patch tokens are progressively refined and upsampled by three stages of bilinear ×2 interpolation, each followed by a 3×3 convolution, SiLU activation, and batch normalization. The channel width tapers from max(4C,64) through max(3C,48) to C , so all learned computation runs at or below half the image resolution. A 1×1 convolution projects the raw patch tokens into C class maps, which are bilinearly enlarged by a factor of 8 and added to the main path, restoring the global context that the narrowed stages would otherwise discard. A final bilinear ×2 and sigmoid activation produce the image-resolution heatmap H∈(0,1)M×M×C .
Figure 7: Fused inverse bottleneck block, as used in Xiong et al. (2021) . In SegmentCAM, the block is used in the feature head and in the classification head of the background activation suppression stage. An input tensor of shape H×W×K1 is passed through a 3×3 convolution with an expansion ratio of r∈R>1 . This increases the number of channels to rK1 . Batch normalization and SiLU activation are then applied. Finally, a 1×1 convolution with batch normalization reduces the number of channels to K2 . If K1=K2 , then a residual connection is added to the output.
Figure 8: Illustration of the SegmentCAM module’s feature and heatmap extraction. The output patch tokens are first reshaped to an N×N×D tensor. Then, in the heatmap branch, the progressive fusion block ( Fig. 6 ) with sigmoid activation produces a heatmap tensor H of shape M×M×C , where M is the input image resolution. A threshold t is applied to H to produce foreground and background segmentation maps Sf and Sb respectively. The heatmap H is used to calculate the soft Dice segmentation loss LDice . In the feature branch, a convolutional head produces feature tensor F of shape N×N×C , which is used in the background activation suppression methodology.
Figure 9: Illustration of the SegmentCAM module’s background activation suppression methodology. The segmentation maps Sf and Sb are derived from the heatmap at the feature resolution. The feature tensor F is passed through the convolutional classification head twice: with unmasked features, and with foreground features suppressed (after fusing with Sb ). In the top branch, the output of the unmasked features has its background features suppressed with Sf , and global average pooling (GAP) is applied to produce foreground logits ℓf . In the middle branch, the output unmasked features are used to produce patch token logits ℓP using GAP. In the bottom branch, the output of the foreground-suppressed features produces background logits ℓb using GAP. Foreground and patch token losses LCLSf and LCLSP are calculated using ℓf and ℓP respectively. Finally, the background activation suppression loss LBAS is calculated using ℓP and ℓb . Using these losses, the classification head learns to classify F while ignoring background features.
Pathology
Anatomical Regions
Atelectasis
Left Lung, Right Lung
Cardiomegaly
Heart
Consolidation
Left Lung, Right Lung
Edema
Left Lung, Right Lung
Effusion
Left Lung, Right Lung
Emphysema
Left Lung, Right Lung
Table 1: Anatomical regions of pathologies in the NIH dataset, as identified by a diagnostic radiologist.
Loss Weight
Value
αC
2.2674
αP
2.4678
αf
3.9670
β
0.4377
γ
0.1427
Table 2: Final optimal scalar loss weights selected to maximize validation classification and weakly-supervised localization. These values are used for all experiments.
Model
Image Size
NIH Training Set Size
Macro AUC
Macro IoU Accuracy at T(IoU)
0.1
0.2
0.3
0.4
0.5
RGT 1 ‡
224
70%
0.839 †
0.591
0.424
0.281
0.173
0.090
ThoraX-PriorNet-224 2 ‡
224
70%
0.844
0.666
0.509
0.398
0.296
0.201
ThoraX-PriorNet-512 2 ‡
512
70%
0.847
0.803
0.626
0.488
0.334
0.217
Wang et al. (2024) ‡
224
–
–
0.450
–
0.230
–
–
Wang et al. (2017)
1024
70%
0.745
0.568
0.373
0.221
0.116
0.062
Table 3: Classification and weakly-supervised localization performance of different models evaluated on the NIH test set. Classification results are macro averaged across all 14 pathologies, and the localization results are macro averaged across the 8 pathologies with bounding box labels. Localization results for the CLARiTy models are obtained with the calibrated OneBox extractor. The highest metric values are shown in bold. The results of cited models are taken from their publications.
Model
Pathology AUC
Atel.
Card.
Cons.
Edem.
Effu.
Emph.
Fibr.
Hern.
Infi.
Mass
Nodu.
Pleu.
Pn. 1
Pn. 2
Mean
Yao et al. (2017) ‡
0.772
0.904
0.788
0.882
0.859
0.829
0.767
0.914
0.695
0.792
0.717
0.765
0.713
0.841
0.798
Li et al. (2018) ‡
0.800
0.870
0.800
0.880
0.870
0.910
0.780
0.770
0.700
0.830
0.750
0.790
0.670
0.870
0.806
Ho and Gwak (2019) ‡
0.795
0.887
0.786
0.892
0.875
0.875
0.756
0.836
0.703
0.835
0.716
0.774
0.742
0.863
0.810
Ho and Gwak (2019) ¶
0.726
0.865
0.717
0.812
0.810
0.740
0.748
0.803
0.682
0.773
0.686
0.730
0.660
0.796
0.753
RGT 1 ‡
0.800 †
0.920 †
–
–
0.780 †
–
–
–
0.860 †
0.880 †
0.880 †
–
0.790 †
0.810 †
0.839 †
Table 4: Classification performance comparison across different models evaluated on the NIH test set. The highest values are shown in bold for models that classify all 14 pathologies. The results of cited models are taken from their publications.
T(IoU)
Model
Pathology IoU Accuracy
Atel.
Card.
Effu.
Infi.
Mass
Nodu.
Pn. 1
Pn. 2
Mean
0.1
RGT 1 ‡
0.610
0.950
0.650
0.820
0.500
0.130
0.790
0.280
0.591
ThoraX-PriorNet-224 2 ‡
0.628
1.000
0.791
0.862
0.518
0.127
0.825
0.577
0.666
ThoraX-PriorNet-512 2 ‡
0.761
1.000
0.837
0.870
0.741
0.582
0.867
0.763
0.803
Wang et al. (2024) ‡
0.250
0.990
0.490
0.630
0.270
0.030
0.680
0.300
0.450
[1pt/1.5pt]
Wang et al. (2017)
0.689
0.938
0.660
0.707
0.400
0.139
0.633
0.378
0.568
Table 5: Weakly-supervised localization performance comparison across different models on the bounding-box-annotated images of the NIH test set. For the CLARiTy models, localization is computed on all images of the localization test split, irrespective of whether the pathology was correctly classified. Localization results for the CLARiTy models are obtained with the calibrated OneBox extractor. The highest values for each T(IoU) are shown in bold. The results of cited models are taken from their publications.
SegmentCAM
Dice
Pretrained Weights
Classification Loss
Class Token Pooling
Validation Metric
ImageNet
DINO
MLSM
WBCE
GAP
Attention
Macro AUC
MSLI
UDP
✓
✓
–
✓
–
✓
–
✓
0.836
0.391
0.614
–
–
–
✓
–
✓
–
✓
0.841
0.339
0.590
✓
–
–
✓
–
✓
–
✓
0.833
0.302
0.567
✓
✓
✓
–
–
✓
–
✓
0.826
0.363
0.595
✓
✓
–
✓
✓
–
–
✓
0.841
0.382
0.611
Table 6: Component ablations on the CLARiTy-S-16-224. Classification and weakly-supervised localization performance was evaluated on the validation set, with localization evaluated using the calibrated OneBox extractor. The highest values are shown in bold.
Figure 10: Selected localization outputs from the CLARiTy-S-16-224. For each pathology, two chest X-rays at 224×224 are shown. All chest X-rays belong to the localization-test set. The model correctly predicted the presence of the pathology in each image. Class-specific outputs: foreground mask, attention map, localization heatmap, and bounding boxes. The predicted bounding boxes are obtained with the calibrated OneBox extractor. Each output is overlaid on the input chest X-ray. Ground-truth bounding boxes: green. Predicted bounding boxes: red.
Image Size
Component
Parameters
FLOPS
224
Vision Transformer Backbone
21.68 M
4.95 G
Attention Pooling Module
5.38 K
5.38 K
SegmentCAM Module
2.94 M
0.87 G
CLARiTy-S-16-224
24.62 M
5.83 G
512
Vision Transformer Backbone
21.99 M
32.41 G
Attention Pooling Module
5.38 K
5.38 K
Table 7: Computing resources of the CLARiTy-S-16-224, CLARiTy-S-16-512, and their components.
Figure 11: Validation UDP as a function of the number of fused attention layers p for the CLARiTy-S-16-224 and CLARiTy-S-16-512 variants. Each point is the highest validation UDP achieved during training for that layer count, and the star marks the maximum of each curve. The maximum validation UDP is reached at p=5 for CLARiTy-S-16-224 and at p=3 for CLARiTy-S-16-512.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
GT box area
Instances
224
512
<0.5%
36
0.056
0.167
0.5 – 1%
38
0.053
0.105
1 – 2%
75
0.173
0.173
2 – 5%
108
0.157
0.111
5 – 10%
87
0.287
0.195
≥10%
149
0.638
0.644
Appendix
Table 1: Detection accuracy at T(IoU)=0.5 by ground-truth box area (fraction of the image), pooled over the 493 localization-test instances.
Finding
Instances
Accuracy
Missed (small/mirr.)
Extra boxes
Atelectasis
91
0.121
80 (12/2)
1.58
Cardiomegaly
73
1.000
0 (0/0)
0.00
Effusion
77
0.078
71 (7/4)
1.77
Infiltration
62
0.274
45 (2/2)
1.35
Mass
43
0.326
29 (10/0)
1.09
Nodule
39
0.154
33 (31/0)
1.79
Appendix
Table 2: Localization failures at T(IoU)=0.5 on all 493 localization-test instances, per variant. “Missed” counts instances without a matching prediction and is followed in parentheses by the number of misses with a small ground-truth box (less than 1% of the image area) and with a mirrored prediction (matching the left–right mirrored ground-truth box). “Extra boxes” is the mean number of predicted boxes per instance that do not match the instance’s ground-truth box.
Group
Images
224
512
All
25,595
0.808
0.825
Male
14,881
0.809
0.825
Female
10,714
0.806
0.824
PA view
11,096
0.803
0.822
AP view
14,499
0.786
0.803
Age <40
8,411
0.749
0.770
Appendix
Table 1: Classification Macro AUC by subgroup on the official NIH test set.
Finding
σ
tlo
thi
s
α
Atelectasis
0.00
0.75
0.80
0.25
1.10
Cardiomegaly
4.00
0.05
0.40
1.25
1.00
Effusion
0.00
0.70
0.75
1.00
1.10
Infiltration
2.00
0.65
0.80
0.50
1.10
Mass
1.00
0.70
0.75
0.50
1.10
Nodule
2.00
0.75
0.80
0.75
0.90
Appendix
Table 1: Calibrated OneBox parameters for the CLARiTy-S-16-224 and CLARiTy-S-16-512 variants. σ is given in pixels at the model’s input resolution; the underlying search grid is relative to the image side length. The minimum gate area is four pixels for both variants.
Table 1: Extractor comparison on the localization test split (438 images, 493 bounding-box instances): macro IoU accuracy over the eight bounding-box findings, mean predicted boxes per instance, and unmatched boxes per instance at T(IoU)=0.5 (a false-positive rate). The localization baseline of Wang et al. (2017) (per-image thresholding at 60/255 and 180/255 with connected components) is evaluated on the same maps; top- k keeps the k highest-peak boxes per instance. The matched-budget comparison is between the OneBox and Wang top- k rows.
Metric
Δ
95% CI
p
Macro AUC
+0.0169
[0.015, 0.019]
< 0.0005
IoU Acc. @0.1
+0.0422
[0.009, 0.077]
0.0130
IoU Acc. @0.2
+0.0294
[-0.007, 0.066]
0.1170
IoU Acc. @0.3
+0.0273
[-0.011, 0.067]
0.1720
IoU Acc. @0.4
+0.0219
[-0.018, 0.061]
0.2700
IoU Acc. @0.5
-0.0034
[-0.037, 0.029]
0.8430
Appendix
Table F.1: Paired (common-resample) bootstrap differences for CLARiTy-S-16-512 minus CLARiTy-S-16-224, for the Macro AUC and the Macro IoU Accuracy at each T(IoU) . Δ is the mean difference across 2,000 paired resamples; the 95% confidence interval is the percentile interval of the per-draw differences and p is the two-sided bootstrap sign test. Classification uses the official NIH test split; localization uses the localization test split (438 images, 493 instances).
T(IoU)
CLARiTy-S-16-224
CLARiTy-S-16-512
0.1
0.780 [0.745, 0.814]
0.822 [0.789, 0.854]
0.2
0.667 [0.631, 0.705]
0.698 [0.662, 0.733]
0.3
0.529 [0.491, 0.568]
0.552 [0.516, 0.596]
0.4
0.401 [0.365, 0.436]
0.419 [0.384, 0.459]
0.5
0.303 [0.271, 0.333]
0.295 [0.263, 0.331]
Appendix
Table F.2: 95% stratified percentile bootstrap confidence intervals for the Macro IoU Accuracy over the 8 bounding-box pathologies at each T(IoU) on the localization test split (438 images, 493 instances; 2,000 image-level resamples). Each cell gives the point estimate followed by the 95% interval.
Pathology
CLARiTy-S-16-224
CLARiTy-S-16-512
ConvNeXtV2-B-224
ConvNeXtV2-B-512
Atelectasis
0.765 [0.757, 0.773]
0.791 [0.783, 0.798]
0.772 [0.764, 0.780]
0.787 [0.779, 0.794]
Cardiomegaly
0.902 [0.894, 0.909]
0.902 [0.894, 0.909]
0.892 [0.884, 0.900]
0.898 [0.890, 0.905]
Consolidation
0.760 [0.751, 0.770]
0.764 [0.754, 0.773]
0.750 [0.740, 0.759]
0.758 [0.749, 0.767]
Edema
0.852 [0.843, 0.861]
0.856 [0.847, 0.864]
0.853 [0.844, 0.861]
0.855 [0.846, 0.863]
Effusion
0.833 [0.827, 0.839]
0.837 [0.831, 0.843]
0.828 [0.822, 0.834]
0.832 [0.826, 0.837]
Emphysema
0.895 [0.885, 0.904]
0.936 [0.929, 0.942]
0.900 [0.891, 0.909]
0.938 [0.931, 0.944]
Appendix
Table 3: 95% stratified percentile bootstrap confidence intervals for the per-pathology and macro AUC of CLARiTy-S-16-224, CLARiTy-S-16-512 and their ConvNeXtV2-B teachers on the official NIH test split (25,595 images; 2,000 image-level resamples). Each cell gives the point estimate followed by the 95% interval.
Chest X-ray classification uses machine learning to detect and categorize diseases and findings in radiographs for screening, triage, and clinical decision support.
Division of Artificial Intelligence & Software, Ewha Womans University · Ewha Medical Artificial Intelligence Research Institute, Ewha Womans University · Department of Nuclear Medicine, Ewha Womans University +2