Organizations: College of Information Science and Engineering, Northeastern University, Shenyang, China · College of Sciences, Northeastern University, Shenyang, China · School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, Qinhuangdao 066004, Hebei, China
Automated tongue-sign recognition can improve the consistency of non-invasive examinations that otherwise rely on subjective visual judgment. A single tongue image often exhibits multiple coexisting signs characterized by heterogeneous evidence, including localized morphology, global chromaticity, and inter-label dependence. However, existing methods typically rely on shared representations and uniform multi-label heads, making it difficult to verify whether predictions stem from clinically relevant evidence rather than spurious visual correlations. We propose SEAT, an evidence-audited framework that matches distinct evidence types with dedicated prediction and evaluation mechanisms. Built on frozen DINOv2 patch features, SEAT couples mask-guided sign-specific attention for localized morphology, a semi-relaxed optimal transport pathway over LAB/HSV color prototypes to enhance pale-tongue recognition, and a co-occurrence-aware transport regularizer to shape the batch-level predicted label marginal. These components are evaluated through audits matched to their evidence types. On the TongueDx benchmark, SEAT achieves a mean macro-F1 of 72.59 with a standard deviation of 1.04 across six random seeds, with a six-model ensemble reaching 74.15. The ensemble outperforms a backbone-matched baseline and a larger, domain-specialized vision-language model by 8.06 and 2.99 percentage points, respectively. Our results demonstrate that aligning dedicated prediction pathways and audits with sign-specific characteristics improves multi-label accuracy while offering more targeted evidential grounding than uniform post-hoc spatial explanations. Code is available at https://github.com/NEU-TCM-Lab/Sign-type-Evidence-Audited-Transport.
Figures & tables
Fig. 1: Overview of SEAT. A frozen DINOv2-Large backbone encodes each image into a 37×37 patch grid. A mask-guided multi-head softmax attention reader forms the discriminative core. C1 adds a color-prototype optimal-transport pathway over per-patch LAB/HSV descriptors, whose pale-tongue output is combined with the reader by a validation-selected late-fusion weight. C2 adds a co-occurrence regularizer that shapes the batch-level predicted label marginal under a 1−Jaccard cost geometry.
Method
Pale
TipSide
Spot
Ecchy.
Crack
Tooth.
FurTh.
FurYe.
Macro
DINOv2-L (global)
35.04
61.56
73.77
31.29
91.21
81.04
98.58
56.19
66.09
ResNet-34
39.85
64.12
80.80
16.15
92.24
82.82
98.64
74.67
68.66
ResNet-50 [ 1 ]
32.68
66.19
76.44
24.72
91.17
84.31
98.65
76.27
68.80
Qwen3-VL-4B LoRA [ 21 ]
36.73
59.59
77.10
27.72
91.84
76.10
98.58
78.48
68.27
ShizhenGPT-7B-VL [ 22 ]
38.83
64.97
80.59
36.51
91.26
81.34
98.58
77.17
71.16
Dr. Tongue [ 1 ] (reported; not directly comparable)
46.00
68.22
81.09
38.02
91.90
82.82
98.54
76.41
72.87
TABLE I: Per-class test F1 on TongueDx (%). DINOv2-L (global), ResNet-34, Qwen3-VL-4B LoRA, ShizhenGPT-7B-VL, and SEAT use the fixed image-level split with per-class thresholds selected on validation; both vision-language baselines are LoRA-tuned. SEAT is reported both as a single model (mean ± std over seeds 42 – 47 ; per-class columns show the mean, only the Macro column carries ± std) and as the deployed six-seed ensemble. ResNet-50 and Dr. Tongue values are reproduced from Table 3 of [ 1 ] as reported contextual references and are not directly comparable to our protocol. Bold marks the highest value across all rows.
Configuration
Macro-F1
Bootstrap 95% CI
Reader
71.47±0.93
[70.43,72.52]
Reader + C1
71.97±0.89
[70.93,73.04]
Reader + C2
71.91±1.24
[70.86,72.95]
Reader + C1 + C2
72.59±1.04
[71.53,73.67]
Paired gain
Gain
Bootstrap 95% CI
C1 − Reader
+0.50
[+0.21,+0.78]
TABLE II: Component 2×2 factorial (seeds 42–47), all selection on validation. Top: validation-thresholded test macro-F1 as mean ± population std across seeds, with a test-image bootstrap 95% CI. Bottom: paired gains with their bootstrap CIs. C1 improves every seed both alone and with C2; C2 is secondary and complementary (see text).
Fig. 2: Multi-seed component effects across seeds 42–47. (a) Per-sign F1 for the Reader and Reader+C2, reported as mean ± standard deviation across seeds; labels show the mean paired change. (b) TonguePale F1 before and after C1 fusion on top of Reader+C2; thin lines connect matched seeds. All thresholds and fusion weights are selected on validation. Aggregate macro-F1 results and bootstrap confidence intervals are reported in Table II .
Fig. 3: Evidence-type coverage, read as a capability matrix. Left to right: (A) a ResNet-34 Grad-CAM, deletion-sensitive within the CNN at 1.851 on Toothmark; (B) our attention-based spatial evidence, load-bearing under readout-mask suppression ( 2.386 , across Toothmark, Crack, and Spot); (C) a descriptor-space color intervention on the color pathway for the pale-tongue sign, whose directed effect exceeds every one of seven matched non-clinical color directions and which a spatial map does not express; and (D) the sign co-occurrence geometry ( 1−Jaccard ) used by C2, an output-space structure rather than an image map.
Configuration
Macro-F1
Attention reader
73.42
SignOT: global + OT (joint / frozen)
72.95 / 72.76
SignOT: OT only (no global)
70.81
AOT: attention-demand sparse (balanced)
72.54
AOT: unbalanced ρ=0.1 / partial m=0.7
72.18 / 72.49
OTGen: K=4 mean / K=4 concat
72.45 / 72.58
TABLE III: OT-as-readout ablation at seed 42 (validation-thresholded macro-F1). All variants < attention reader.
Color model
Macro-F1
AUROC
Micro-F1
Pale-F1
global LAB/HSV + MLP
59.53
61.53
71.88
27.78
softmax-prototype
63.74
69.74
76.95
37.89
C1: prototype OT
64.79
71.65
78.28
40.15
TABLE IV: Color-pathway ablation at seed 42 (validation-thresholded test F1). C1’s prototype OT is a mild refinement over softmax-prototype attention; only TonguePale is fused into the deployed model.
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · College of Computer Science and Engineering, Northeastern University, Shenyang 110189, China