Organizations: College of Information Science and Engineering, Northeastern University, Shenyang, China · College of Sciences, Northeastern University, Shenyang, China · School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, Qinhuangdao 066004, Hebei, China
Automated tongue-sign recognition can improve the consistency of non-invasive examinations that otherwise rely on subjective visual judgment. A single tongue image often exhibits multiple coexisting signs characterized by heterogeneous evidence, including localized morphology, global chromaticity, and inter-label dependence. However, existing methods typically rely on shared representations and uniform multi-label heads, making it difficult to verify whether predictions stem from clinically relevant evidence rather than spurious visual correlations. We propose SEAT, an evidence-audited framework that matches distinct evidence types with dedicated prediction and evaluation mechanisms. Built on frozen DINOv2 patch features, SEAT couples mask-guided sign-specific attention for localized morphology, a semi-relaxed optimal transport pathway over LAB/HSV color prototypes to enhance pale-tongue recognition, and a co-occurrence-aware transport regularizer to shape the batch-level predicted label marginal. These components are evaluated through audits matched to their evidence types. On the TongueDx benchmark, SEAT achieves a mean macro-F1 of 72.59 with a standard deviation of 1.04 across six random seeds, with a six-model ensemble reaching 74.15. The ensemble outperforms a backbone-matched baseline and a larger, domain-specialized vision-language model by 8.06 and 2.99 percentage points, respectively. Our results demonstrate that aligning dedicated prediction pathways and audits with sign-specific characteristics improves multi-label accuracy while offering more targeted evidential grounding than uniform post-hoc spatial explanations. Code is available at https://github.com/NEU-TCM-Lab/Sign-type-Evidence-Audited-Transport.
Figures & tables
Fig. 1: Overview of SEAT. A frozen DINOv2-Large backbone encodes each image into a 37×37 patch grid. A mask-guided multi-head softmax attention reader forms the discriminative core. C1 adds a color-prototype optimal-transport pathway over per-patch LAB/HSV descriptors, whose pale-tongue output is combined with the reader by a validation-selected late-fusion weight. C2 adds a co-occurrence regularizer that shapes the batch-level predicted label marginal under a 1−Jaccard cost geometry.
Method
Pale
TipSide
Spot
Ecchy.
Crack
Tooth.
FurTh.
FurYe.
Macro
DINOv2-L (global)
35.04
61.56
73.77
31.29
91.21
81.04
98.58
56.19
66.09
ResNet-34
39.85
64.12
80.80
16.15
92.24
82.82
98.64
74.67
68.66
ResNet-50 [ 1 ]
32.68
66.19
76.44
24.72
91.17
84.31
98.65
76.27
68.80
Qwen3-VL-4B LoRA [ 21 ]
36.73
59.59
77.10
27.72
91.84
76.10
98.58
78.48
68.27
ShizhenGPT-7B-VL [ 22 ]
38.83
64.97
80.59
36.51
91.26
81.34
98.58
77.17
71.16
Dr. Tongue [ 1 ] (reported; not directly comparable)
46.00
68.22
81.09
38.02
91.90
82.82
98.54
76.41
72.87
TABLE I: Per-class test F1 on TongueDx (%). DINOv2-L (global), ResNet-34, Qwen3-VL-4B LoRA, ShizhenGPT-7B-VL, and SEAT use the fixed image-level split with per-class thresholds selected on validation; both vision-language baselines are LoRA-tuned. SEAT is reported both as a single model (mean ± std over seeds 42 – 47 ; per-class columns show the mean, only the Macro column carries ± std) and as the deployed six-seed ensemble. ResNet-50 and Dr. Tongue values are reproduced from Table 3 of [ 1 ] as reported contextual references and are not directly comparable to our protocol. Bold marks the highest value across all rows.
Configuration
Macro-F1
Bootstrap 95% CI
Reader
71.47±0.93
[70.43,72.52]
Reader + C1
71.97±0.89
[70.93,73.04]
Reader + C2
71.91±1.24
[70.86,72.95]
Reader + C1 + C2
72.59±1.04
[71.53,73.67]
Paired gain
Gain
Bootstrap 95% CI
C1 − Reader
+0.50
[+0.21,+0.78]
TABLE II: Component 2×2 factorial (seeds 42–47), all selection on validation. Top: validation-thresholded test macro-F1 as mean ± population std across seeds, with a test-image bootstrap 95% CI. Bottom: paired gains with their bootstrap CIs. C1 improves every seed both alone and with C2; C2 is secondary and complementary (see text).
Fig. 2: Multi-seed component effects across seeds 42–47. (a) Per-sign F1 for the Reader and Reader+C2, reported as mean ± standard deviation across seeds; labels show the mean paired change. (b) TonguePale F1 before and after C1 fusion on top of Reader+C2; thin lines connect matched seeds. All thresholds and fusion weights are selected on validation. Aggregate macro-F1 results and bootstrap confidence intervals are reported in Table II .
Fig. 3: Evidence-type coverage, read as a capability matrix. Left to right: (A) a ResNet-34 Grad-CAM, deletion-sensitive within the CNN at 1.851 on Toothmark; (B) our attention-based spatial evidence, load-bearing under readout-mask suppression ( 2.386 , across Toothmark, Crack, and Spot); (C) a descriptor-space color intervention on the color pathway for the pale-tongue sign, whose directed effect exceeds every one of seven matched non-clinical color directions and which a spatial map does not express; and (D) the sign co-occurrence geometry ( 1−Jaccard ) used by C2, an output-space structure rather than an image map.
Configuration
Macro-F1
Attention reader
73.42
SignOT: global + OT (joint / frozen)
72.95 / 72.76
SignOT: OT only (no global)
70.81
AOT: attention-demand sparse (balanced)
72.54
AOT: unbalanced ρ=0.1 / partial m=0.7
72.18 / 72.49
OTGen: K=4 mean / K=4 concat
72.45 / 72.58
TABLE III: OT-as-readout ablation at seed 42 (validation-thresholded macro-F1). All variants < attention reader.
Color model
Macro-F1
AUROC
Micro-F1
Pale-F1
global LAB/HSV + MLP
59.53
61.53
71.88
27.78
softmax-prototype
63.74
69.74
76.95
37.89
C1: prototype OT
64.79
71.65
78.28
40.15
TABLE IV: Color-pathway ablation at seed 42 (validation-thresholded test F1). C1’s prototype OT is a mild refinement over softmax-prototype attention; only TonguePale is fused into the deployed model.
Deep learning has shown promise for automated tongue diagnosis in traditional Chinese medicine (TCM), yet the design space remains underexplored. We conducted a systematic ablation study spanning 20+ model versions under rigorous 5-fold cross-validation on TongueDx2 (5,109 images, 976 expert-annotated) and a merged dataset of 11,101 samples. We compared six backbone architectures, four loss functions, five augmentation strategies, and six training strategies. The best 976-sample model achieved weighted-F1 of 0.6625 using ConvNeXt-Tiny with restrained augmentation and weak-group ensemble, while the best 11,101-sample model reached weighted-F1 of 0.7761. Six key design principles emerged: (1) ConvNeXt-Tiny offers optimal parameter efficiency; (2) BCE substantially outperforms Asymmetric Loss (+2.7%); (3) restrained color augmentation is critical; (4) weak-group ensemble replacement (+2.1%) outperforms probability averaging; (5) data scaling yielded +20.6% improvement; (6) expanding from 13 to 45 label dimensions caused catastrophic collapse (0.78 to 0.22). These principles are generalizable to multi-label medical image classification with class imbalance.
Longxia Gao, Linan Wang, Yuhe Han +3
College of Traditional Chinese Medicine, Hebei University, Baoding, Hebei, China
Multimodal Large Language Models (MLLMs) have excelled in Western medical imaging, yet their potential in Traditional Chinese Medicine (TCM) tongue diagnosis remains systematically unevaluated. Tongue diagnosis requires sophisticated global and local perception, serving as a rigorous touchstone for assessing the visual understanding depth of MLLMs. However, existing datasets are limited by single dimensions, coarse granularity, and ambiguous labels, hindering the evaluation of authentic clinical capabilities. To bridge this gap, we introduce OmniTongue, the first comprehensive, fine-grained Visual Question Answering benchmark for TCM tongue diagnosis. This benchmark integrates 1,351 expert-validated images and 21,616 instruction pairs. Its core design features 16 clinical dimensions for multidimensional evaluation, region-specific annotations for pathological localization, and 66 fine-grained labels for precise diagnostic representation. Extensive evaluations of existing MLLMs reveal significant performance gaps, demonstrating that medical-specific models lack domain advantages, while universally poor Macro-F1 scores expose a reliance on statistical shortcuts rather than genuine understanding of tongue pathology. These findings not only highlight the current limitations of MLLMs in interpreting regional medical features but also underscore OmniTongue's significance for developing robust automated tongue diagnosis models. The code and dataset will be made publicly available in a future release.
Liuyi Xu, Yun Guo, Shusheng Li +3
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · College of Computer Science and Engineering, Northeastern University, Shenyang 110189, China
In segmentation-based continuous sign language recognition, an untrimmed video is first partitioned into lexical units; segmentation is therefore a means to an end---downstream recognition. However, segmenters are usually judged by local boundary detection rather than by the quality of the lexical segments and the recognition they enable. We present BEACON, a begin/end-aware segmenter that combines four-class Outside/Begin/Inside/End (BIOE) labeling, constrained Viterbi decoding over eight legal transitions, and boundary knowledge distilled from a binary teacher into a shared student. Under strict no-cut evaluation with validation-based model selection, BEACON improves segment-F1 by 14.58/14.59 points and boundary-F1 by 2.17/1.22 points, and reduces the downstream word error rate (WER) by 9.10/4.15 points relative to the state-of-the-art in the PHOENIX-14/BSL Corpus. To examine which segmentation metric better reflects downstream recognition, we evaluated five distinct architecture families. Segment-F1 aligns more consistently with lower WER than boundary-F1 in both corpora (Spearman ρ=−1.0/−0.9 versus −0.9/−0.8). These results support evaluating sign language segmentation in terms of segment quality and downstream recognition utility rather than local boundary accuracy alone.
Guohou Zhang, Kalin Stefanov
Monash University, Australia · Japan Advanced Institute of Science and Technology, Japan