Structure-aware Keypoint Localization for Videofluoroscopic Swallowing Study

Kai Zhou; Chuanshen Chen; Runhao Zeng; Meng Dai; Yifan Yang; Jinwu Hu; Daiyuan Li; Mingkui Tan

cs.CVOct 6, 2026

Structure-aware Keypoint Localization for Videofluoroscopic Swallowing Study

Authors: Kai Zhou, Chuanshen Chen, Runhao Zeng, Meng Dai, Yifan Yang, Jinwu Hu, Daiyuan Li, Mingkui Tan, +1 more

Organizations: South China University of Technology · Shenzhen MSU-BIT University · The Third Affiliated Hospital of Sun Yat-sen University · Electric Power Research Institute, China South Grid

Abstract

Videofluoroscopic Swallowing Study (VFSS) is one of the gold standard for diagnosing swallowing disorders, providing dynamic X-ray imaging of the swallowing process. Automated kinematic analysis in VFSS relies fundamentally on precise anatomical keypoint localization. However, existing studies focus on limited keypoints (e.g., cervical vertebrae or the hyoid) and overlook critical regions such as the soft palate, while annotating only active swallowing segments and ignoring abundant non-swallowing data, resulting in poor data efficiency. Moreover, leveraging this unlabeled data via standard semi-supervised learning is suboptimal, as generic methods are prone to spatial bias. In medical X-rays with fixed layouts, models tend to memorize absolute coordinates rather than understanding anatomical structures. To tackle these challenges, we introduce VFSSKep, a novel dataset that extends annotations to the soft palate and incorporates large-scale unlabeled data. We further propose S $^3$ KL, a Structure-aware Semi-Supervised Keypoint Localization framework designed to overcome spatial bias. It integrates a Structure-Aware Learning strategy to extract high-resolution structural cues for structure-aware representation learning, and a Structural Representation Consistency Learning strategy with block shuffling to enforce invariant structural recognition. Experiments show our method achieves state-of-the-art semi-supervised performance, even with unlabeled and 25% labeled data surpassing fully supervised learning with 100% labeled data. Code and data will be made publicly available at: https://github.com/kaai520/S3KL.

Figures & tables

Fig. 1: Illustration of the eight keypoints in our VFSSKep dataset.

Dataset	Year	Anatomical Structure	Labeled Frames	Unlabeled Frames
Zhang et al. [ 6 ]	2021	Cervical Vertebrae	60k	-
Hsiao et al. [ 7 ]	2023	Hyoid + Vertebrae	7k	-
Ours (VFSSKep)	2025	Hyoid + Vertebrae + Soft Palate	14k	32k

TABLE I: Comparison of VFSS keypoint datasets. Our VFSSKep is the first to include soft palate annotations and leverage large-scale unlabeled data for semi-supervised learning.

Fig. 2: Comparison of the traditional (a) and our proposed (b) paradigms for utilizing unlabeled data in semi-supervised keypoint localization. Our paradigm leverages dense anatomical context to assist keypoint localization, learning structural priors without extra data or annotations.

Fig. 3: Summary of the construction of our VFSSKep dataset.

Fig. 4: An overview of our proposed S 3 KL framework. Our proposed S 3 KL framework applies easy augmentation $T_{e}$ to an unlabeled image for teacher predictions, transformed via easy-to-hard mapping $T_{e\to h}$ as pseudo-labels for the student. Beyond keypoint supervision, we propose a Structure-Aware Learning (SAL) strategy to leverage dense anatomical cues, assisting keypoint localization by aligning the auxiliary prediction head output with structural prior from the Structure Pairing Module. The output from the Structure Parsing Module is visualized using PCA [ 14 ] . In addition, we introduce a Structural Representation Consistency (SRC) Learning strategy, where the Block Shuffle operation disrupts spatial structure, encouraging the model to maintain consistent representations despite such perturbations. The training process of labeled images is omitted for simplicity.

Fig. 5: (a) X-ray image segmented by SAM [ 16 ] with regions highlighted in different colors, but without semantic labels. (b) Output features from DINOv2 [ 13 ] for the same image, visualized by mapping PCA results to RGB channels. (c) Output features from our Structure Parsing Module (SPM), also visualized using PCA mapping.

Method	25%		50%		100%
Method	[email protected] (%)↑	MED (mm)↓	[email protected] (%)↑	MED (mm)↓	[email protected] (%)↑	MED (mm)↓
Supervised [ 20 ]	63.2	4.34	73.5	3.47	75.6	3.29
SemiPose [ 10 ]	77.2	3.63	79.0	3.14	81.9	2.65
+S 3 KL (Ours)	78.8 ( ↑2.1% )	2.88 ( ↓20.7% )	82.1 ( ↑3.9% )	2.64 ( ↓15.9% )	83.6 ( ↑2.1% )	2.48 ( ↓6.4% )
G2LCPS [ 12 ]	78.2	2.98	81.0	2.76	83.1	2.54
+S 3 KL (Ours)	79.4 ( ↑1.5% )	2.87 ( ↓3.7% )	82.9 ( ↑2.3% )	2.57 ( ↓6.9% )	83.8 ( ↑0.8% )	2.46 ( ↓3.1% )

TABLE II: Comparison with methods using different proportions of labeled data. [email protected] (%) and MED (mm) are reported. Our S 3 KL is plug-and-play, where variations in unsupervised loss

\mathcal{L}_{u}

across different methods do not affect its integration. Best results are in bold , and blue numbers show relative improvements over SemiPose and G2LCPS.

Method	25%		50%		100%
Method	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
Baseline [ 10 ]	77.2	3.63	79.0	3.14	81.9	2.65
Baseline w/ SAL	78.1	3.22	81.6	2.69	82.4	2.58
Baseline w/ SRC	78.4	3.62	81.2	2.72	82.7	2.57
S 3 KL (Ours)	78.8	2.88	82.1	2.64	83.6	2.48

TABLE III: Ablation study on the proposed Structure-Aware Learning (SAL) and Structural Representation Consistency (SRC) Learning. [email protected] (%) and Mean Euclidean Distance (MED, in mm) are reported.

Model	Upsample	25%		50%		100%
Model	Upsample	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
CLIP	FeatUp	77.8	3.73	80.5	2.86	81.9	2.74
DINOv2	FeatUp	78.1	3.22	81.6	2.69	82.4	2.58
DINOv2	Bilinear	77.8	2.97	80.9	2.79	81.9	2.68

TABLE IV: Effect of auxiliary models and upsampling strategies on the Structure Parsing Module. [email protected] (%) and Mean Euclidean Distance (MED, in mm) are reported.

Method	25%		50%		100%
Method	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
w/o BS	77.2	3.63	79.0	3.14	81.9	2.65
BS (final)	77.7	3.16	80.9	2.77	82.5	2.59
BS (encoder)	78.4	3.62	81.2	2.72	82.7	2.57

TABLE V: Effect of unshuffle position on the Block Shuffle (BS) operation. “final” indicates applying unshuffle to the final output, while “encoder” indicates applying unshuffle to the encoder output.

Fig. 6: Comparison of keypoint localization results between the baseline method [ 10 ] and our S 3 KL under 25% supervision. Green points represent the ground truth positions, while blue points represent the predicted keypoints.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Unlabeled ( $n=62$ )	Train ( $n=28$ )	Val ( $n=9$ )	Test ( $n=28$ )
Male	37.1%	39.3%	55.6%	39.3%
Female	62.9%	60.7%	44.4%	60.7%
Age 10–30	9.7%	7.1%	33.3%	7.1%
Age 30–50	25.8%	25.0%	11.1%	21.4%
Age 50–70	41.9%	42.9%	44.4%	46.4%
Age 70+	22.6%	25.0%	11.1%	25.0%

Appendix

TABLE I: Demographic distribution of VFSSKep subjects. Values are percentages;

n

: number of subjects.

Stage	Architecture	Output sizes $C\times H\times W$
input		$512\times 8\times 8$
dec 1	deconv $4\times 4,256$ (stride=2, padding=1)	$256\times 16\times 16$
dec 2	deconv $4\times 4,256$ (stride=2, padding=1)	$256\times 32\times 32$
dec 3	deconv $4\times 4,256$ (stride=2, padding=1)	$256\times 64\times 64$
pred	conv $1\times 1,384$ (stride=1, padding=0)	$384\times 64\times 64$

Appendix

TABLE II: An example implementation of the SAB prediction head using ResNet18 as the backbone. The deconvolutional layers (dec 1 , dec 2 , dec 3 ) progressively upsample the features, and a final

1\times 1

convolution (pred) produces the output.

$\lambda_{3}$	25%		50%		100%
$\lambda_{3}$	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
10 -1	69.2	3.60	73.7	4.08	75.3	3.14
10 -2	78.1	3.56	80.0	2.90	81.1	2.75
10 -3	78.8	2.88	82.1	2.64	83.6	2.48
10 -4	78.4	3.62	80.6	3.32	82.4	2.56

Appendix

TABLE III: Effect of loss weight

\lambda_{3}

for feature alignment loss. [email protected] (%) and Mean Euclidean Distance (MED, in mm) are reported.

$\alpha$	25%		50%		100%
$\alpha$	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
0.7	78.4	2.94	82.0	2.87	83.4	2.49
0.85	78.8	2.87	82.1	2.64	83.6	2.48
1.0	78.7	3.66	81.6	2.87	82.8	2.58

Appendix

TABLE IV: Effect of the similarity threshold

\alpha

in the cosine similarity loss

\mathcal{L}_{cos}

. [email protected] (%) and Mean Euclidean Distance (MED, in mm) are reported.

Size	25%		50%		100%
Size	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
32	78.5	2.95	81.8	2.75	82.6	2.68
64	78.8	2.87	82.1	2.64	83.6	2.48
128	78.4	3.65	81.4	3.13	82.3	2.60

Appendix

TABLE V: Effect of input block size on the Block Shuffle operation. [email protected] (%) and Mean Euclidean Distance (MED, in mm) are reported.

Backbone	Param	FLOPs	25%		50%		100%
Backbone	Param	FLOPs	PCK↑	MED↓	PCK↑	MED↓	PCK↑	MED↓
ResNet18	15.6M	2.4G	78.8	2.88	82.1	2.64	83.6	2.48
ViT-B	87.6M	21.6G	79.9	2.74	82.3	2.55	83.5	2.51

Appendix

TABLE VI: Effect of different backbones.

Fig. 1 : Partial screenshot of the clinical prototype system.

Explore similar work

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective
Dec 11, 2025Tian Liu, Anwesha Basu, James Caverlee +1VLM Adaptation Few-Shot Learning
Texas A&M University · University of Macau · Institute of Collaborative Innovation
ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation
Jun 18, 2026Tong Wang, Siwen Wang, Yaolei Qi +4Video Object Segmentation Semi-Supervised Medical Image Segmentation
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Ministry of Education, Jiangsu, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · School of Medicine, Case Western Reserve University, Cleveland, OH, USA
SCKAN: Structural Consensus-based KAN Prototype Learning for Semi-Supervised Pancreas Segmentation
May 26, 2026Yuqi Liu, Yufei Chen, Wei Fu +2Prototype Learning Medical Image Segmentation
School of Computer Science and Technology, Tongji University, Shanghai, China · Artificial Intelligence Institute, Shanghai University, Shanghai, China · Department of Computer and Data Science and Department of Biomedical2026 Engineering, Case Western Reserve University, Cleveland, USA

Dec 11, 2025, cs.CVJ/K move · Enter open · S save

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

Texas A&M University · University of Macau · Institute of Collaborative Innovation

VLM Adaptation Few-Shot Learning Pseudo-Labeling Semi-Supervised Learning

Open paper AskPDF arXiv

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones. Despite the availability of powerful open-source Vision-Language Models (VLMs) and open-world data, existing SSFSL literature largely neglects these resources. In contrast, the related area few-shot learning (FSL) has already exploited them to boost performance. Arguably, to solve real-world auto-annotation, SSFSL should leverage such open resources. To bridge this gap, we explore established SSL methods to finetune a VLM. Unexpectedly, they significantly underperform FSL baselines that do not use unlabeled data. Our in-depth analysis reveals the root cause of failure: VLMs produce flat distributions of softmax probabilities, resulting in zero utilization of unlabeled data and weak supervision signals. To address this challenge, we propose an embarrassingly simple solution that uses temperatures to sharpen the softmax output, which not only increases the confidence scores of pseudo-labels to improve the utilization of unlabeled data, but also strengthens training supervision for effective finetuning. Furthermore, we exploit task-relevant open data, e.g., those retrieved from VLMs' publicly available pretraining set. To mitigate the imbalance and domain gaps in retrieved data, we employ a stage-wise training strategy. Building on the successful finetuning of VLMs and the exploitation of open data, we present a simple yet effective SSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT). Across five benchmarks, SWIFT outperforms recent FSL and SSL methods by

\sim

5 accuracy points. SWIFT even rivals supervised learning, which finetunes a VLM assuming unlabeled data having ground-truth labels!

Abstract

Explore similar work

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation

SCKAN: Structural Consensus-based KAN Prototype Learning for Semi-Supervised Pancreas Segmentation