SemReward-VL: Semantic Reward-Guided Video-Language Adaptation for Developmental Behavior Assessment
Authors: De Jiang, Shuo Zhang, Kehong Yuan, Hongen Liao
Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · School of Biomedical Engineering, Tsinghua University, Beijing, China
Developmental screening videos show how children perform specific behaviors, but clinical records usually contain outcomes rather than descriptions of what happened. We present SemReward-VL, which learns to describe item-specific behavior from these outcomes. A vision-language model generates a description, and a frozen language model scores its agreement with the clinical outcome, relevance to the item, abstention on unrelated video-item pairs, and clarity. Group relative policy optimization (GRPO) updates LoRA adapters using this semantic reward. On 13,379 videos covering 41 items, the method improves aggregate accuracy and the number of items with recall above 0.5. Errors remain in temporal direction, duration, and age-specific interpretations of behavior.
Figures & tables
Figure 1: Overview of SemReward-VL. The VLM generates item-conditioned descriptions, a frozen LLM supplies semantic rewards, and GRPO updates rank-16 LoRA adapters; inference reuses the adapted descriptor before the final item decision.
Setting
Value
Group size G
16
LoRA target
VLM language layers
LoRA rank
16
Optimizer / learning rate
AdamW / 10−5
Batch size / epochs
8 / 2
Training iterations
5
Table 1: Training configuration.
Statistic
Value
Children
4,670
Videos
13,379
Age range / mean
2–42 / 12.73 months
Male / female
2,313 / 2,357
Key developmental ages
8
Evaluated behavior items
41
Table 2: Dataset summary.
Figure 2: Item-level recall before and after adaptation. Bars denote the base VLM and lines the GRPO-LoRA model over 41 age–item codes; adapted Qwen3-VL reaches recall >0.5 on 38/41 items.
Backbone
Base
+GRPO-LoRA
Abs. gain
Qwen2.5-VL
0.669
0.817
+0.148
Qwen3-VL
0.808
0.890
+0.082
Table 3: Aggregate accuracy under the per-item 90/10 split.
Setting
Items with recall >0.5
Generic domain prompt
5 / 41
Item-specific prompt
30 / 41
Item-specific prompt + SemReward-VL
38 / 41
Table 4: Behavior coverage counted by items with recall >0.5 .
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.