Organizations: School of Robotics, Beijing Union University, Beijing 100101, China · College of Computer and Cyber Security, Hebei Normal University, Shijiazhuang 050010, China · Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100190, China · PeopleAI, Inc., Beijing 100190, China · People Youhe Education Technology Co., Ltd., Shandong 250033, China
Image captioning for Early Childhood Education (ECE) is essential for automated activity understanding and educational assessment. However, existing methods face two key challenges. First, the lack of large-scale, domain-specific datasets limits the model's ability to capture fine-grained semantic concepts unique to ECE scenarios, resulting in generic and imprecise descriptions. Second, conventional training paradigms exhibit limitations in enhancing professional object description capability, as supervised learning tends to favor high-frequency expressions, while reinforcement learning may suffer from unstable optimization on difficult samples. To address these limitations, we introduce ECAC, a large-scale benchmark for ECE daily activity image captioning, comprising 256,121 real-world images annotated with expert-level captions and fine-grained labels. ECAC is further equipped with a domain-oriented evaluation protocol, the Teaching Toy Recognition Score (TTS), to explicitly measure professional object naming accuracy. Furthermore, we propose RSRS (Reward-Conditional Switch of Reinforcement Learning and Supervised Fine-Tuning), a hybrid training framework that dynamically alternates between RL and supervised optimization. By rerouting hard samples with zero rewards to supervised fine-tuning, RSRS effectively mitigates advantage collapse and enables stable optimization for fine-grained recognition. Leveraging ECAC and RSRS, we develop KinderMM-Cap-3B, a domain-adapted multimodal large language model. Extensive experiments demonstrate that our model achieves a TTS of 51.06, substantially outperforming state-of-the-art baselines while maintaining superior caption quality, highlighting its potential for specialized educational applications.
Figures & tables
Fig. 1: Daily Activity Images Across Three Scenario Categories
Prompt
根据图片定位到幼儿园的具体场景,根据这张图片内容生成一段流畅且连贯的对主要内容的描述。我希望句子是简洁但包含关键细节,例如发生的具体场景,主要人物的动作、表情以及精确识别与主要人物有关的物品。不要描述与主要人物无关的背景,不要描述与当前主要活动无关的物体,不要描述没有露脸或与画面主要内容无关的人物,不要描述穿着的日常衣服(特殊装扮需要描述),不要描述不必要的表情。(Locate the specific kindergarten scene based on the image, and generate a smooth and coherent description of its main content according to the image. I want the sentences to be concise but include key details, such as the exact setting, the main characters’ actions and expressions, and accurate identification of objects related to the main characters. Do not describe background elements unrelated to the main characters, objects irrelevant to the main activity, people whose faces are not visible or who are unrelated to the main content, ordinary clothing (describe only special costumes), or unnecessary expressions.)
根据图片定位到幼儿园的具体场景,根据这张图片内容生成一段流畅且连贯的对主要内容的描述。我希望句子是简洁但包含关键细节,例如发生的具体场景,主要人物的动作、表情以及精确识别与主要人物有关的物品。不要描述与主要人物无关的背景,不要描述与当前主要活动无关的物体,不要描述没有露脸或与画面主要内容无关的人物,不要描述穿着的日常衣服(特殊装扮需要描述),不要描述不必要的表情。(Locate the specific kindergarten scene based on the image, and generate a smooth and coherent description of its main content according to the image. I want the sentences to be concise but include key details, such as the exact setting, the main characters’ actions and expressions, and accurate identification of objects related to the main characters. Do not describe background elements unrelated to the main characters, objects irrelevant to the main activity, people whose faces are not visible or who are unrelated to the main content, ordinary clothing (describe only special costumes), or unnecessary expressions.)
Generation
班级美工区中,两个小女孩正在桌子上专注地进行手工制作。左边的小女孩穿着粉色带花图案的上衣,她正用手摆弄着 桌上的材料 ;右边的小女孩穿着粉色上衣,上面有卡通图案,她也在认真地操作着 手中的物品 。桌上摆放着各种 手工材料 、一台显示着画面的平板电脑以及一些已经制作好的 小物件 ,她们似乎很享受这充满创意的手工制作过程。(In the classroom art corner, two little girls are intently engaged in hand-icraft work at the table. The girl on the left is wearing a pink top with floral patterns, and she is fiddling with the materials on the table. The girl on the right is wearing a pink top with cartoon patterns, and she is also carefully working with the item in her hands. On the table are various handicraft materials , a tablet computer displaying an image, as well as some small items that have already been made. They seem to be thoroughly enjoying this creative handicraft-making process.)
在班级美工区,一个小男孩正坐在铺有花卉图案桌布的桌子旁,手里拿着一块 绿色的彩泥 ,似乎正在专注地进行手工创作,他神情认真,沉浸在自己的手工世界里,周围还有其他小朋友也在进行着类似的活动。(In the classroom art corner, a little boy is sitting at a table covered with a floral-patterned tablecloth, holding a piece of green clay in his hand. He seems to be deeply focused on his handicraft creation, with a serious expression, fully immersed in his own world of craft-making. Around him, other children are also engaged in similar activities.)
TABLE I: Different descriptions of the same educational toy
Category
Number of Images
Outdoor
56,665
Classroom
195,397
Specialty
4,059
Total
256,121
TABLE II: Distribution of images across different activity Areas in the dataset.
Fig. 2: Distribution of images across different regions in the dataset.
Fig. 3: Overview of the revised KinderMM-Cap framework. The training pipeline comprises three stages: (1) Stage 1 (Base Model), where a general vision–language model generates inaccurate descriptions of Early Childhood Education (ECE) activities; (2) Stage 2 (SFT Warm-up), where supervised fine-tuning improves caption quality but still yields semantic deviations; and (3) Stage 3 (RSRS Core Training), where samples are adaptively split by intra-group rewards—hard samples (zero reward) are optimized via the SFT branch, while others are refined through the GRPO branch to encourage exploration. The final model produces accurate and semantically grounded ECE captions. In the figure, red-highlighted text denotes incorrect descriptions, purple indicates imprecise descriptions, and green represents accurate descriptions.
Fig. 4: Diagram of the RSRS Framework. In the diagram, O denotes the candidate captions generated by the model given the input image and prompt; R represents the reward computed from the consistency between the generated captions and annotated teaching toys; A refers to the normalized within-group advantage, which measures the relative quality of each candidate caption compared to the group average. In addition, the buffer stores samples whose entire group receives zero reward, and once it reaches the batch size, the SFT branch is activated to prevent advantage collapse and enhance the model’s learning ability on difficult samples.
Model
TTS
Score
GLM-4v-9B
29.36
78.19
Gemma3-27B
10.74
12
Qwen2.5-VL-3B
38.07
84.63
Qwen2.5-VL-7B
45.25
85.70
KinderMM-Cap-3B(ours)
51.06
86.20
TABLE III: Performance Comparison Table of MLLMs for Daily Activity Image Captioning in ECE
Method
Kendall’s τ
Spearman’s ρ
CN-CLIP
0.11
0.13
GPT-4o
0.43
0.52
Gemma3-27B
0.26
0.32
Ours
0.67
0.72
TABLE IV: Correlation Between Automatic Ranking Methods and Human Judgments.
Fig. 5: Ablation Study on Training Strategies for ECE Daily Activity Captioning.
Model
TTS-FL
TTS-FM
TTS-FH
TTS-BL
TTS-BM
TTS-BH
baseline
57.47
8.88
0.03
24.52
1.53
0.01
baseline+SFT
68.21
15.71
0.04
24.15
2.01
0.0
baseline+SFT+GRPO
65.66
14.39
0.02
37.97
2.86
0.02
baseline+SFT+RSRS
66.59
17.18
0.09
39.69
3.87
0.03
TABLE V: Multi-Dimensional Evaluation of Toy Recognition Metrics Across Training Methods
Model
Avg. Toy Count
Precision
baseline
1.45
64.15
baseline+SFT
1.66
69.59
baseline+SFT+GRPO
1.84
64.43
baseline+SFT+RSRS
2.08
56.43
TABLE VI: Toy Recognition Precision in ECE Captioning with Different Training Strategies
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China · LLM Team, Shopee Pte. Ltd., Shanghai, China