cs.CVSep 26, 2026
SaveREMEDY: How Far Is Video Generation from Medical Education World Models?
Organizations: Beihang University · Beijing Institute of Technology · Beijing Information Science and Technology University
Abstract
Recent video generation models produce realistic videos and show potential as a foundation for world models. These advances create opportunities for generating medical teaching demonstrations, which requires both convincing visual quality and precise procedural actions. However, whether current generators can meet these requirements has not been measured. To address this problem, we introduce Readiness Evaluation of Medical Education Demonstration sYnthesis (REMEDY), to our knowledge, the first benchmark for AI-generated medical teaching demonstrations. REMEDY provides 900 first frames from real demonstration videos, covering 12 tasks across four scenarios: operating room, imaging, clinic and bedside, and resuscitation. Five contemporary open-source video generation models produce 4,500 videos from these frames. We combine task-specific clinical checklists with video and motion quality metrics. Evaluation covers four dimensions: clinical action following, clinical profiles, video quality, and motion quality. Our results show that realistic appearance and temporal consistency do not ensure correct clinical actions. Even the most advanced MiniMax-H3 achieves only 28.25% on strict clinical success rate, and fine-grained clinical actions remain challenging. These findings establish a foundation and roadmap for developing future medical education world models.
Figures & tables
| Clinical data | Evaluation dimensions | |||||||
| Benchmark | Task | Medical tasks | Real clinical frames | Performer in frame | Clinical action following | Clinical profiles | Video quality | Motion quality |
| VBench ( Huang et al., 2024 ) | T2V | – | – | – | – | – | – | |
| Video-Bench ( Han et al., 2025 ) | T2V | – | – | – | – | – | – | |
| UI2V-Bench ( Zhang et al., 2025 ) | I2V | – | – | – | – | – | ||
| HumanScore ( Fang et al., 2026 ) | T2V | – | – | – | – | |||
| EduVQA ( Chen et al., 2026a ) | T2V | – | – | – | – | – | ||
Table 1: Comparison of REMEDY with existing video generation benchmarks. Benchmarks are compared by generation setting, clinical data, and evaluation dimensions. = full coverage, = partial coverage, – = absent.
Figure 1: Overview of the REMEDY benchmark. Left: data collection, prompt construction, and video generation. Middle: example videos across four medical scenarios. Right: case-specific questions illustrate clinical action following, clinical profiles, video quality, and motion quality. Green, yellow, and red marks indicate satisfied, partly satisfied, and unsatisfied criteria.
Figure 2: Data construction pipeline of REMEDY. (1) Definition of four medical scenarios and selection of source datasets. (2) Screening, frame cleaning, and de-duplication yield 900 cases. (3) Prompt construction and human review align each case with a first frame, reference video, and clinical checklist. (4) Five contemporary open-source TI2V models generate 4,500 videos from the prompts and first frames.
Figure 3: Dataset statistics of REMEDY. (a) Taxonomy of twelve tasks across four medical scenarios. (b) Task distribution over 900 cases; bar length indicates case count, and icons correspond to (a). (c) Prompt word cloud, with word size indicating case frequency.
| Dimension | Metric | Evaluation focus |
|---|---|---|
| Clinical action following | Strict clinical success | Whether every task-essential atom passes with no critical failure. |
| Clinical Atom Score (CAS) | Mean value of the task’s atoms on 0-100. | |
| Clinical profiles | Clinical technique (1-5) | Correctness of the visible technique for the requested procedure. |
| Biomechanics (1-5) | Physical plausibility of body motion and clinician-patient contact. | |
| Professionalism (1-5) | Professional conduct, including step order and asepsis where required. | |
| Instruction following (1-5) | Adherence to the prompt’s action and camera instructions. |
Table 2: Hierarchical evaluation framework of REMEDY. The framework is organized into four major dimensions: clinical action following, clinical profiles, video quality, and motion quality, with 12 metrics for assessing generated medical teaching demonstrations.
| Strict clinical success (%) | ||||||||||||
| Model | PKA | TKA | MISS | US | Inj. | Exam | Wound | Dent. | IV | Sut. | CPR | Air |
| MiniMax-H3-FL2VA | 25.07 | 6.00 | 13.00 | 9.52 | 16.00 | 74.00 | 68.00 | 62.50 | 50.00 | 6.67 | 15.00 | 40.00 |
| Wan2.2-TI2V-5B | 21.97 | 2.00 | 5.05 | 13.10 | 12.00 | 44.00 | 34.00 | 17.50 | 7.50 | 0.00 | 5.00 | 16.67 |
| LTX-2.5 | 12.11 | 6.00 | 10.00 | 4.71 | 6.00 | 50.00 | 16.00 | 15.00 | 2.50 | 0.00 | 0.00 | 33.33 |
| LTX-2.3v | 0.85 | 0.00 | 0.00 | 14.29 | 4.00 | 36.00 | 6.00 | 2.50 | 2.50 | 3.45 | 0.00 | 6.67 |
| LTX-Video-2B | 1.41 | 0.00 | 0.00 | 3.53 | 4.00 | 34.00 | 6.00 | 10.00 | 0.00 | 0.00 | 0.00 | 3.33 |
Table 3: Evaluation results on REMEDY. Five contemporary open-source models are compared on task-level strict clinical success and CAS (top and middle), and clinical profiles, video quality, motion quality, and generation speed (bottom). Bold and underlined values indicate the best and second-best results.
Figure 4: Scenario-level clinical performance. (a) Strict clinical success (%); (b) CAS (0-100); (c) mean clinical profile score (1-5).
Figure 5: Typical failure modes in generated medical videos. (a) Action omission; (b) anatomical distortion; (c) procedure interruption; (d) tool-tissue interaction failure; (e) temporal ghosting; (f) camera instruction violation. Each example shows three temporally ordered frames. Red dashed boxes highlight the failures.
| MiniMax-H3 | Wan2.2-5B | LTX-2.5 | LTX-2.3 | LTX-2B | |
| Ranking score (1-5) | 4.58 | 3.29 | 2.50 | 2.63 | 1.75 |
Table 4: Human rankings of clinical credibility. Six raters assess 100 generated videos each, totaling 600 videos, and rank the five model outputs within each group sharing the same input conditions. Scores are (1-5, higher is better). Bold indicates the highest mean.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Question | Required evidence |
|---|---|
| Is the requested procedure visibly performed with task-relevant motion, rather than only a pose, hand jitter, or unrelated movement? | Cite separated labelled frames showing the task begins and continues or reaches a later state. Hidden or omitted action is FAIL. |
| Does the camera remain locked off, without viewpoint motion, zoom, reframing, or crop drift beyond negligible encoding jitter? | Compare stable background landmarks near the beginning, middle, and end. |
| Are the visible people, anatomy, clothing, equipment, layout, framing, and scale preserved from the conditioning frame? | Compare the conditioning frame with labelled generated frames and name material drift. |
Table 5: Shared checks for generated videos.
| Task | Task-specific question |
|---|---|
| Partial knee arthroplasty | Are visible instruments handled deliberately without implausible detachment or penetration? |
| Total knee arthroplasty | Does visible team activity sustain or advance the ongoing procedure? |
| Minimally invasive spine surgery | Is visible imaging or spine equipment positioned and handled in a controlled way? |
| Ultrasound | Does the transducer maintain continuous plausible contact with the intended body surface? |
| Injection | Does the visible sequence progress through a coherent injection action? |
| Physical examination | Are deliberate examination maneuvers visibly performed rather than a static touch? |
Table 6: Example clinical questions for all twelve tasks. One question per task is shown; scoring uses the full checklist.
| Metric | Features and aggregation |
|---|---|
| Visual aesthetics | LAION aesthetic head ( Schuhmann and others, 2022 ; Huang et al., 2024 ) over CLIP ViT-L/14 features ( Radford and others, 2021 ) ; 32 uniformly sampled frames. |
| Subject consistency | DINO ViT-B/16 features ( Caron and others, 2021 ) ; 32 uniformly sampled frames. |
| Background consistency | CLIP ViT-B/32 features; 32 uniformly sampled frames. |
| Technical continuity | Full-clip penalties for black frames, repetition, freezes, and flashes. |
| PoseAct | YOLO11m-Pose body tracks ( Jocher and Qiu, 2024 ) ; smoothed, torso-normalized trajectories, joint angles, velocities, and motion energy compared with the real reference. Average valid pairs within tasks, then equally across twelve tasks. |
| PoseCov | Task-balanced fraction of valid generated/reference track pairs; failed tracks are not imputed. |
Table 7: Implementation of video and motion metrics.
| Comparison | Strict Clinical Success Disagreements | Clinical Technique | Biomechanics | Professionalism | Instruction Following |
|---|---|---|---|---|---|
| Inter-Human | 0 | 0.907 | 0.921 | 1.121 | 1.501 |
| GPT-4o vs. Human | 2 | 1.217 | 1.705 | 1.600 | 1.035 |
| GPT-4o vs. Qwen-32B | 4 | 1.317 | 1.045 | 1.340 | 1.515 |
| GPT-4o vs. Gemini | 1 | 1.402 | 1.410 | 1.595 | 1.627 |
Table 8: Agreement of clinical scores on 450 human-reviewed generated videos. Strict Clinical Success reports the number of videos with disagreeing binary judgments out of 450. The four clinical-profile columns report Mean L1 on the original 1–5 score scales. Lower is better in all columns.
| Judge | Strict Clinical Success (%) | CAS | Clinical Technique | Biomechanics | Professionalism |
|---|---|---|---|---|---|
| GPT-4o | 93.33 | 94.79 | 4.79 | 4.63 | 4.86 |
| Qwen3-VL-32B | 85.00 | 88.75 | 4.38 | 4.33 | 4.40 |
Table 9: Clinical scores on real reference videos. Strict clinical success is reported as a percentage. CAS and clinical-profile scores are means across the videos, on 0–100 and 1–5 scales, respectively.
Figure 6: Model profiles across clinical and video-quality dimensions. Five individual radar plots and an overlay compare task-balanced scores. Scores are averaged equally across twelve tasks and divided by their metric’s maximum to share a 0–1 scale. Clinical profiles summarize valid judgments from the 4,500 generated clips.
Figure 7: Action omission, anatomical distortion, and procedure interruption. Two examples illustrate each failure mode. Red circles, boxes, and arrows highlight the affected regions or the change of scene.
Figure 8: Tool–tissue interaction failures, temporal ghosting, and camera instruction violations. Two examples illustrate each failure mode. Red marks highlight the tool–target region, overlapping silhouettes, or changes in framing despite the fixed-camera prompt.