Toward Cross-Rubric Surgical Skill Assessment via Joint Embedding Prediction
Organizations: School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA · Department of Urology, Mayo Clinic, Phoenix, AZ, USA
Abstract
Expert review of surgical videos is essential for surgical training but labor-intensive, motivating AI-assisted skill assessment to provide feedback at scale. Although surgical assessment rubrics may use different language to describe similar skills, existing automated methods commonly predict predefined skill levels or numerical scores tied to fixed rubrics, limiting their ability to transfer across semantically equivalent assessment criteria. We propose Predictive Rubric Alignment (PRA), a joint embedding prediction approach that scores surgical videos using natural-language descriptions of skill levels. Rather than matching level probabilities, PRA predicts a rater-weighted combination of skill-level description embeddings. Adjacent description features define a score-parameterized path, and the predicted embedding is projected onto this path to read the score. We introduce two variants: PRA-item, which uses learned queries for predefined assessment items, and PRA-cond, which generates queries from textual item names to enable assessment of unseen items without retraining. On AIxSuture, a dataset of expert-rated suturing videos, PRA outperforms published baselines in skill classification. We further evaluate zero-shot transfer to the smaller surgical skill assessment benchmark JIGSAWS. Trained on only two AIxSuture rubric items, PRA-cond assesses unseen JIGSAWS items without further training, yielding positive score correlations and greater Grad-CAM overlap with instruments and suture than a description-matching baseline. These results support predictive rubric alignment as a promising direction for flexible and explainable automated surgical skill assessment.
Figures & tables
| AIxSuture item | JIGSAWS item |
|---|---|
| Respect for tissue | Respect for tissue |
| Time and motion | Time and motion |
| Instrument handling | – |
| Suture technique | Suture/needle handling |
| Flow of operation & forward planning | Flow of operation |
| Knowledge of specific procedure | – |
| Item-specific | 3-class skill classification | Rubric scores | ||||||
| Method | parameters | Acc | macro-F1 | F1-N | F1-I | F1-P | GRS | item MAE |
| I3D [ 16 ] | – | 0.746 0.063 | 0.692 0.029 | 0.857 | 0.403 | 0.751 | – | – |
| Video Swin-T [ 16 ] | – | 0.732 0.033 | 0.716 0.043 | 0.837 | 0.538 | 0.772 | – | – |
| Regression | yes | 0.768 0.037 | 0.749 0.047 | 0.843 0.084 | 0.568 0.066 | 0.837 0.020 | 0.882 0.010 | 0.458 0.016 |
| CE-8, path readout | yes | 0.783 0.031 | 0.755 0.038 | 0.833 0.034 | 0.581 0.090 | 0.850 0.024 | 0.846 0.016 | 0.539 0.019 |
| CE-8, prob. readout | yes | 0.746 0.051 | 0.712 0.069 | 0.798 0.048 | 0.496 0.139 | 0.841 0.023 | 0.866 0.019 | 0.461 0.015 |
| with the GRS | IoU with instruments and suture | |||
| Suturing | Knot Tying | Suturing | Knot Tying | |
| Random | 0.001 0.161 | 0.002 0.170 | 0.057 | 0.060 |
| Trial duration | 0.342 | 0.723 | – | – |
| BCE, path readout | 0.247 0.018 | 0.310 0.041 | 0.024 0.019 | 0.021 0.008 |
| BCE, prob. readout | 0.319 0.029 | 0.351 0.167 | 0.088 0.077 | 0.061 0.048 |
| PRA-cond | 0.272 0.144 | 0.252 0.126 | 0.185 0.122 | 0.140 0.101 |
Rubric-Based Evaluation
Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.
See all 208 papers