cs.CVOct 29, 2026
SavePathCowork: Learning from real-world pathology practice to build AI copilots
Organizations: Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA · Department of Electrical and Systems Engineering, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA · Department of Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA · Graduate Group in Genomics and Computational Biology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA ·
Abstract
Medical AI learns from the answers doctors record, but rarely from how they interpret the evidence to reach them. Here we introduce PathCowork, a framework that turns pathologists' narrated examinations into supervision for visual grounding and procedures that guide diagnostic AI agents. Synchronized gaze, narration and navigation link visual findings to the process of interpreting them. Across 48 sessions from 16 pathologists, the platform captured 1,028 minutes of slide review. Using these data, we trained PathCowork-1T for server deployment and PathCowork-27B for local deployment, with improved visual grounding. Across five organ-specific evaluations, agents combining this grounding with recorded diagnostic procedures achieved primary task scores averaging 17.9 and 24.1 percentage points higher than Claude Code and Codex, respectively, with shorter processing times and lower inference costs. Compared to the state-of-the-art multiple-instance learning AIs, the PathCowork agent-proposed regions achieved higher prostate-grading scores with limited slide labels. PathCowork can also help construct pathology AI benchmarks where questions derived from the recordings identified difficulties in recognizing viewing context and rejecting inapplicable premises. In summary, PathCowork proposes a new paradigm to scale up the data collection and train pathology AI through transparent reasoning from diagnostic sessions, opening a path towards systems that learn directly from routine clinical practice.
Figures & tables
Figure 1: Legend on page 2 .
Figure 3: Legend on page 4 .
Figure 5: Legend on page 6 .
Figure 7: Legend on page 8 .
Figure 9: Legend on page 10 .
Methods Figure M1: Gaze processing and extraction of attention regions. Each row shows a recorded segment and its aligned narration. Columns show the initial gaze samples, samples retained after velocity- and dispersion-based fixation filtering, and the resulting attention boundaries over the viewed tissue. The side-by-side views allow the extracted regions to be inspected against the tissue and narrated findings, including microvascular proliferation and mitosis in glioblastoma.
Methods Figure M2: Interface for reviewing generated examples. The reviewer displays the original utterance, the generated description, preceding narration and the corresponding image. Controls record approval or rejection and export the review decisions. The screenshot illustrates an unreviewed example.
Figure S1: Additional skin grounding examples for confluent growth and pagetoid spread. Rows show two examples of confluent melanocytic proliferation and one of pagetoid spread. Columns compare the reference annotation with a PathCowork SFT localizer and three frontier models. Reference regions are green and model predictions are red. IoU and Dice are percentages; “Not detected” denotes a negative presence prediction.
Figure S2: Additional skin grounding examples for pigment, junctional anatomy and nests. Rows show pigment incontinence, the dermoepidermal junction and a junctional melanocytic nest. Columns, colors and percentage metrics follow Supplementary Figure S1 .
Figure S3: Detailed agent outcomes across five organs. Rows compare PathCowork (purple), Claude Code (orange) and Codex (grey) on primary and secondary endpoints; shaded panels identify the primary endpoints used in Figure 3e . Colon node detection recall measures the proportion of reference lymph nodes located. In the breast row, Tub., Nuc. and Mit. denote tubule formation, nuclear pleomorphism and mitotic count; G1–G3 denote reference grades. Costs and wall-clock times are per slide; time is the median for colon and the mean for other tasks. Arrows indicate the direction of improvement. Cohorts, model configurations and scoring rules are detailed in Supplementary Note Supplementary Note 2 | Detailed agent outcomes across five organs .
Figure S4: General navigation learned from pathologist behavior. a, PathCowork-skill follows a compiled examination procedure; PathCowork-general predicts viewing actions from a learned policy; stepwise Claude Code or Codex agents generate an LLM response before acting. Icon sizes are schematic. Time and cost are indicative navigation estimates (Supplementary Note Supplementary Note 3 | General navigation learned from pathologist behavior ). b, Six patient-held-out CAMELYON17 units, showing tissue, annotated tumor, successive fields and high-power reads. Other patients from this dataset contributed training data. Scale bars, 5 mm. c, All 12 evaluated units, one review each with the supervised navigator, deterministic coverage rules and a nominal 60-step budget. Entries include the terminal observation; budget-limited runs contain 61. Stopped indicates an unforced stop within budget. Survey measures tissue coverage at survey level or higher. High-power (HP) reads count 0.44-mm fields on lesion over all HP fields. Pieces counts fragments of at least 2 mm 2 visited at 1.33 mm or higher; foci counts annotations of at least 0.02 mm 2 touched at these scales. A dash denotes no eligible focus. Evidence counts retained views overlapping tumor over all retained views. Reverse pans and repeated views are separate counts. Time excludes downstream LLM interpretation.
Figure S5: Examples of questions from three benchmark sources. Top, PathMMU and textbook items. Below, recorded-review items address viewing scale, specimen location, navigation, multiple sections and a false premise; the first two are environmental-sensing subtypes in Figure 5 . Green marks accepted answers and orange marks the named models’ responses. The magnification example retains the source version’s five options and two accepted adjacent bands. Question-only percentages describe source-level Opus 4.8 results.
Figure S6: Coverage of recorded clinical checks by four question sources. For each of 103 reader–case reviews, up to two questions were generated from the recorded narration or a final-impression summary, or retrospectively selected to maximize coverage from the archived PathMMU (210 items) or corrected textbook-derived (160 items) bank. The reference contains 347 selected check occurrences, including similar checks recorded by different readers, across 21 case groups. Coverage requires agreement of two model-based coders on the same check–question match, with verbatim supporting text. Dots show case means after averaging readers; filled diamonds show the equal-case mean and bars the 95% percentile interval from 10,000 case-bootstrap resamples. Hollow diamonds show full-bank coverage without the two-question limit. The textbook bank consists of atlas-derived diagnosis questions. Asterisk: textbook coverage is 0.2% in the post-hoc sensitivity analysis without verbatim-support filtering; the same sensitivity rule was applied to all conditions. The analysis measures textual correspondence between recorded checks and question content.
Figure S7: Data scaling of prostate concept grounding. Left, detection recall, localization intersection over union (IoU) and output parse rate as the number of annotated training WSIs increases, with two training epochs at each size. Right, numbers of training polygons and represented concepts. The smallest set includes all eight concepts; the number of polygons increases from 752 to 4,024.
Figure S8: Detection and localization by concept and training-set size. Recall (left) and localization IoU (right) are plotted against the number of training regions for six prostate architectural patterns: small crowded glands, cribriform architecture, glands out of place, infiltrative growth, sheets, and fused or poorly formed glands. Colors identify concepts; dotted lines show the fitted trends against log-transformed region counts.
Figure S9: Fixed-step and equal-epoch training comparisons. Output parse rate (left) and localization IoU (right) for a fixed budget of 100 optimization steps (red) and two epochs at each dataset size (green). The largest fixed-step run receives approximately 0.28 epochs.
Figure S10: Localization of six prostate architectural concepts. IoU for the adapted PathCowork localizer and Gemini 3.1, GPT-5.4 and Opus 4.7. Concepts are small crowded glands, infiltrative growth, glands out of place, fused or poorly formed glands, cribriform architecture and sheets.
Figure S11: Qualitative grounding comparisons in prostate tissue. Two examples of small crowded glands and two of fused glands are shown with their reference regions and model outputs. Outlines show reference and predicted regions. Values below each field are example-level IoU percentages. FOV, field of view.
Figure S12: Additional fused-gland grounding examples. Three further prostate fields are shown with the same column order as Supplementary Figure S11 . Values are example-level IoU percentages. “Unparseable” denotes an output without valid spatial geometry.
Figure S13: Localization after withholding cribriform supervision. Cribriform IoU for a localizer trained without cribriform examples, the localizer trained with the full concept set, and three frontier baselines. Labels above the bars give IoU percentages.
Figure S14: Cribriform localization with and without direct cribriform supervision. Two prostate fields compare the reference region, PathCowork trained on the full concept set, PathCowork trained with cribriform examples withheld, and three frontier models. Values are example-level IoU percentages. The two fields illustrate variation in transfer to the withheld concept; aggregate results are reported in Supplementary Figure S13 . FOV, field of view.
Figure S15: Development studies of model and deployment choices. a, DEJ localization versus total and active parameter counts, showing the same measurements on two logarithmic axes. Filled markers denote dense models and open markers mixture-of-experts models. b, DEJ-region Dice under bf16 server inference and 8-bit and 4-bit local inference (left); model size and per-image latency for the local configurations (right). c, Kimi K2.6 training cross-entropy over one or two epochs. Thin lines show step-level loss and thick lines its exponential moving average (weight 0.1 on the current step). DEJ, dermoepidermal junction.
| Dataset | Tissue or subspecialty | Source type |
|---|---|---|
| TCGA-COAD | Colorectal pathology | Public |
| TCGA-STAD | Gastric pathology | Public |
| TCGA-BLCA | Bladder pathology | Public |
| TCGA-PRAD | Prostate pathology | Public |
| TCGA-DLBC | Hematopathology | Public |
| TCGA-GBM | Neuropathology | Public |
Table S1: Datasets represented in the recording collection. Image sources include TCGA [ 25 ] , DROID-skin [ 26 , 27 ] and LNCO2 [ 28 ] ; behavioral recordings were acquired through the study platform.
| Field | Representation | Content |
|---|---|---|
| Image | Crop and WSI metadata | Viewed tissue region, slide identifier, coordinates and magnification |
| Narration | Text and timestamps | Transcribed utterance, neighboring speech and its time interval |
| Grounding | Spatial coordinates | Polygon, bounding box or polyline associated with a narrated concept |
| Trajectory | Ordered event records | Sequence of views, positions, magnifications and transitions |
| Revisit | Linked event records | Previous observations of the same WSI region |
| Derived context | Structured text | Normalized description, case summary and interpretation of the narration |
Table S2: Intermediate record fields. Coordinates and timestamps link derived examples to the original slide and recording.
| Analysis | Model or configuration | Role |
|---|---|---|
| Main grounding comparison | Kimi K2.6; Qwen3.6-27B | PathCowork-1T and PathCowork-27B; LoRA rank 32, peak learning rate , batch size 16, two epochs; final checkpoint |
| Dermatopathology skill | PathCowork-1T (Kimi K2.6) | Adapted concept localizer evaluated in Figure 2e,f and used by the agent in Figure 3 |
| Brain agent comparison | PathCowork v4.4 High; Codex GPT-5.6-sol | Same 30 cases; Claude Code also evaluated, with two missing reports counted as incorrect |
| PathCowork-general | Qwen3.5-4B; LoRA rank 32 | Supervised general navigator; 29 decision slots (Note Supplementary Note 3 | General navigation learned from pathologist behavior ) |
| Small-model development | Qwen3.5-4B; LoRA rank 8 | SFT and subsequent appearance-aware RL experiment (Note Supplementary Note 1 | Additional skin grounding and an ancillary adaptation study ) |
| Glioma format and RL recipe | Qwen3.5-4B; LoRA rank 32 | Prompt-format selection and base versus SFT-initialized RL (Note Supplementary Note 8 | Spatial formats and reinforcement learning ) |
Table S3: Model configurations by analysis. Backbone identifiers and training conditions are listed for the individual experiments.
| Task | Source | Evaluation set | Clinical endpoint |
|---|---|---|---|
| Skin | HISTAI [ 38 ] | 20 cases; 69 WSIs | Melanoma versus dysplastic nevus |
| Prostate | PANDA [ 39 ] | 20 slides | Cancer detection and ISUP grade |
| Breast | HISTAI [ 38 ] | 39 cases; grades 1/2/3: 10/14/15 | Nottingham grade and component scores |
| Brain | TCGA-GBM; TCGA-LGG [ 25 ] | 30 cases; 15 per class | GBM versus lower-grade glioma |
| Colorectal lymph nodes | LNCO2 [ 28 ] | 77 slides; 10 positive, 67 negative | Node detection and metastasis classification |
Table S4: Sources of cases used in agent evaluation. All three agents were assigned the same cases within each task. Evaluation cases were unseen during grounding training and skill construction.
| Encoder | CoworkMIL | ABMIL | Difference | ||
|---|---|---|---|---|---|
| UNI2-h | 10 | 0.321 | 0.279 | +0.041 | 0.153646 |
| UNI2-h | 20 | 0.544 | 0.466 | +0.078 | 0.001017 ∗∗ |
| UNI2-h | 30 | 0.620 | 0.552 | +0.067 | 0.029575 ∗ |
| UNI2-h | 40 | 0.690 | 0.660 | +0.030 | 0.230513 |
| UNI2-h | 50 | 0.712 | 0.724 | -0.012 | 0.348810 |
| Virchow2 | 10 | 0.376 | 0.261 | +0.115 | 0.010689 ∗ |
Table S5: Paired comparisons of CoworkMIL with ABMIL. Means and mean paired differences in quadratic-weighted are based on 20 matched training seeds per setting, evaluated on the same 30-slide test set. is the number of training slides. Two-sided exact Wilcoxon signed-rank tests use the archived seed-level scores, stored to four decimal places; none of these contrasts has a zero difference or tied absolute differences. All 15 comparisons are reported; * , ** . Positive differences favor CoworkMIL.
| Pipeline | (mean s.d.) | Mean MAE |
|---|---|---|
| EAGLE, 0.5 m per pixel | 1.225 | |
| EAGLE, 2.0 m per pixel | 1.298 | |
| ABMIL, Virchow2 | 1.347 | |
| CoworkMIL, Virchow2 | 1.178 |
Table S6: EAGLE and MIL pipelines on matched training subsets and test slides. All rows use the same 20 subsets of 20 training slides and the same 30-slide PANDA test cohort. Quadratic-weighted is reported as mean sample standard deviation across subsets; MAE is the mean absolute error in ISUP grade. EAGLE uses frozen CTransPath–CHIEF tile selection [ 42 , 43 ] , frozen Virchow2 class-token features, mean pooling and a fitted logistic-regression classifier. Its indicated resolution applies to both tile selection and re-encoding. The MIL rows use frozen Virchow2 class-token plus mean patch-token features from native level-0 tiles and a trained attention aggregator; CoworkMIL additionally uses grounding-derived training supervision.
| Category | Content | Questions |
|---|---|---|
| Rejection of an inapplicable premise | Finding or premise verification | 128 |
| Multi-slide reasoning | Evidence across fields or slides | 62 |
| Goal-directed navigation | Next region to inspect | 59 |
| Environmental sensing | Viewing scale | 63 |
| Tissue location | 59 | |
| Total | 371 |
Table S7: Composition of the retained recorded-review question bank. Counts refer to unique retained question identifiers in the two source-bank exports. Environmental sensing comprises two subtypes. The separate 100-question source comparisons and 148-question paired image-input analysis use the subsets described in Methods.