PathCowork: Learning from real-world pathology practice to build AI copilots
Authors: Sheng Wang, Jialu Yao, Maria V. Rossi, Shunsuke Koga, Esther Baranov, MacLean P. Nasrallah, Songhao Li, Jonathan Xu, Wei Du, Cindy McGrath, Timothy Chao, Anne Wang, Zhaohai Yang, Daniel Akuma, Selemon M. Walle, Olga Pozdnyakova, Edward B. Lee, Zahra Alipour, Annapurna Saksena, Rashmi Tondon, Nikolina Dioufa, David E. Elder, George J. Netto, Xiaowei Xu, Zhi Huang
Organizations: Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA · Department of Electrical and Systems Engineering, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA · Department of Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA · Graduate Group in Genomics and Computational Biology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA · Department of Biostatistics, Epidemiology & Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
Medical AI learns from the answers doctors record, but rarely from how they interpret the evidence to reach them. Here we introduce PathCowork, a framework that turns pathologists' narrated examinations into supervision for visual grounding and procedures that guide diagnostic AI agents. Synchronized gaze, narration and navigation link visual findings to the process of interpreting them. Across 48 sessions from 16 pathologists, the platform captured 1,028 minutes of slide review. Using these data, we trained PathCowork-1T for server deployment and PathCowork-27B for local deployment, with improved visual grounding. Across five organ-specific evaluations, agents combining this grounding with recorded diagnostic procedures achieved primary task scores averaging 17.9 and 24.1 percentage points higher than Claude Code and Codex, respectively, with shorter processing times and lower inference costs. Compared to the state-of-the-art multiple-instance learning AIs, the PathCowork agent-proposed regions achieved higher prostate-grading scores with limited slide labels. PathCowork can also help construct pathology AI benchmarks where questions derived from the recordings identified difficulties in recognizing viewing context and rejecting inapplicable premises. In summary, PathCowork proposes a new paradigm to scale up the data collection and train pathology AI through transparent reasoning from diagnostic sessions, opening a path towards systems that learn directly from routine clinical practice.
Figures & tables
Figure 1: Legend on page 2 .
Figure 3: Legend on page 4 .
Figure 5: Legend on page 6 .
Figure 7: Legend on page 8 .
Figure 9: Legend on page 10 .
Methods Figure M1: Gaze processing and extraction of attention regions. Each row shows a recorded segment and its aligned narration. Columns show the initial gaze samples, samples retained after velocity- and dispersion-based fixation filtering, and the resulting attention boundaries over the viewed tissue. The side-by-side views allow the extracted regions to be inspected against the tissue and narrated findings, including microvascular proliferation and mitosis in glioblastoma.
Methods Figure M2: Interface for reviewing generated examples. The reviewer displays the original utterance, the generated description, preceding narration and the corresponding image. Controls record approval or rejection and export the review decisions. The screenshot illustrates an unreviewed example.
Figure S1: Additional skin grounding examples for confluent growth and pagetoid spread. Rows show two examples of confluent melanocytic proliferation and one of pagetoid spread. Columns compare the reference annotation with a PathCowork SFT localizer and three frontier models. Reference regions are green and model predictions are red. IoU and Dice are percentages; “Not detected” denotes a negative presence prediction.
Figure S2: Additional skin grounding examples for pigment, junctional anatomy and nests. Rows show pigment incontinence, the dermoepidermal junction and a junctional melanocytic nest. Columns, colors and percentage metrics follow Supplementary Figure S1 .
Figure S3: Detailed agent outcomes across five organs. Rows compare PathCowork (purple), Claude Code (orange) and Codex (grey) on primary and secondary endpoints; shaded panels identify the primary endpoints used in Figure 3e . Colon node detection recall measures the proportion of reference lymph nodes located. In the breast row, Tub., Nuc. and Mit. denote tubule formation, nuclear pleomorphism and mitotic count; G1–G3 denote reference grades. Costs and wall-clock times are per slide; time is the median for colon and the mean for other tasks. Arrows indicate the direction of improvement. Cohorts, model configurations and scoring rules are detailed in Supplementary Note Supplementary Note 2 | Detailed agent outcomes across five organs .
Figure S4: General navigation learned from pathologist behavior. a, PathCowork-skill follows a compiled examination procedure; PathCowork-general predicts viewing actions from a learned policy; stepwise Claude Code or Codex agents generate an LLM response before acting. Icon sizes are schematic. Time and cost are indicative navigation estimates (Supplementary Note Supplementary Note 3 | General navigation learned from pathologist behavior ). b, Six patient-held-out CAMELYON17 units, showing tissue, annotated tumor, successive fields and high-power reads. Other patients from this dataset contributed training data. Scale bars, 5 mm. c, All 12 evaluated units, one review each with the supervised navigator, deterministic coverage rules and a nominal 60-step budget. Entries include the terminal observation; budget-limited runs contain 61. Stopped indicates an unforced stop within budget. Survey measures tissue coverage at survey level or higher. High-power (HP) reads count 0.44-mm fields on lesion over all HP fields. Pieces counts fragments of at least 2 mm 2 visited at 1.33 mm or higher; foci counts annotations of at least 0.02 mm 2 touched at these scales. A dash denotes no eligible focus. Evidence counts retained views overlapping tumor over all retained views. Reverse pans and repeated views are separate counts. Time excludes downstream LLM interpretation.
Figure S5: Examples of questions from three benchmark sources. Top, PathMMU and textbook items. Below, recorded-review items address viewing scale, specimen location, navigation, multiple sections and a false premise; the first two are environmental-sensing subtypes in Figure 5 . Green marks accepted answers and orange marks the named models’ responses. The magnification example retains the source version’s five options and two accepted adjacent bands. Question-only percentages describe source-level Opus 4.8 results.
Figure S6: Coverage of recorded clinical checks by four question sources. For each of 103 reader–case reviews, up to two questions were generated from the recorded narration or a final-impression summary, or retrospectively selected to maximize coverage from the archived PathMMU (210 items) or corrected textbook-derived (160 items) bank. The reference contains 347 selected check occurrences, including similar checks recorded by different readers, across 21 case groups. Coverage requires agreement of two model-based coders on the same check–question match, with verbatim supporting text. Dots show case means after averaging readers; filled diamonds show the equal-case mean and bars the 95% percentile interval from 10,000 case-bootstrap resamples. Hollow diamonds show full-bank coverage without the two-question limit. The textbook bank consists of atlas-derived diagnosis questions. Asterisk: textbook coverage is 0.2% in the post-hoc sensitivity analysis without verbatim-support filtering; the same sensitivity rule was applied to all conditions. The analysis measures textual correspondence between recorded checks and question content.
Figure S7: Data scaling of prostate concept grounding. Left, detection recall, localization intersection over union (IoU) and output parse rate as the number of annotated training WSIs increases, with two training epochs at each size. Right, numbers of training polygons and represented concepts. The smallest set includes all eight concepts; the number of polygons increases from 752 to 4,024.
Figure S8: Detection and localization by concept and training-set size. Recall (left) and localization IoU (right) are plotted against the number of training regions for six prostate architectural patterns: small crowded glands, cribriform architecture, glands out of place, infiltrative growth, sheets, and fused or poorly formed glands. Colors identify concepts; dotted lines show the fitted trends against log-transformed region counts.
Figure S9: Fixed-step and equal-epoch training comparisons. Output parse rate (left) and localization IoU (right) for a fixed budget of 100 optimization steps (red) and two epochs at each dataset size (green). The largest fixed-step run receives approximately 0.28 epochs.
Figure S10: Localization of six prostate architectural concepts. IoU for the adapted PathCowork localizer and Gemini 3.1, GPT-5.4 and Opus 4.7. Concepts are small crowded glands, infiltrative growth, glands out of place, fused or poorly formed glands, cribriform architecture and sheets.
Figure S11: Qualitative grounding comparisons in prostate tissue. Two examples of small crowded glands and two of fused glands are shown with their reference regions and model outputs. Outlines show reference and predicted regions. Values below each field are example-level IoU percentages. FOV, field of view.
Figure S12: Additional fused-gland grounding examples. Three further prostate fields are shown with the same column order as Supplementary Figure S11 . Values are example-level IoU percentages. “Unparseable” denotes an output without valid spatial geometry.
Figure S13: Localization after withholding cribriform supervision. Cribriform IoU for a localizer trained without cribriform examples, the localizer trained with the full concept set, and three frontier baselines. Labels above the bars give IoU percentages.
Figure S14: Cribriform localization with and without direct cribriform supervision. Two prostate fields compare the reference region, PathCowork trained on the full concept set, PathCowork trained with cribriform examples withheld, and three frontier models. Values are example-level IoU percentages. The two fields illustrate variation in transfer to the withheld concept; aggregate results are reported in Supplementary Figure S13 . FOV, field of view.
Figure S15: Development studies of model and deployment choices. a, DEJ localization versus total and active parameter counts, showing the same measurements on two logarithmic axes. Filled markers denote dense models and open markers mixture-of-experts models. b, DEJ-region Dice under bf16 server inference and 8-bit and 4-bit local inference (left); model size and per-image latency for the local configurations (right). c, Kimi K2.6 training cross-entropy over one or two epochs. Thin lines show step-level loss and thick lines its exponential moving average (weight 0.1 on the current step). DEJ, dermoepidermal junction.
Dataset
Tissue or subspecialty
Source type
TCGA-COAD
Colorectal pathology
Public
TCGA-STAD
Gastric pathology
Public
TCGA-BLCA
Bladder pathology
Public
TCGA-PRAD
Prostate pathology
Public
TCGA-DLBC
Hematopathology
Public
TCGA-GBM
Neuropathology
Public
Table S1: Datasets represented in the recording collection. Image sources include TCGA [ 25 ] , DROID-skin [ 26 , 27 ] and LNCO2 [ 28 ] ; behavioral recordings were acquired through the study platform.
Field
Representation
Content
Image
Crop and WSI metadata
Viewed tissue region, slide identifier, coordinates and magnification
Narration
Text and timestamps
Transcribed utterance, neighboring speech and its time interval
Grounding
Spatial coordinates
Polygon, bounding box or polyline associated with a narrated concept
Trajectory
Ordered event records
Sequence of views, positions, magnifications and transitions
Revisit
Linked event records
Previous observations of the same WSI region
Derived context
Structured text
Normalized description, case summary and interpretation of the narration
Table S2: Intermediate record fields. Coordinates and timestamps link derived examples to the original slide and recording.
Analysis
Model or configuration
Role
Main grounding comparison
Kimi K2.6; Qwen3.6-27B
PathCowork-1T and PathCowork-27B; LoRA rank 32, peak learning rate 10−4 , batch size 16, two epochs; final checkpoint
Dermatopathology skill
PathCowork-1T (Kimi K2.6)
Adapted concept localizer evaluated in Figure 2e,f and used by the agent in Figure 3
Brain agent comparison
PathCowork v4.4 High; Codex GPT-5.6-sol
Same 30 cases; Claude Code also evaluated, with two missing reports counted as incorrect
PathCowork-general
Qwen3.5-4B; LoRA rank 32
Supervised general navigator; 29 decision slots (Note Supplementary Note 3 | General navigation learned from pathologist behavior )
Small-model development
Qwen3.5-4B; LoRA rank 8
SFT and subsequent appearance-aware RL experiment (Note Supplementary Note 1 | Additional skin grounding and an ancillary adaptation study )
Glioma format and RL recipe
Qwen3.5-4B; LoRA rank 32
Prompt-format selection and base versus SFT-initialized RL (Note Supplementary Note 8 | Spatial formats and reinforcement learning )
Table S3: Model configurations by analysis. Backbone identifiers and training conditions are listed for the individual experiments.
Task
Source
Evaluation set
Clinical endpoint
Skin
HISTAI [ 38 ]
20 cases; 69 WSIs
Melanoma versus dysplastic nevus
Prostate
PANDA [ 39 ]
20 slides
Cancer detection and ISUP grade
Breast
HISTAI [ 38 ]
39 cases; grades 1/2/3: 10/14/15
Nottingham grade and component scores
Brain
TCGA-GBM; TCGA-LGG [ 25 ]
30 cases; 15 per class
GBM versus lower-grade glioma
Colorectal lymph nodes
LNCO2 [ 28 ]
77 slides; 10 positive, 67 negative
Node detection and metastasis classification
Table S4: Sources of cases used in agent evaluation. All three agents were assigned the same cases within each task. Evaluation cases were unseen during grounding training and skill construction.
Encoder
N
CoworkMIL
ABMIL
Difference
P
UNI2-h
10
0.321
0.279
+0.041
0.153646
UNI2-h
20
0.544
0.466
+0.078
0.001017 ∗∗
UNI2-h
30
0.620
0.552
+0.067
0.029575 ∗
UNI2-h
40
0.690
0.660
+0.030
0.230513
UNI2-h
50
0.712
0.724
-0.012
0.348810
Virchow2
10
0.376
0.261
+0.115
0.010689 ∗
Table S5: Paired comparisons of CoworkMIL with ABMIL. Means and mean paired differences in quadratic-weighted κ are based on 20 matched training seeds per setting, evaluated on the same 30-slide test set. N is the number of training slides. Two-sided exact Wilcoxon signed-rank tests use the archived seed-level scores, stored to four decimal places; none of these contrasts has a zero difference or tied absolute differences. All 15 comparisons are reported; * P<0.05 , ** P<0.01 . Positive differences favor CoworkMIL.
Pipeline
κ (mean ± s.d.)
Mean MAE
EAGLE, 0.5 μ m per pixel
0.430±0.160
1.225
EAGLE, 2.0 μ m per pixel
0.435±0.108
1.298
ABMIL, Virchow2
0.385±0.133
1.347
CoworkMIL, Virchow2
0.486±0.120
1.178
Table S6: EAGLE and MIL pipelines on matched training subsets and test slides. All rows use the same 20 subsets of 20 training slides and the same 30-slide PANDA test cohort. Quadratic-weighted κ is reported as mean ± sample standard deviation across subsets; MAE is the mean absolute error in ISUP grade. EAGLE uses frozen CTransPath–CHIEF tile selection [ 42 , 43 ] , frozen Virchow2 class-token features, mean pooling and a fitted logistic-regression classifier. Its indicated resolution applies to both tile selection and re-encoding. The MIL rows use frozen Virchow2 class-token plus mean patch-token features from native level-0 tiles and a trained attention aggregator; CoworkMIL additionally uses grounding-derived training supervision.
Category
Content
Questions
Rejection of an inapplicable premise
Finding or premise verification
128
Multi-slide reasoning
Evidence across fields or slides
62
Goal-directed navigation
Next region to inspect
59
Environmental sensing
Viewing scale
63
Tissue location
59
Total
371
Table S7: Composition of the retained recorded-review question bank. Counts refer to unique retained question identifiers in the two source-bank exports. Environmental sensing comprises two subtypes. The separate 100-question source comparisons and 148-question paired image-input analysis use the subsets described in Methods.
Pathology is the cornerstone of modern medicine, where accurate decision-making relies heavily on evidence-based practices. While artificial intelligence (AI) has the potential to transform clinical workflows, the intersection of AI and evidence-based medicine remains under-explored, with primitive attempts restricted to text-only general medicine. In this work, we present PathPocket, a multimodal AI agentic co-pilot designed specifically for evidence grounded pathology. We construct the most comprehensive pathology evidence corpus to date, encompassing approximately 110,472 public and authorized documents structured across a rigorous hierarchy of evidence from clinical guideline to expert opinion. From this meticulously graded foundation, we build a large-scale multimodal pathology hypergraph containing over 4.55 million entities and 7.10 million relations. Serving as a robust knowledge engine, this hypergraph provides traceable evidence for a collaborative multi-agent reasoning framework integrating input understanding, evidence retrieval, filtering, and diagnosis generation. This enables PathPocket to seamlessly resolve a wide spectrum of clinical tasks, ranging from text-only queries to complex multimodal diagnostics involving region-of-interest (ROI) and gigapixel whole-slide images (WSIs). We rigorously evaluate the system on a multidimensional benchmark of over 200,000 real-world cases, where it significantly outperforms existing state-of-the-arts. Crucially, extensive user studies demonstrate that PathPocket substantially improves the diagnostic accuracy and confidence of pathologists. By directly grounding pathology interpretations in verifiable literature, PathPocket offers a practical and scalable solution for the future of evidence grounded computational pathology.
Zhe Xu, Zhengyu Zhang, Zhiyuan Cai +12
Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China · Department of Pathology, Nanfang Hospital, Southern Medical University, Guangzhou, China · Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, China +7
Foundation models with visual question answering capabilities for digital pathology are emerging. Such unprecedented technology requires independent benchmarking to assess its potential in assisting pathologists in routine diagnostics. We created DALPHIN, the first multicentric open benchmark for pathology AI copilots, comprising 1236 images from 300 cases, spanning 130 rare to common diagnoses, 6 countries, and 14 subspecialties. The DALPHIN design and dataset are introduced alongside a human performance benchmark of 31 pathologists from 10 countries with varying expertise. We report results for two general-purpose (GPT-5, Gemini 2.5 Pro) and one pathology-specific copilot (PathChat+) for sequential and independent answer generation. We observed no statistically significant difference from expert-level performance in four of six tasks for PathChat, 2/6 tasks for Gemini, and 1/6 tasks for GPT. DALPHIN is publicly released with sequestered, indirectly accessible ground truth to foster robust and enduring benchmarking. Data, methods, and the evaluation platform are accessible through dalphin.grand-challenge.org.
Training AI models for computational pathology currently requires access to expensive whole-slide-image datasets, GPU infrastructure, deep expertise in machine learning, and substantial engineering effort. We present CellDX AI Autopilot, a platform that lets users -- from pathologists with no ML background to ML practitioners running many parallel experiments -- train, evaluate, and deploy whole-slide image classifiers through natural language interaction with an AI agent. The platform provides a structured set of agent skills that guide the user through dataset curation, automated hyperparameter tuning, multi-strategy model comparison, and human-in-the-loop deployment, all on a pre-built dataset of over 32,000 cases and 66,000 H&E-stained whole-slide images with pre-extracted features. We describe the agent skill architecture, the underlying Multiple Instance Learning (MIL) training framework supporting four classification strategies, and an iterative pairwise hyperparameter search (grid or seeded random) that reduces tuning cost by over 30x compared to exhaustive search. CellDX AI Autopilot is, to our knowledge, the first system to expose pathology-specialized agent skills and a pathology-specialized training platform to general-purpose AI agents (e.g. any LLM-based agent runtime), delivering end-to-end automated model training without requiring the agent itself to be domain-specific. The platform addresses both the ML-expertise bottleneck that limits adoption in diagnostic pathology and the engineering bottleneck that limits how many experiments a researcher can run cost-effectively.