Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes
Organizations: Department of Radiation Oncology, Tata Medical Center, Kolkata, India · Department of Pathology, Tata Medical Center, Kolkata, India
Abstract
Decision models are a new class of language model that, instead of generating text, choose one option per question from a fixed list and return a probability for every option. They have not yet been evaluated on clinical text. We evaluated two open-weights decision models, Nimble 9B and Tev1 4B, for extracting 25 fields from 1,113 oral cavity cancer pathology reports from a single centre, against a prospectively maintained clinical database. We compared them with two generative models: Qwen3.5 9B, run locally, and GLM-5.3-flash, run through a cloud API. Tev1 reached a mean field accuracy of 0.917 (95% CI 0.913 to 0.920), the same as Qwen3.5 9B (0.918; p = 0.26), and was four times faster (15 s against 61 s per report). Nimble reached 0.862 and GLM-5.3-flash 0.986. Both decision models matched GLM-5.3-flash on simple dichotomous categories. They were weak on measurements binned into categories, on fields whose correct answer was not reported, and on some complex decisions (e.g. involved margins). We found that the main cause of the inferior performance of Nimble's came from asking 25 questions in one request: with two questions per request, its accuracy improved. Pooled over fields, the returned probabilities were well calibrated (expected calibration error 0.022 and 0.026), but they did not flag systematic errors within single fields. A simulated strategy of sending answers with a probability below 0.94 to GLM-5.3-flash raised Tev1's accuracy to 0.982, with 38% of answers escalated. Decision models can extract explicitly stated findings from pathology reports quickly on local hardware and return useful probabilities. They should be validated in the configuration in which they will be used, restricted to the fields in which they are reliable, and combined with other methods for numeric values.
Figures & tables
| Field | Categories |
| Tumour size (greatest dimension) | 20 mm; 20 to 40 mm; 40 mm |
| Depth of invasion | 5 mm; 5 to 10 mm; 10 mm |
| Tumour thickness | 5 mm; 5 to 10 mm; 10 mm |
| Number of positive nodes | 0; 1; 2 or more; no neck dissection |
| Number of nodes examined | 1 to 17; 18 or more; no neck dissection |
| Largest involved node | 30 mm; 30 to 60 mm; 60 mm; not applicable |
| Field and label | (%) |
| Grade | |
| Moderately differentiated | 648 (58.2%) |
| Poorly differentiated | 274 (24.6%) |
| Well differentiated | 111 (10.0%) |
| Not stated | 80 (7.2%) |
| PNI | |
| Accuracy by field group | Reports | ||||||||
| Model | Type | MFA (95% CI) | Primary tumour | Nodal | Staging | Binned numeric | Mean kappa | fully correct | Time (s) |
| GLM-5.3-flash | Generative, cloud | 0.986 (0.984–0.987) | 0.984 | 0.983 | 0.998 | 0.985 | 0.951 | 73.3% | 23.6 |
| Qwen3.5 9B | Generative, local | 0.918 (0.916–0.921) | 0.887 | 0.971 | 0.953 | 0.971 | 0.822 | 5.7% | 60.8 |
| Tev1 4B | Decision, local | 0.917 (0.913–0.920) | 0.905 | 0.927 | 0.951 | 0.868 | 0.810 | 10.0% | 15.0 |
| Nimble 9B | Decision, local | 0.862 (0.858–0.866) | 0.834 | 0.962 | 0.769 | 0.871 | 0.740 | 1.8% | 44.3 |
| Finding | Tev1 4B | Nimble 9B | Qwen3.5 9B | GLM-5.3-flash | |
| PNI present | 464 | 1.000 / 1.000 | 1.000 / 0.998 | 1.000 / 0.997 | 1.000 / 0.998 |
| LVI present | 336 | 0.997 / 1.000 | 1.000 / 0.997 | 1.000 / 0.997 | 1.000 / 0.997 |
| WPOI-5 present | 127 | 1.000 / 0.999 | 1.000 / 0.995 | 1.000 / 1.000 | 1.000 / 1.000 |
| Cortical bone invasion | 239 | 0.983 / 0.994 | 0.987 / 0.999 | 0.983 / 0.994 | 0.996 / 1.000 |
| Masticator space involved | 63 | 0.952 / 0.995 | 1.000 / 0.964 | 0.968 / 0.990 | 1.000 / 0.998 |
| Skin involved | 54 | 1.000 / 0.983 | 0.981 / 0.990 | 1.000 / 0.985 | 1.000 / 0.995 |
| Rule | Reference | GLM-5.3-flash | Qwen3.5 9B | Tev1 4B | Nimble 9B |
| Neck dissection side vs right/left node status | 3 | 0 | 4 | 42 | 11 |
| Positive-node count vs right/left node status | 6 | 2 | 17 | 13 | 50 |
| Positive-node count vs pN | 4 | 8 | 10 | 23 | 67 |
| ENE with no positive nodes | 0 | 0 | 0 | 1 | 34 |
| Largest node size vs node count | 0 | 0 | 0 | 5 | 47 |
| “No neck dissection” vs other node fields | 1 | 1 | 4 | 0 | 5 |
| Model | Field | 25 questions (main run) | 2 questions, original order | Reversed | Shuffled | Without Tx, T0, Tis |
| Nimble 9B | pT | 0.490 | 0.850 | 0.840 (92.5%) | 0.830 (95.0%) | 0.855 (92.5%) |
| Nimble 9B | Margin | 0.785 | 0.850 | 0.795 (94.5%) | 0.855 (98.5%) | 0.860 (99.0%) |
| Tev1 4B | pT | 0.990 | 0.990 | 1.000 (99.0%) | 0.995 (99.5%) | 0.995 (98.5%) |
| Tev1 4B | Margin | 0.895 | 0.900 | 0.895 (95.5%) | 0.875 (94.5%) | 0.885 (98.5%) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Rule | Applies when | Counted as a contradiction when |
|---|---|---|
| Neck dissection side vs right/left node status | Neck dissection side is left, right or bilateral | A side that was not dissected has a node status (Yes or No), or a dissected side is “Not applicable” |
| Positive-node count vs right/left node status | Positive-node count and both side statuses are recorded | 0 positive nodes but either side is “Yes”, or 1 or more positive nodes but neither side is “Yes” |
| Positive-node count vs pN | pN (N0 to N3b) and positive-node count are recorded | pN0 with positive nodes; pN1 or higher with 0 positive nodes; pN1 with 2 or more positive nodes; pN2b or pN2c with 1 positive node |
| ENE with no positive nodes | ENE is “Yes” and positive-node count is recorded | 0 positive nodes, or no neck dissection |
| Largest node size vs node count | Largest involved node size and positive-node count are recorded | A node size with 0 positive nodes or no neck dissection, or “Not applicable” with positive nodes |
| “No neck dissection” vs other node fields | Positive nodes or nodes examined is “No neck dissection” | Only one of these two fields is “No neck dissection”, or a neck dissection side is recorded |
| Field | Question | Options ( label : description) |
|---|---|---|
| Primary subsite | What is the anatomic subsite of the primary tumor? If there are several tumor foci, give the site of the dominant (largest) tumor. | Oral Tongue : Mobile (anterior two-thirds) tongue. Base of tongue is not oral cavity. Floor of Mouth : Floor of mouth. Buccal Mucosa : Buccal mucosa / cheek. Lower Alveolus : Lower alveolus, lower gingiva or mandibular alveolar ridge. Upper Alveolus : Upper alveolus, upper gingiva or maxillary alveolar ridge. Hard Palate : Hard palate. Retromolar trigone : Retromolar trigone. Lip : Lip. Overlapping : A large tumor extending across several sites without one stated as the primary. Not reported : Site not stated, primary tumor absent, or site is outside the oral cavity (e.g. base of tongue). |
| Laterality | What is the side of the primary tumor? Use the side given for the primary tumor specimen, not the side of the neck dissection. If the tumor crosses the midline, give the primary side if reported. | Right : Primary tumor is on the right. Left : Primary tumor is on the left. Central : Report states the tumor is central / midline. Bilateral : Report states the tumor is bilateral. Not reported : Laterality of the primary tumor is not mentioned. |
| Focality | How many separate invasive tumor foci are in the primary specimen? Carcinoma in situ, dysplasia and lymph node deposits do not count. A satellite tumor counts only if reported as a distinct primary focus. | 1 : One focus, or the report does not mention more than one focus. 2 : Two separate foci, or ‘multifocal’ without a number. 3 : Three or more separate foci. |
| Histological type | What is the histologic type of the carcinoma? Ignore the grade. | SCC conventional : Squamous cell carcinoma, keratinizing or non-keratinizing, any grade, with no variant named. verrucous : Verrucous carcinoma. basaloid : Basaloid squamous cell carcinoma. papillary : Papillary squamous cell carcinoma. spindle cell : Spindle cell (sarcomatoid) carcinoma. adenosquamous : Adenosquamous carcinoma. other / not reported : Any other histology, or histology not stated. |
| Grade | What is the grade of differentiation of the carcinoma? When two grades are given (e.g. moderately to poorly differentiated), choose the worse one. | Well differentiated : Well differentiated (grade 1). Moderately differentiated : Moderately differentiated (grade 2), and the worse grade is not poorly. Poorly differentiated : Poorly differentiated (grade 3), including ‘moderately to poorly’. Not reported : Differentiation is not reported, or there is no primary tumor. |
| PNI | Is perineural invasion (PNI) present in the primary tumor? | Present : Report states perineural invasion, perineural involvement or nerve invasion is identified. Absent : Report states perineural invasion is absent, not identified or negative. Indeterminate : Report states perineural invasion is indeterminate. Not reported : Perineural invasion is not mentioned at all. |
Clinical Information Extraction
Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.
See all 115 papers