JANUS: Anatomy-Conditioned Gating for Robust CT Triage Under Distribution Shift

We present Aegis, a joint-embedding predictive architecture for breast cancer detection and density assessment in mammography. We train three Vision Transformer variants (Small/Base/Large) using self-supervised joint-embedding predictive architecture (JEPA) pre-training on 71,103 studies from 14 clinical sites, followed by supervised fine-tuning with progressive resolution scaling up to 2048x1536. On a curated 785-study test set, our largest model achieves area under the receiver operating characteristic curve (AUC) 0.949 for breast cancer triage with 93% sensitivity and 75% specificity at the optimal operating point. An ensemble combining our model with a U.S. Food and Drug Administration-cleared baseline further improves discrimination to 0.952 AUC. For breast density classification, the model achieves 0.953 AUC for binary (dense vs. non-dense) classification and 62.6% exact accuracy across four Breast Imaging Reporting and Data System (BI-RADS) categories, with 98.8% adjacent accuracy comparable to reported human inter-reader agreement. External validation on the public VinDr-Mammo dataset provides evidence of cross-population transfer under a different reference standard, with the largest model achieving 0.871 AUC for triage in a zero-shot setting.

A vision-language model can answer a question about a medical image fluently and confidently while barely using the image, leaning instead on language priors. In medicine this is the failure that matters most, because the answer looks trustworthy and is not, and the only protection is a confidence score reliable enough to tell the system when to abstain. We ask a deployment question rather than an accuracy one: how much imaging work a model can safely handle alone, and which confidence signal makes that possible. We evaluate seven confidence estimators across five open-weight LVLMs and three medical visual-question-answering datasets spanning broad clinical imaging, radiology, and pathology, with every probe trained only on natural images and applied without adaptation. Recast as bounded selective prediction (automate a case only when confidence clears a threshold, defer the rest), the comparison is cautionary. The standard metrics are poor guides: discrimination barely separates the methods, and the weak calibration of a cheap self-report is cheaply removed by off-domain temperature scaling without changing deployable yield. What distinguishes a usable estimator is the high-confidence region a clinician acts on: the weakest baselines are confidently wrong on 41 to 45 percent of their errors against 1 to 4 percent for the best probe, and no estimator is reliably best across domains or models. Safe handoff is governed at two levels: base-model competence sets a ceiling, so a well-calibrated score recovers roughly a third of radiology cases at a 20 percent error tolerance but almost none of pathology; the confidence layer then decides how much of that ceiling is reachable. The usable role today is calibrated triage, not autonomy: automate the cases a calibrated score marks safe, route the rest to a clinician. We release all outputs, correctness judgments, and confidence scores, with code.

Deep learning models for medical image analysis typically apply a fixed amount of computation to every input, regardless of case difficulty. Anatomy-guided dual-stream architectures have been shown to improve diagnostic performance, but they evaluate both streams unconditionally, even on cases a single stream could already resolve confidently. We propose SecondOpinion, a framework in which a fast primary stream processes every case, while a second, anatomy-guided stream is invoked only when GateKeeper, a gating mechanism trained explicitly as a binary correctness classifier, judges that the primary stream's prediction needs additional scrutiny, much as a clinician might seek a second opinion on a difficult case. When activated, the two streams are combined through a lightweight cross-attention fusion module. We evaluate SecondOpinion on a unified five-class chest X-ray dataset and a pelvic fracture dataset, the latter including a held-out, harder subset of fractures that are invisible on X-ray but confirmed via CT. SecondOpinion matches or exceeds prior state-of-the-art performance on both tasks, while activating its anatomy-guided stream on only 9.23% of chest X-ray cases, rising to 24.12% on visible fractures and 45.71% on invisible fractures, an activation rate that tracks task difficulty directly. These results suggest that supervising a gating signal toward correctness, rather than relying on unsupervised confidence, allows a model to allocate anatomical reasoning where it is actually needed.