Bilateral Evidence Selection and Reasoning for Mammography Vision--Language Models
Organizations: Intelligent Systems Program University of Pittsburgh, Pittsburgh, PA, USA · Department of Radiology University of Pittsburgh, Pittsburgh, PA, USA · Department of Biomedical Informatics and Department of Bioengineering University of Pittsburgh, Pittsburgh, PA, USA
Abstract
Medical vision--language models (VLMs) often need to identify small, subtle lesions within high-resolution images containing extensive normal anatomy and redundant visual information. Visual-token compression improves efficiency, but preserving salient regions alone can discard the context needed for diagnostic interpretation. This context can extend across images: in bilateral mammography, the opposite breast provides a patient-specific anatomical reference. Yet retaining such evidence does not ensure that the model uses it effectively. We propose Bilateral Evidence Selection and Reasoning (BESR), which connects evidence preservation with comparative assessment through a bilateral-guided token selector and a bilateral reward. Under a shared token budget, the selector jointly prioritizes lesion appearance and its local and bilateral context, while the reward encourages assessments that benefit from the preserved comparison. Experiments on a bilateral dataset constructed from CBIS-DDSM and on VinDr-Mammo demonstrate improved evidence retention and diagnostic performance under constrained visual-token budgets. Inference benchmarking further shows reduced prefill latency.
Figures & tables
| (a) Token-selection comparison | ||||||
| Bilateral CBIS-DDSM | VinDr-Mammo | |||||
| Input / selector | BAcc. | F1 | Cov. | BAcc. | F1 | Cov. |
| Full tokens | ||||||
| GT lesion | ||||||
| Budget-matched selection ( ) | ||||||
| Spatial Sampling | ||||||
| Configuration | BAcc. (%) | F1 (%) | Coverage (%) |
| Target only | |||
| + Bilateral enhancement | |||
| + Joint selection |
| Method | Tokens | Prefill | End-to-end | Memory |
| (ms) | (s) | (GiB) | ||
| Full tokens | 1,979 | 183.16 | 1.780 | 18.198 |
| Ours, | 256 | 40.50 | 1.595 | 18.095 |
| Ours, | 128 | 35.86 | 1.576 | 18.087 |
| Ours, | 64 | 27.70 | 1.563 | 18.083 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Class | Train | Val. | Test |
| CBIS-DDSM | Benign | 759 | 60 | 232 |
| Malignant | 672 | 80 | 180 | |
| Total | 1,431 | 140 | 412 | |
| VinDr-Mammo | BI-RADS 1–3 | 507 | 58 | 151 |
| BI-RADS 4–5 | 424 | 39 | 120 | |
| Total | 931 | 97 | 271 |
| Setting | Value | Setting | Value |
| Backbone | Qwen2.5-VL-7B-Instruct | Paired projection | CC/CC or MLO/MLO |
| Pixel cap per image | 802,816 | Total visual budget | 64 / 128 / 256 |
| Matching projection dimension | 64 | Enhancement hidden dimension | 128 |
| Matching temperature | 0.1 | Normalized matching radius | 0.25 |
| Target-anchor allocation | Up to | Target/reference allocation | Adaptive |
| Coverage-neighborhood radius | 1 grid cell | Neighbor credit | 0.25 |
| Setting | Value |
| Initial / KL-reference policy | SFT checkpoint |
| Training examples / epochs | 1,431 / 3 |
| Optimizer updates | 1,074 |
| Prompts per update / group size | 4 / 4 |
| Sampling temperature | 1.0 |
| Maximum response length | 192 tokens |
Radiology VLMs
Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.
See all 128 papers