cs.CVMay 6, 2025

Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks

Authors: Haonan Wang, Jiaji Mao, Lehan Wang, Qixiang Zhang, Marawan Elbatel, Yi Qin, Huijun Hu, Baoxun Li, +6 more

Abstract

Accurate delineation of tumors and surrounding organs-at-risk is essential for radiotherapy, surgery and treatment response assessment, yet remains time-consuming and expertise-intensive. Existing artificial intelligence systems often require manual spatial prompts or task-specific retraining, while generic class labels provide limited semantic grounding for heterogeneous disease targets. Here we present SyRe, a promptable segmentation foundation model based on Synergistic vision-language Reinforcement. SyRe strengthens bidirectional interaction between visual and linguistic representations to improve semantically grounded spatial understanding. To support large-scale training, we introduce the Color Region Description strategy and construct SyReData, comprising 20 million image-mask-description triplets across 9 modalities and 229 segmentation tasks. Training with diversified prompt forms further enables open-ended prompting, invalid-prompt rejection and flexible switching between single- and multi-target analysis. SyRe achieves accurate text-prompted segmentation across diverse clinical scenarios, with particularly strong performance on disease-related targets. Across 28 unseen external datasets, including 20 cancer types and multinational in-house cohorts, SyRe generalizes robustly under real-world distribution shifts. SyRe-generated masks also preserve clinically relevant quantitative information in pathology and yield radiomics features that stratify survival and improve prognostic modeling across five retrospective CT and MRI tumor cohorts. Finally, clinician-in-the-loop refinement enables efficient case-level correction when greater precision is required. These results establish SyRe as a generalizable foundation for scalable quantitative oncology and clinician-guided segmentation refinement.

Explore similar work

CardsList
May 15, 2026cs.CV

Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning

Recent progress in deep learning has significantly advanced CT image analysis, particularly for segmentation tasks. However, these advances are largely confined to image-level pattern recognition, with most methods lacking explicit anatomical or contextual reasoning. Large vision-language models introduce linguistic context into image analysis, yet most approaches typically focus on a single task, which is insufficient for clinical workflow analysis that requires multiple fine-grained types of analysis, such as anatomy detection and segmentation. In this paper, we propose a unified autoregressive framework that integrates language-guided visual reasoning into CT interpretation. Our method introduces task-routing tokens that trigger detection and segmentation heads conditioned on the hidden states of a large vision-language model, enabling coherent generation of visual outputs (e.g., masks and bounding boxes) and textual reasonings. To progressively enhance localisation accuracy and semantic clarity, we further design a "closer-look" mechanism that allows the model to perform progressive coarse-to-fine visits to regions of interest under refined fields of view. To support model training and evaluation, we curated a new multimodal CT dataset containing pixel-wise masks, bounding boxes, spatial prompts, and structured descriptions for visual objects constructed through an AI-assisted annotation process with human verification. Experiments on public benchmarks demonstrate consistent improvements over the SoTA, achieving up to 1.0% Dice on BTCV and 1.7% Dice on MosMed+, while additionally providing appearance reasoning outputs. The code and dataset will be available.
Yuyuan Liu, Can Peng, Yingyu Yang +3
Medical Vision-Language ModelsComputed TomographyExplanationsAutoregressive Model
AskPDF
May 18, 2026cs.CV

Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis

Medical image segmentation is more clinically valuable when it supports diagnosis rather than merely producing lesion masks. However, diagnostically relevant lesion cues are often subtle and localized, while existing models may be distracted by background tissues, acoustic artifacts, and irrelevant visual correlations. To address this problem, we propose Rad-VLSM, a two-stage cross-modal framework for semantics-assisted lesion focusing, robust segmentation, and visually grounded diagnosis. In the first stage, a BLIP-2-based vision-language alignment module identifies lesion-related candidate regions under semantic guidance and converts them into box prompts. In the second stage, these prompts are fed into a SAM-based multitask network, where a multi-candidate region aggregation strategy improves prompt stability and guides lesion segmentation. The predicted masks are then used as spatial priors for diagnosis, and a visual-radiomics fusion head integrates lesion-aware visual features with selected radiomics descriptors. By using semantic information for localization rather than direct prediction, Rad-VLSM reduces text-to-diagnosis dependence and grounds diagnosis in lesion-level evidence. Experiments on a private clinical breast ultrasound dataset and public benchmarks show that Rad-VLSM achieves strong segmentation and diagnostic performance with favorable generalization.
Fengyi Zhang, Xujie Zeng, Mohan Liu +2
Lesion SegmentationMedical Image SegmentationMedical Vision-Language ModelsMultimodal Clinical DataVisible Image FusionCross-Modal
AskPDF
Aug 5, 2026cs.CV

Text-Guided Refinement of Multi-sequence Glioma Subregion Segmentation with a Vision-Language Foundation Model

Background: Accurate glioma subregion delineation is important for radiotherapy planning and longitudinal monitoring, but manual contour correction is time-consuming. Models such as nnU-Net may generalize imperfectly and lack clinician-directed text correction. Purpose: We investigated adapting a three-dimensional (3D) vision-language foundation model for text-guided brain tumor segmentation refinement. Methods: We developed a lightweight VoxTell-based framework. Pretrained VoxTell generated initial masks. Oracle prompts derived from segmentation errors encoded target, action, location, imaging evidence, edit size, and preservation constraints. Frozen Qwen/VoxTell prompt embeddings were injected through trainable projections into its multiscale decoder conditioning; other weights remained frozen. Training, validation, and testing used 901, 100, and 250 BraTS-GLI cases. Cross-dataset transfer was evaluated on 100 meningioma, metastasis, pediatric tumor, and UPENN-GBM cases. Results: On the internal test set using post-contrast T1-weighted input, correct instructions improved subregion Dice similarity coefficient (DSC; enhancing tumor, edema, and necrotic/non-enhancing core) from 0.774±0.1580.774\pm0.158 to 0.796±0.1370.796\pm0.137. They outperformed blank prompts (0.762±0.1550.762\pm0.155; Holm-adjusted p<0.001p<0.001, dz=0.71d_z=0.71) and contradictory prompts (0.770±0.1630.770\pm0.163; p<0.001p<0.001, dz=0.48d_z=0.48). In cross-dataset testing, correct instructions improved DSC from 0.527±0.2870.527\pm0.287 to 0.550±0.2780.550\pm0.278 and outperformed contradictory instructions (0.504±0.2750.504\pm0.275; p<0.001p<0.001, dz=0.43d_z=0.43). Conclusion: A 3D vision-language foundation model can perform instruction-guided refinement of glioma subregion segmentations. Sensitivity to correct, blank, and contradictory prompts suggests text-dependent contour editing rather than nonspecific post-processing, supporting further evaluation as a clinician-in-the-loop tool.
Zach Eidex, Yu-nong Lin, Mojtaba Safari +4
Brain Tumor SegmentationAdaptive Shape RefinementVision-Language Reasoning
AskPDF