Five Models for Five Modalities: Open-Vocabulary Segmentation in Medical Imaging

We present a multimodal approach to open-vocabulary segmentation in medical imaging by training five modality-specific models using a unified architecture based on the SAT model. Each model is tailored to a specific imaging modality—CT, MRI, Ultrasound, Microscopy, and PET, while maintaining architectural consistency to ensure comparability and generalizability. To address the challenge of limited data availability, particularly in modalities like Ultrasound and Microscopy, we implement distinct sampling strategies designed to maximize anatomical and pathological diversity across training cases. We aim to evaluate the effectiveness of open-vocabulary segmentation across diverse medical imaging modalities using consistent text prompts and unified label representations. For CT, MRI, and Ultrasound, performance is reported using Dice Similarity Coefficient (DSC) and Normalized Surface Dice (NSD), while for Microscopy and PET, we follow challenge-specific guidelines and report F1 scores. On the official test set, the models achieved: CT (DSC: 0.2884, NSD: 0.2114), MRI (DSC: 0.1644, NSD: 0.1474), Microscopy (F1: 0.4502), and PET (F1: 0.0728). These results demonstrate the viability of modality-specific training within an open-vocabulary framework and provide a foundation for further improvements.

Duke Scholars

Author Joseph Yuan-Chieh Lo Radiology

Author William Paul Segars Radiology