Segmentation-Prior Guided Multimodal Learning for Breast Ultrasound Diagnosis

Southeast University, Nanjing, China

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Deep learning shows promise for breast ultrasound segmentation and classification, yet modality heterogeneity and scarce annotations often limit performance. We propose Segmentation-Prior Guided Multimodal Learning for Breast Ultrasound Diagnosis, a unified framework for joint lesion segmentation and malignancy classification. An optimized SAM-based SAMUS generates multimodal lesion masks as spatial priors. Guided by these masks, GAB extracts lesion-specific features, while a hybrid fusion module integrates cross-modal multi-head attention for semantic alignment with a dynamic weight generator for sample- adaptive modality weighting. Experiments demonstrate consistent gains, achieving Dice scores of 82.13%/80.98%/73.84% (B-mode/SE/CEUS) and 93.62% classification accuracy.