Research Area:  Machine Learning

Abstract:

Skin cancer is one of the most prevalent malignancies in the world. Deep learning-based methods have been successfully used for skin disease diagnosis and achieved great recognition performance, most of which relied on dermoscopic images alone. Existing multi-model methods of skin lesion diagnosis have the following two shortcomings: 1) They mainly focus on learning complementary information while ignoring the correlation between clinical and dermoscopic images. 2) The feature extractor is not optimized by imposing constraints, which may result in limited expression of the extracted features. To address these issues, this study proposes a new method, named multi-modal bilinear fusion with hybrid attention mechanism (MBF-HA) for multi-modal skin lesion classification. Specifically, MBF-HA introduced a common representation learning framework to learn the correlated features by exploring the shared characteristics between two modalities. Moreover, MBF-HA uses the hybrid attention-based reconstruction module which encourages the feature extractor to detect and localize lesion regions on each modality, thus, enhancing the discriminative power of the output feature representation. We perform comprehensive experiments on a well-established multi-modal and multi-label skin disease dataset: a 7-point Checklist database, MBF-HA achieves an average accuracy of 76.3% in the multi-classification tasks and 76.0% in the diagnostic task. The experimental results show that MBF-HA outperforms known related works and is expected to help physicians make more precise clinical diagnoses.

Keywords:  

Author(s) Name:  Yun Wei & Lin Ji

Publisher name:  Springer

DOI:  10.1007/s11042-023-18027-5

Volume Information:  Volume 83, Pages 65221-65247, (2024)

Paper Link:   https://link.springer.com/article/10.1007/s11042-023-18027-5