Research Area: Machine Learning
Medical Visual Question Answering (VQA) is a multimodal task to answer clinical questions about medical images. Existing methods have achieved good performance, but most medical VQA models focus on visual contents while ignoring the influence of textual contents. To address this issue, this paper proposes an Attention-based Multimodal Alignment Model (AMAM) for medical VQA, aiming for an alignment of text-based and image-based attention to enrich the textual features. First, we develop an Image-to-Question (I2Q) attention and a Word-to-Question (W2Q) attention to model the relations of both visual and textual contents to the question. Second, we design a composite loss composed of a classification loss and an Image–Question Complementary (IQC) loss. The IQC loss concentrates on aligning the importance of the questions learned from visual and textual features to emphasize meaningful words in questions and improve the quality of predicted answers. Benefiting from the attention mechanisms and the composite loss, AMAM obtains rich semantic textual information and accurate answers. Finally, due to some data errors and missing labels on the VQA-RAD dataset, we further constructed an enhanced dataset, VQA-RAD, to raise data quality. Experimental results on public datasets show better performance of AMAM compared with the advanced methods.
Keywords:
Attention
Medical
Visual Question Answering
Multimodal Alignment
Recurrent Neural Network
Deep Learning
Machine Learning
Author(s) Name: Haiwei Pan, Shuning He, Kejia Zhang, Bo Qu, Chunling Chen, Kun Shi
Journal name: Knowledge-Based Systems
Conferrence name:
Publisher name: Elsevier
DOI: 10.1016/j.knosys.2022.109763
Volume Information: Volume 255, 14 November 2022, 109763
