Research Area:  Machine Learning

Abstract:

Multimodal summarization has drawn much attention due to the rapid growth of multimedia data. The output of the current multimodal summarization systems is usually represented in texts. However, we have found through experiments that multimodal output can significantly improve user satisfaction for informativeness of summaries. In this paper, we propose a novel task, multimodal summarization with multimodal output (MSMO). To handle this task, we first collect a large-scale dataset for MSMO research. We then propose a multimodal attention model to jointly generate text and select the most relevant image from the multimodal input. Finally, to evaluate multimodal outputs, we construct a novel multimodal automatic evaluation (MMAE) method which considers both intramodality salience and intermodality relevance. The experimental results show the effectiveness of MMAE.

Keywords:  
multimodal summarization
texts
large-scale dataset
automatic evaluation
modality salience
inter-modality relevance

Author(s) Name:  Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, Chengqing Zong

Publisher name:  Aclanthology

DOI:  10.18653/v1/D18-1448

Volume Information:  Volume: 2018

Paper Link:   https://aclanthology.org/D18-1448