Research Area: Machine Learning
Although video summarization has achieved tremendous success benefiting from Recurrent Neural Networks (RNN), RNN-based methods neglect the global dependencies and multi-hop relationships among video frames, which limits the performance. Transformer is an effective model to deal with this problem, and surpasses RNN-based methods in several sequence modeling tasks, such as machine translation, video captioning, etc. Motivated by the great success of transformer and the natural structure of video (frame-shot-video), a hierarchical transformer is developed for video summarization, which can capture the dependencies among frame and shots, and summarize the video by exploiting the scene information formed by shots. Furthermore, we argue that both the audio and visual information are essential for the video summarization task. To integrate the two kinds of information, they are encoded in a two-stream scheme, and a multimodal fusion mechanism is developed based on the hierarchical transformer. In this paper, the proposed method is denoted as Hierarchical Multimodal Transformer (HMT).
Keywords:
Recurrent Neural Networks
video captioning
Hierarchical Multimodal Transformer
Multimodal
Audio and Visual
Author(s) Name: Bin Zhao, Gong, Maoguo, Li, Xuelong
Journal name: Neurocomputing
Conferrence name:
Publisher name: Elsevier
DOI: 10.1016/j.neucom.2021.10.039
Volume Information: Volume 468
