Research Area:  Machine Learning

Abstract:

Labelling image-sentence is expensive and some unsupervised image captioning methods show promising results on caption generation. However, the generated captions are not very relevant to images due to the excessive dependence on the corpus. In order to overcome that drawback, we focus on the correspondence between image and sentence to construct an image caption with better mapping relation. In this paper, we present a novel triple sequence generative adversarial net including an image generator, a discriminator, and a sentence generator. The image generator is used to generate the image regions for words. Meanwhile, the sentence corpus guides the sentence generator based on the generated image regions. The discriminator judges the relevance between the words in the sentence and the generated image regions. In the experiments, we use a large number of unpaired images and sentences to train our model on the unsupervised and unpaired setting. The experimental results demonstrate that our method achieves significant improvements as compared to all baselines.

Keywords:  
labelling image
caption generation
corpus
discriminator
sentence generator

Author(s) Name:  Yucheng Zhou, Wei Tao, Wenqiang Zhang

Publisher name:  IEEE

DOI:  https://doi.org/10.1109/ICASSP39728.2021.9414335

Volume Information:  -

Paper Link:   https://ieeexplore.ieee.org/abstract/document/9414335