Skip to main navigation Skip to search Skip to main content

Multiple Objects-Aware Visual Question Generation

  • Jiayuan Xie
  • , Yi Cai*
  • , Qingbao Huang
  • , Tao Wang
  • *Corresponding author for this work
  • South China University of Technology
  • MOE of China
  • Guangxi University
  • King's College London

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

27 Citations (Scopus)

Abstract

Visual question generation task aims to generate meaningful questions about an image according to a target answer. Existing studies mainly focus on merely one object related to the target answer in an image to generate a question. However, a target answer is often related to multiple key objects in an image, which focuses on only one object may mislead its model to generate questions that are only related to partial fragments of the answer. To address this problem, we propose a multi-objects aware generation model to capture all key objects related to an answer and generate the corresponding question. We first introduce a co-attention network to capture the relationship between each object in an image and the answer, and then extract the key objects that are related to the answer. Then, a graph network is introduced to capture the relationships between the key objects and other objects in the image that are not related to the answer, which helps generate questions that involve more visual content. Finally, the learned information from the graph network is fed into a standard decoder module to produce questions. Extensive experiments on the VQA v2.0 dataset show that the proposed model outperforms the state-of-the-art models.

Original languageEnglish
Title of host publicationMM 2021 - Proceedings of the 29th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery, Inc
Pages4546-4554
Number of pages9
ISBN (Electronic)9781450386517
DOIs
Publication statusPublished - 17 Oct 2021
Event29th ACM International Conference on Multimedia, MM 2021 - Virtual, Online, China
Duration: 20 Oct 2021 → 24 Oct 2021

Publication series

NameProceedings of the 29th ACM International Conference on Multimedia
PublisherACM

Conference

Conference29th ACM International Conference on Multimedia, MM 2021
Country/TerritoryChina
CityVirtual, Online
Period20/10/2021 → 24/10/2021

Keywords

  • multimodal representations
  • multiple objects-aware
  • visual question generation

Fingerprint

Dive into the research topics of 'Multiple Objects-Aware Visual Question Generation'. Together they form a unique fingerprint.

Cite this