Skip to main content

Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference

Publication ,  Journal Article
Shen, K; Wu, L; Tang, S; Xu, F; Long, B; Zhuang, Y; Pei, J
Published in: IEEE Transactions on Pattern Analysis and Machine Intelligence
January 1, 2024

The visual question generation (VQG) task aims to generate human-like questions from an image and potentially other side information (e.g. answer type). Previous works on VQG fall in two aspects: i) They suffer from problem, which leads to the failure of generating referential and meaningful questions from an image. ii) They fail to among the visual objects in an image and also overlook potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. Concretely, we aim to ask the right visual questions with Double Hints - textual answers and visual regions of interests, which could effectively mitigate the existing one-to-many mapping issue. Particularly, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework, which first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilizes a graph-to-sequence model to generate the questions with double hints. Experimental results demonstrate the priority of our proposed method.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI

EISSN

1939-3539

ISSN

0162-8828

Publication Date

January 1, 2024

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4611 Machine learning
  • 4603 Computer vision and multimedia computation
  • 0906 Electrical and Electronic Engineering
  • 0806 Information Systems
  • 0801 Artificial Intelligence and Image Processing
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Shen, K., Wu, L., Tang, S., Xu, F., Long, B., Zhuang, Y., & Pei, J. (2024). Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2024.3425222
Shen, K., L. Wu, S. Tang, F. Xu, B. Long, Y. Zhuang, and J. Pei. “Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference.” IEEE Transactions on Pattern Analysis and Machine Intelligence, January 1, 2024. https://doi.org/10.1109/TPAMI.2024.3425222.
Shen K, Wu L, Tang S, Xu F, Long B, Zhuang Y, et al. Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024 Jan 1;
Shen, K., et al. “Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference.” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jan. 2024. Scopus, doi:10.1109/TPAMI.2024.3425222.
Shen K, Wu L, Tang S, Xu F, Long B, Zhuang Y, Pei J. Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024 Jan 1;

Published In

IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI

EISSN

1939-3539

ISSN

0162-8828

Publication Date

January 1, 2024

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4611 Machine learning
  • 4603 Computer vision and multimedia computation
  • 0906 Electrical and Electronic Engineering
  • 0806 Information Systems
  • 0801 Artificial Intelligence and Image Processing