Skip to main content
Journal cover image

elBERto: Self-supervised commonsense learning for question answering

Publication ,  Journal Article
Zhan, X; Li, Y; Dong, X; Liang, X; Hu, Z; Carin, L
Published in: Knowledge-Based Systems
December 22, 2022

Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoning using these evidence. In this paper, we propose a Self-supervised Bidirectional Encoder Representation Learning of Commonsense (elBERto) framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. The tasks include a novel Contrastive Relation Learning task to encourage the model to distinguish between logically contrastive contexts, a new Jigsaw Puzzle task that requires the model to infer logical chains in long contexts, and three classic self-supervised learning(SSL) tasks to maintain pre-trained models’ language encoding ability. On the representative WIQA, CosmosQA, and ReClor datasets, elBERto outperforms all other methods using the same backbones and the same training set, including those utilizing explicit graph reasoning and external knowledge retrieval. Moreover, elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help, indicating that it successfully learns commonsense and is able to leverage it when given dynamic context.

Duke Scholars

Published In

Knowledge-Based Systems

DOI

ISSN

0950-7051

Publication Date

December 22, 2022

Volume

258

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4611 Machine learning
  • 4605 Data management and data science
  • 4602 Artificial intelligence
  • 17 Psychology and Cognitive Sciences
  • 15 Commerce, Management, Tourism and Services
  • 08 Information and Computing Sciences
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Zhan, X., Li, Y., Dong, X., Liang, X., Hu, Z., & Carin, L. (2022). elBERto: Self-supervised commonsense learning for question answering. Knowledge-Based Systems, 258. https://doi.org/10.1016/j.knosys.2022.109964
Zhan, X., Y. Li, X. Dong, X. Liang, Z. Hu, and L. Carin. “elBERto: Self-supervised commonsense learning for question answering.” Knowledge-Based Systems 258 (December 22, 2022). https://doi.org/10.1016/j.knosys.2022.109964.
Zhan X, Li Y, Dong X, Liang X, Hu Z, Carin L. elBERto: Self-supervised commonsense learning for question answering. Knowledge-Based Systems. 2022 Dec 22;258.
Zhan, X., et al. “elBERto: Self-supervised commonsense learning for question answering.” Knowledge-Based Systems, vol. 258, Dec. 2022. Scopus, doi:10.1016/j.knosys.2022.109964.
Zhan X, Li Y, Dong X, Liang X, Hu Z, Carin L. elBERto: Self-supervised commonsense learning for question answering. Knowledge-Based Systems. 2022 Dec 22;258.
Journal cover image

Published In

Knowledge-Based Systems

DOI

ISSN

0950-7051

Publication Date

December 22, 2022

Volume

258

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4611 Machine learning
  • 4605 Data management and data science
  • 4602 Artificial intelligence
  • 17 Psychology and Cognitive Sciences
  • 15 Commerce, Management, Tourism and Services
  • 08 Information and Computing Sciences