Integrating task specific information into pretrained language models for low resource fine tuning

Conference Paper

Pretrained Language Models (PLMs) have improved the performance of natural language understanding in recent years. Such models are pretrained on large corpora, which encode the general prior knowledge of natural languages but are agnostic to information characteristic of downstream tasks. This often results in overfitting when fine-tuned with low resource datasets where task-specific information is limited. In this paper, we integrate label information as a task-specific prior into the self-attention component of pretrained BERT models. Experiments on several benchmarks and real-word datasets suggest that the proposed approach can largely improve the performance of pretrained models when fine-tuning with small datasets. The code repository is released in https://github.com/RayWangWR/BERT_label_embedding.

Duke Authors

Cited Authors

  • Wang, R; Si, S; Wang, G; Zhang, L; Carin, L; Henao, R

Published Date

  • January 1, 2020

Published In

  • Findings of the Association for Computational Linguistics Findings of Acl: Emnlp 2020

Start / End Page

  • 3181 - 3186

International Standard Book Number 13 (ISBN-13)

  • 9781952148903

Citation Source

  • Scopus