Improving text generation with student-forcing optimal transport
Publication
, Conference
Li, J; Li, C; Wang, G; Fu, H; Lin, YC; Chen, L; Zhang, Y; Tao, C; Zhang, R; Wang, W; Shen, D; Yang, Q; Carin, L
Published in: EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
January 1, 2020
Neural language models are often trained with maximum likelihood estimation (MLE), where the next word is generated conditioned on the ground-truth word tokens. During testing, however, the model is instead conditioned on previously generated tokens, resulting in what is termed exposure bias. To reduce this gap between training and testing, we propose using optimal transport (OT) to match the sequences generated in these two modes. An extension is further proposed to improve the OT learning, based on the structural and contextual information of the text sequences. The effectiveness of the proposed method is validated on machine translation, text summarization, and text generation tasks.
Duke Scholars
Published In
EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
Publication Date
January 1, 2020
Start / End Page
9144 / 9156
Citation
APA
Chicago
ICMJE
MLA
NLM
Li, J., Li, C., Wang, G., Fu, H., Lin, Y. C., Chen, L., … Carin, L. (2020). Improving text generation with student-forcing optimal transport. In EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference (pp. 9144–9156).
Li, J., C. Li, G. Wang, H. Fu, Y. C. Lin, L. Chen, Y. Zhang, et al. “Improving text generation with student-forcing optimal transport.” In EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 9144–56, 2020.
Li J, Li C, Wang G, Fu H, Lin YC, Chen L, et al. Improving text generation with student-forcing optimal transport. In: EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 2020. p. 9144–56.
Li, J., et al. “Improving text generation with student-forcing optimal transport.” EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 2020, pp. 9144–56.
Li J, Li C, Wang G, Fu H, Lin YC, Chen L, Zhang Y, Tao C, Zhang R, Wang W, Shen D, Yang Q, Carin L. Improving text generation with student-forcing optimal transport. EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 2020. p. 9144–9156.
Published In
EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
Publication Date
January 1, 2020
Start / End Page
9144 / 9156