Skip to main content

Reinforced Multi-Teacher Selection for Knowledge Distillation

Publication ,  Conference
Yuan, F; Shou, L; Pei, J; Lin, W; Gong, M; Fu, Y; Jiang, D
Published in: 35th AAAI Conference on Artificial Intelligence, AAAI 2021
January 1, 2021

In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models in production. As a popular method for model compression, knowledge distillation transfers knowledge from one or multiple large (teacher) models to a small (student) model. When multiple teacher models are available in distillation, the state-of-the-art methods assign a fixed weight to a teacher model in the whole distillation. Furthermore, most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled. We systematically develop a reinforced method to dynamically assign weights to teacher models for different training instances and optimize the performance of student model. Our extensive experimental results on several NLP tasks clearly verify the feasibility and effectiveness of our approach.

Duke Scholars

Published In

35th AAAI Conference on Artificial Intelligence, AAAI 2021

ISBN

9781713835974

Publication Date

January 1, 2021

Volume

16

Start / End Page

14284 / 14291
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Yuan, F., Shou, L., Pei, J., Lin, W., Gong, M., Fu, Y., & Jiang, D. (2021). Reinforced Multi-Teacher Selection for Knowledge Distillation. In 35th AAAI Conference on Artificial Intelligence, AAAI 2021 (Vol. 16, pp. 14284–14291).
Yuan, F., L. Shou, J. Pei, W. Lin, M. Gong, Y. Fu, and D. Jiang. “Reinforced Multi-Teacher Selection for Knowledge Distillation.” In 35th AAAI Conference on Artificial Intelligence, AAAI 2021, 16:14284–91, 2021.
Yuan F, Shou L, Pei J, Lin W, Gong M, Fu Y, et al. Reinforced Multi-Teacher Selection for Knowledge Distillation. In: 35th AAAI Conference on Artificial Intelligence, AAAI 2021. 2021. p. 14284–91.
Yuan, F., et al. “Reinforced Multi-Teacher Selection for Knowledge Distillation.” 35th AAAI Conference on Artificial Intelligence, AAAI 2021, vol. 16, 2021, pp. 14284–91.
Yuan F, Shou L, Pei J, Lin W, Gong M, Fu Y, Jiang D. Reinforced Multi-Teacher Selection for Knowledge Distillation. 35th AAAI Conference on Artificial Intelligence, AAAI 2021. 2021. p. 14284–14291.

Published In

35th AAAI Conference on Artificial Intelligence, AAAI 2021

ISBN

9781713835974

Publication Date

January 1, 2021

Volume

16

Start / End Page

14284 / 14291