Skip to main content

Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition

Publication ,  Journal Article
Yu, C; Zhang, D; Zou, W; Li, M
Published in: IEEE Transactions on Affective Computing
January 1, 2024

One potential way to enhance the performance of facial expression recognition (FER) is to augment the training set by increasing the number of samples. By incorporating multiple FER datasets, deep learning models can extract more discriminative features. However, the inconsistent labeling criteria and subjective biases found in annotated FER datasets can significantly hinder the recognition accuracy of deep learning models when handling mixed datasets. Effectively perform joint training on multiple datasets remains a challenging task. In this study, we propose a joint training method for training an FER model using multiple FER datasets. Our method consists of four steps: (1) selecting a subset from the additional dataset, (2) generating pseudo-continuous labels for the target dataset, (3) refining the labels of different datasets using continuous label mapping and discrete label relabeling according to the labeling criteria of the target dataset, and (4) jointly training the model using multi-task learning. We conduct joint training experiments on two popular in-the-wild FER benchmark databases, RAF-DB and CAER-S, while utilizing the AffectNet dataset as an additional dataset. The experimental results demonstrate that our proposed method outperforms the direct merging of different FER datasets into a single training set and achieves state-of-the-art performance on RAF-DB and CAER-S with accuracies of 92.24% and 94.57%, respectively.

Duke Scholars

Published In

IEEE Transactions on Affective Computing

DOI

EISSN

1949-3045

Publication Date

January 1, 2024

Related Subject Headings

  • 4608 Human-centred computing
  • 4603 Computer vision and multimedia computation
  • 4602 Artificial intelligence
  • 1702 Cognitive Sciences
  • 0806 Information Systems
  • 0801 Artificial Intelligence and Image Processing
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Yu, C., Zhang, D., Zou, W., & Li, M. (2024). Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2024.3382618
Yu, C., D. Zhang, W. Zou, and M. Li. “Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition.” IEEE Transactions on Affective Computing, January 1, 2024. https://doi.org/10.1109/TAFFC.2024.3382618.
Yu C, Zhang D, Zou W, Li M. Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition. IEEE Transactions on Affective Computing. 2024 Jan 1;
Yu, C., et al. “Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition.” IEEE Transactions on Affective Computing, Jan. 2024. Scopus, doi:10.1109/TAFFC.2024.3382618.
Yu C, Zhang D, Zou W, Li M. Joint Training on Multiple Datasets With Inconsistent Labeling Criteria for Facial Expression Recognition. IEEE Transactions on Affective Computing. 2024 Jan 1;

Published In

IEEE Transactions on Affective Computing

DOI

EISSN

1949-3045

Publication Date

January 1, 2024

Related Subject Headings

  • 4608 Human-centred computing
  • 4603 Computer vision and multimedia computation
  • 4602 Artificial intelligence
  • 1702 Cognitive Sciences
  • 0806 Information Systems
  • 0801 Artificial Intelligence and Image Processing