Assessing the Social Skills of Children with Autism Spectrum Disorder via Language-Image Pre-training Models

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that has gained global attention due to its prevalence. Clinical assessment measures rely heavily on manual scoring conducted by specialized physicians. However, this approach exhibits subjectivity and challenges in regions without sufficient medical resources. This study presents paradigms designed to automatically evaluate various aspects of social skills in children with ASD, utilizing our multiview and multimodal behavior system. Moreover, we propose a new pipeline to predict autism-related social skill scores using the language-image pre-training model. Our multimodal behavioral database comprises 12 subjects (511 videos with labeled social skill scores). Finally, we achieve 81.46% accuracy on the paradigm success prediction task and 69.23% accuracy on the social skill ability scoring task. The results demonstrate that the language-image pre-training model can effectively introduce domain knowledge into video assessment tasks.

Duke Scholars

Author Ming Li DKU Faculty