Expressed emotion (EE), a measure of the family's emotional climate, is a fundamental measure in caregiving research. A core dimension of EE is the level of criticism expressed by the caregiver to the care recipient, with a high level of criticism a marker of significant distress in the household. The Five-Minute Speech Sample (FMSS), the most commonly used brief measure of EE, requires time-consuming manual processing and scoring by a highly trained expert. In this study, we used natural language processing and supervised machine learning techniques to develop a fully automated framework to evaluate caregiver criticism level based on the verbatim transcript of the FMSS. The success of the machine learning algorithm was established by demonstrating that the classification of maternal caregivers as high versus low EE was consistent with the classification of these 298 maternal caregivers of adult children with schizophrenia using standard manual coding procedures, with area under the receiver operating characteristic curve (AUROC) of 0.76. Evidence of construct validity was established by demonstrating that maternal caregivers of adults with schizophrenia, who were classified as having a high level of criticism had higher levels of caregiver burden, reported that their child had more psychiatric symptoms and behaviors and perceived that their child had greater control over these symptoms and behaviors. Additionally, maternal caregivers who had high levels of criticism reported having a poorer quality of relationship with their child with schizophrenia than maternal caregivers low on criticism. Rapid measurement of criticism facilitates the incorporation of this dimension into research across a broad range of caregiving contexts. (PsycInfo Database Record (c) 2021 APA, all rights reserved).