Skip to main content
Journal cover image

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

Publication ,  Journal Article
Stahl, CC; Jung, SA; Rosser, AA; Kraut, AS; Schnapp, BH; Westergaard, M; Hamedani, AG; Minter, RM; Greenberg, JA
Published in: Am J Surg
February 2021

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

Am J Surg

DOI

EISSN

1879-1883

Publication Date

February 2021

Volume

221

Issue

2

Start / End Page

369 / 375

Location

United States

Related Subject Headings

  • Surgery
  • Surgeons
  • Specialties, Surgical
  • Professional Autonomy
  • Natural Language Processing
  • Models, Educational
  • Machine Learning
  • Internship and Residency
  • Humans
  • Formative Feedback
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Stahl, C. C., Jung, S. A., Rosser, A. A., Kraut, A. S., Schnapp, B. H., Westergaard, M., … Greenberg, J. A. (2021). Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy. Am J Surg, 221(2), 369–375. https://doi.org/10.1016/j.amjsurg.2020.11.044
Stahl, Christopher C., Sarah A. Jung, Alexandra A. Rosser, Aaron S. Kraut, Benjamin H. Schnapp, Mary Westergaard, Azita G. Hamedani, Rebecca M. Minter, and Jacob A. Greenberg. “Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.Am J Surg 221, no. 2 (February 2021): 369–75. https://doi.org/10.1016/j.amjsurg.2020.11.044.
Stahl CC, Jung SA, Rosser AA, Kraut AS, Schnapp BH, Westergaard M, et al. Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy. Am J Surg. 2021 Feb;221(2):369–75.
Stahl, Christopher C., et al. “Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.Am J Surg, vol. 221, no. 2, Feb. 2021, pp. 369–75. Pubmed, doi:10.1016/j.amjsurg.2020.11.044.
Stahl CC, Jung SA, Rosser AA, Kraut AS, Schnapp BH, Westergaard M, Hamedani AG, Minter RM, Greenberg JA. Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy. Am J Surg. 2021 Feb;221(2):369–375.
Journal cover image

Published In

Am J Surg

DOI

EISSN

1879-1883

Publication Date

February 2021

Volume

221

Issue

2

Start / End Page

369 / 375

Location

United States

Related Subject Headings

  • Surgery
  • Surgeons
  • Specialties, Surgical
  • Professional Autonomy
  • Natural Language Processing
  • Models, Educational
  • Machine Learning
  • Internship and Residency
  • Humans
  • Formative Feedback