Skip to main content

Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study.

Publication ,  Journal Article
Corey, KM; Kashyap, S; Lorenzi, E; Lagoo-Deenadayalan, SA; Heller, K; Whalen, K; Balu, S; Heflin, MT; McDonald, SR; Swaminathan, M; Sendak, M
Published in: PLoS medicine
November 2018

Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk.A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis.Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

PLoS medicine

DOI

EISSN

1549-1676

ISSN

1549-1277

Publication Date

November 2018

Volume

15

Issue

11

Start / End Page

e1002701

Related Subject Headings

  • Young Adult
  • Surgical Procedures, Operative
  • Risk Factors
  • Risk Assessment
  • Retrospective Studies
  • Reproducibility of Results
  • Postoperative Complications
  • Middle Aged
  • Male
  • Machine Learning
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Corey, K. M., Kashyap, S., Lorenzi, E., Lagoo-Deenadayalan, S. A., Heller, K., Whalen, K., … Sendak, M. (2018). Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study. PLoS Medicine, 15(11), e1002701. https://doi.org/10.1371/journal.pmed.1002701
Corey, Kristin M., Sehj Kashyap, Elizabeth Lorenzi, Sandhya A. Lagoo-Deenadayalan, Katherine Heller, Krista Whalen, Suresh Balu, et al. “Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study.PLoS Medicine 15, no. 11 (November 2018): e1002701. https://doi.org/10.1371/journal.pmed.1002701.
Corey KM, Kashyap S, Lorenzi E, Lagoo-Deenadayalan SA, Heller K, Whalen K, Balu S, Heflin MT, McDonald SR, Swaminathan M, Sendak M. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study. PLoS medicine. 2018 Nov;15(11):e1002701.

Published In

PLoS medicine

DOI

EISSN

1549-1676

ISSN

1549-1277

Publication Date

November 2018

Volume

15

Issue

11

Start / End Page

e1002701

Related Subject Headings

  • Young Adult
  • Surgical Procedures, Operative
  • Risk Factors
  • Risk Assessment
  • Retrospective Studies
  • Reproducibility of Results
  • Postoperative Complications
  • Middle Aged
  • Male
  • Machine Learning