Impact of Imaging Acquisition and Protocol Variability on Artificial Intelligence Model Performance: A Secondary Analysis of the ASFNR Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models have shown promise in neuroradiology, yet their real-world generalizability remains uncertain, partly due to variability in imaging acquisition and protocols. We aimed to evaluate the impact of data source, scanner manufacturer, scan mode, slice thickness, and the AI models developed by participating teams on AI performance in this secondary analysis of the 2019 American Society of Functional Neuroradiology (ASFNR) AI Competition. MATERIALS AND METHODS: We included 1,177 anonymized noncontrast head CT scans from five institutions. Four teams participated, developing models to detect acute ischemic stroke, intracranial hemorrhage, mass effect, and to assess age-appropriate normality. Generalized estimating equations (GEE) were used to evaluate the effects of the aforementioned variables on model performance, and collinearity diagnostics were applied to exclude redundant variables. RESULTS: Due to collinearity with scanner manufacturer, data source was excluded from the model. Across all tasks, the AI model employed significantly influenced performance. Scanner manufacturer was significantly associated with accuracy in detecting intracranial hemorrhage and acute ischemic stroke but not mass effect or age-based normality. Slice thickness significantly associated with detection of intracranial hemorrhage and mass effect, with thinner slices yielding higher accuracy, but showed no effect on ischemic stroke or normality assessments. Scan mode did not significantly influence performance for any task. CONCLUSION: This secondary analysis demonstrates that imaging acquisition and protocol variability may significantly affect AI model performance. Scanner manufacturer, slice thickness, and the developed AI model were significantly associated with model accuracy, whereas scan mode had no significant impact. Among these, the developed AI model consistently proved most influential, reflecting the importance of training data, model architecture, and preprocessing methods.

Duke Scholars

Author Daniel P. Barboriak Radiology, Neuroradiology