Proof-of-Concept Prompted Large Language Model for Radiology Procedure Request Routing.

PURPOSE: To measure the accuracy and cost of a proof-of-concept prompted large language model (LLM) to route procedure requests to the appropriate phone number or pager at a single large academic hospital. MATERIALS AND METHODS: At a large academic hospital, existing teams, pager/phone numbers, and schedules were used to create text-based rules for procedure requests. A prompted LLM was created to route procedure requests at specific days and times to the appropriate teams. The prompted LLM was tested on 250 "in-scope" requests (explicitly defined by provided rules) and 25 "out-of-scope" requests using generative pretrained transformer (GPT)-3.5-turbo and GPT-4 models from OpenAI and 4 open-weight models. RESULTS: The prompted LLM correctly routed 96.4% of in-scope and 76.0% of out-of-scope requests using GPT-4, which outperformed all other models (P < .001). All models demonstrated worse performance for requests during evening and weekend hours (P < .001). OpenAI application programming interface costs were approximately $0.03 per request for GPT-4 and $0.0006 per request for GPT-3.5-turbo. CONCLUSIONS: This study demonstrates the accuracy of low-cost prompted LLMs to appropriately route procedure requests in a large academic hospital system. A similar approach may be used to help clinicians navigate a radiology phone tree or as a tool to help reading room coordinators route requests effectively.

Duke Scholars

Author Walter Wiggins Radiology, Neuroradiology

Author Nicholas Befera Radiology, Interventional Radiology

Author Brian Prescott Triana Radiology, Interventional Radiology

Author Christopher John Roth Radiology, Neuroradiology