A68-18 Accuracy of a Deep Learning Model for Radiographic Pneumonia in Urgent Care Clinics
R Sheth, J R Eve, E Prakash, J H Hart, K Kuttler, P J Haug, J R Carr, C P Langlotz, N C Dean
Abstract
Rationale Community-acquired pneumonia (CAP) is a common outpatient concern; to aid in its detection and management, an electronic clinical decision support tool providing real-time likelihood of pneumonia (ePneumonia) was implemented in 5 Intermountain Health (IH) urgent care clinics. A key component of CAP diagnosis and ePneumonia is the finding of lung opacity on chest imaging. Our group previously developed a deep learning neural network (CheXED) to ascertain radiographic pneumonia in emergency departments and provide data for ePneumonia. While CheXED has been updated for outpatient use, the model�s real-world accuracy has not been assessed. Methods We obtained chest imaging from IH urgent care visits between 4/1/24 and 3/31/25 that were associated with a pneumonia ICD-10 code, resulting in a set of 1011 chest PA and lateral radiographs. CheXED evaluated these images in real-time, providing a result of �positive� or �negative� for radiographic pneumonia. Clinical radiologist interpretations of these chest radiographs were screened to �positive� or �negative� by an internal natural language processing algorithm. Interpretations that were indeterminate or �positive� were then manually categorized. CheXED output was then compared with the radiology interpretations. Of the 1011 images, 33 patients had follow-up chest imaging (radiograph or CT scan) within 3 days. Radiologist interpretation of follow-up imaging was used as a gold standard for the presence of radiographic pneumonia and compared to the CheXED output of initial imaging. Results CheXED findings were consistent with radiologist interpretation in 79% of initial images (n = 802), with 71% �positive� for pneumonia (n = 720) and 8% �negative� for pneumonia (n = 82) (see Table 1). CheXED findings on initial radiographs were consistent with radiologist interpretation of follow-up imaging in 82% of imaging sets (n = 27), with 73% �positive� for pneumonia (n = 24) and 9% �negative� for pneumonia (n = 3). When compared to radiologist interpretation of follow-up imaging, CheXED had 92% sensitivity, 43% specificity, 86% positive predictive value, and 60% negative predictive value. Conclusion CheXED was consistent with radiologist interpretation in 79% of the initial imaging and 82% of the follow-up imaging; this is similar to the interpretation variability rate among board-certified radiologists (Albaum CHEST 1996). Using the radiologist interpretation of follow-up imaging as the gold standard for presence of pneumonia, CheXED had a positive predictive value of 86%. Therefore, a �positive� CheXED result is likely to represent radiographic pneumonia, with a low false positive rate. Overall, CheXED can accurately inform outpatient providers about the presence of radiographic pneumonia.