Imaging Pearls ❯ Neuroradiology ❯ Sinuses
|
-- OR -- |
|
- Summary
The performance of an artificial intelligence clinical decision support solution for intracranial hemorrhage detection was low in a low prevalence environment; falsely flagged studies led to increased radiologist interpretation time, potentially reducing system efficiency.
Key Points
■ An artificial intelligence (AI) clinical decision support solution for intracranial hemorrhage detection yielded a positive predictive value of 21.1% in a low prevalence (2.70%) environment.
■ Falsely flagged studies by the AI solution led to lengthened radiologist read times and system inefficiencies (median read time increased 1 minute 14 seconds [P < .001] for examinations with false-positive findings and 1 minute 5 seconds [P = .04] for examinations with false-negative findings).
■ Factoring in prevalence of a condition in varying clinical settings and the impact that falsely flagged studies will have on system efficiency may aid institutional decision-making for use of an AI solution and help set clearer expectations for end users.
Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time
Andrew James Del Gaizo, et al.
Radiology: Artificial Intelligence 2024; 6(5):e240067 - The diagnostic performance of an artificial intelligence (AI) clinical decision support solution for acute intracranial hemorrhage (ICH) detection was assessed in a large teleradiology practice. The impact on radiologist read times and system efficiency was also quantified. A total of 61 704 consecutive noncontrast head CT examinations were retrospectively evaluated. System performance was calculated along with mean and median read times for CT studies obtained before (baseline, pre-AI period; August 2021 to May 2022) and after (post-AI period; January 2023 to February 2024) AI implementation. The AI solution had a sensitivity of 75.6%, specificity of 92.1%, accuracy of 91.7%, prevalence of 2.70%, and positive predictive value of 21.1%. Of the 56 745 post-AI CT scans with no bleed identified by a radiologist, examinations falsely flagged as suspected ICH by the AI solution (n = 4464) took an average of 9 minutes 40 seconds (median, 8 minutes 7 seconds) to interpret as compared with 8 minutes 25 seconds (median, 6 minutes 48 seconds) for unremarkable CT scans before AI (n = 49 007) (P < .001) and 8 minutes 38 seconds (median, 6 minutes 53 seconds) after AI when ICH was not suspected by the AI solution (n = 52 281) (P < .001).
Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time
Andrew James Del Gaizo, et al.
Radiology: Artificial Intelligence 2024; 6(5):e240067 - The AI solution had a sensitivity of 75.6%, specificity of 92.1%, accuracy of 91.7%, prevalence of 2.70%, and positive predictive value of 21.1%. Of the 56 745 post-AI CT scans with no bleed identified by a radiologist, examinations falsely flagged as suspected ICH by the AI solution (n = 4464) took an average of 9 minutes 40 seconds (median, 8 minutes 7 seconds) to interpret as compared with 8 minutes 25 seconds (median, 6 minutes 48 seconds) for unremarkable CT scans before AI (n = 49 007) (P < .001) and 8 minutes 38 seconds (median, 6 minutes 53 seconds) after AI when ICH was not suspected by the AI solution (n = 52 281) (P < .001). CT scans with no bleed identified by the AI but reported as positive for ICH by the radiologist (n = 384) took an average of 14 minutes 23 seconds (median, 13 minutes 35 seconds) to interpret as compared with 13 minutes 34 seconds (median, 12 minutes 30 seconds) for CT scans correctly reported as a bleed by the AI (n = 1192) (P = .04). With lengthened read times for falsely flagged examinations, system inefficiencies may outweigh the potential benefits of using the tool in a high volume, low prevalence environment.
Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time
Andrew James Del Gaizo, et al.
Radiology: Artificial Intelligence 2024; 6(5):e240067 - “CT scans with no bleed identified by the AI but reported as positive for ICH by the radiologist (n = 384) took an average of 14 minutes 23 seconds (median, 13 minutes 35 seconds) to interpret as compared with 13 minutes 34 seconds (median, 12 minutes 30 seconds) for CT scans correctly reported as a bleed by the AI (n = 1192) (P = .04). With lengthened read times for falsely flagged examinations, system inefficiencies may outweigh the potential benefits of using the tool in a high volume, low prevalence environment.”
Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time
Andrew James Del Gaizo, et al.
Radiology: Artificial Intelligence 2024; 6(5):e240067 - “In conclusion, use of an AI tool for ICH detection in our teleradiology practice yielded reduced sensitivity and specificity compared with the published literature. However, a low prevalence of ICH in our patients contributed to a substantially lower positive predictive value. Noncontrast head CT examinations falsely flagged by an AI solution lengthened mean and median read times. In aggregate, this led to system inefficiencies that reduced the potential benefit of using the AI tool in our environment. A broader understanding of an AI solution’s impact on system efficiency may aid institutional decision-making and help set clearer expectations for end users.”
Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time
Andrew James Del Gaizo, et al.
Radiology: Artificial Intelligence 2024; 6(5):e240067 - “The experience reported by Del Gaizo et al has important implications and lessons for anyone planning to introduce AI solutions into clinical radiology practice or undertake similar research. Prospective users should assess whether their patient population is a close enough match to the population on which an AI program was developed for it to be used: They should assess the potential impact of prevalence on accuracy and predictive value. For a given combination of sensitivity and specificity, lower prevalence will result in lower estimates of PPV. Other important issues to assess are the impact on radiologists’ interpretation times and, for many clinical scenarios, impact on time to therapy. Conservatively, radiology practices introducing AI applications into their clinical operations should always undertake an assessment after implementation to determine how well the program is functioning in their respective unique environments.”
Challenges of Implementing Artificial Intelligence–enabled Programs in the Clinical Practice of Radiology
James H. Thrall
Radiology: Artificial Intelligence 2024; 6(5):e240411 - “A striking finding in the study reported by Del Gaizo et al was a positive predictive value (PPV) of only 21.1%. PPV is a function of sensitivity, specificity, and prevalence: It is the probability that a patient with a positive (abnormal) test result actually has the disease. The authors observe that the low prevalence of 2.7% in their study is the likely reason for the low PPV. Of note, McLouth et al reported a prevalence of ICH of 31% (255 of 814), indicating a different patient population than the current study. The corresponding PPV in the McLouth et al study was 91.4%. McLouth et al modeled different levels of prevalence, holding sensitivity and specificity constant, which showed PPV ranged from 80.2% at 10% prevalence to 97.3% at 50% prevalence.”
Challenges of Implementing Artificial Intelligence–enabled Programs in the Clinical Practice of Radiology
James H. Thrall
Radiology: Artificial Intelligence 2024; 6(5):e240411
- Differential Diagnosis: Unilateral Mass in Sinus
- Mucocele
- Fungus ball
- Inflammatory polyp
- Cholesterol granuloma
- Inverted papilloma
- Hemangioma
- Carcinoma
- Sinonasal hemangioma - "Sinonasal organized hematoma (OH) is an uncommon, nonneoplastic benign condition that can be locally aggressive. Without careful evaluation of all the imaging features, this may be mistaken for a malignant lesion both clinically and radiologically"
Sinonasal Organized Hematoma: CT and MR Imaging Findings
Kim EY et al.
AJNR 29:1204-1208 June/July 2008
case courtesy of Jim Zinreich at Johns Hopkins Hospital - Sinonasal Organized Hematoma: CT Findings
- Expansile mass (9/12)
- Ill defined mass (12/12)
- Typically maxillary sinus involved (11/12)
- Left side more common (8/12)
- Smooth, nonaggressive cortical breakthrough and scalloping of the maxillary sinus bony walls (8/12) - Sinonasal Organized Hematoma: MR Findings
- Isointensity on T1-weighted images compared with soft tissue of inferior turbinate (10/10)
- Marked heterogenous, mixed hypointense and isointense signal intensity on T2 weighted images (10/10)
- T2-weighted images showed hypointense peripheral rim surrounding the lesion (10/100 - "An expansile soft tissue mass, smooth sinus wall erosion, marked heterogeneous signal intensity with a hypointense peripheral rim on T2-weighted MR images, and marked irregular nodular, papillary, or frondlike enhancement are characteristic CT and MR imaging findings of sinonasal organized hematoma."
Sinonasal Organized Hematoma: CT and MR Imaging Findings
Kim EY et al.
AJNR 29:1204-1208 June/July 2008 - Sinonasal Organized Hematoma: How Does It Develop?
- Egardless of initial cause of bleeding poor drainage with development of a fibrous capsule prevent resorption of hematoma
- Neovascularization and fibrosis with recurrent bleeding lead to the eventual formation of the SOH