Elliot K. Fishman M.D.
Professor of Radiology, Surgery, Oncology and Urology
Johns Hopkins Hospital

Click here to view this module as a video lecture.

“In an independent study of six radiologists, the AI system outperformed all of the human readers: the area under the receiver operating characteristic curve (AUC-ROC) for the AI system was greater than the AUC-ROC for the average radiologist by an absolute margin of 11.5%. We ran a simulation in which the AI system participated in the double-reading process that is used in the UK, and found that the AI system maintained non-inferior performance and reduced the workload of the second reader by 88%. This robust assessment of the AI system paves the way for clinical trials to improve the accuracy and efficiency of breast cancer screening.”
International evaluation of an AI system for breast cancer screening 
Scott Mayer McKinney et al
Nature Vol577 2January2020

Objectives: Digital breast tomosynthesis (DBT) increases sensitivity of mammography and is increasingly implemented in breast cancer screening. However, the large volume of images increases the risk of reading errors and reading time. This study aims to investigate whether the accuracy of breast radiologists reading wide-angle DBT increases with the aid of an artificial intelligence (AI) support system. Also, the impact on reading time was assessed and the stand-alone performance of the AI system in the detection of malignancies was compared to the average radiologist.
Conclusions: Radiologists improved their cancer detection and reduced reading time when evaluating DBT examinations using an AI reading support system.
Impact of artificial intelligence support on accuracy and reading time in breast tomosynthesis image interpretation: a multi-reader multi-case study. 
van Winkel SL et al 
Eur Radiol. 2021 Nov;31(11):8682-8691.

• Radiologists improved their cancer detection accuracy in digital breast tomosynthesis (DBT) when using an AI system for support, while simultaneously reducing reading time. 
• The stand-alone breast cancer detection performance of an AI system is non-inferior to the average performance of radiologists for reading digital breast tomosynthesis exams. 
• The use of an AI support system could make advanced and more reliable imaging techniques more accessible and could allow for more cost-effective breast screening programs with DBT.
Impact of artificial intelligence support on accuracy and reading time in breast tomosynthesis image interpretation: a multi-reader multi-case study. 
van Winkel SL et al 
Eur Radiol. 2021 Nov;31(11):8682-8691.

• A conservative estimate found that 5 percent of U.S. adults who seek outpatient care each year experience a diagnostic error.
• Postmortem examination research spanning decades has shown that diagnostic errors contribute to approximately 10 percent of patient deaths.
• Medical record reviews suggest that diagnostic errors account for 6 to 17 percent of hospital adverse events. 
• Diagnostic errors are the leading type of paid medical malpractice claims, are almost twice as likely to have resulted in the patient’s death compared to other claims, and represent the highest proportion of total payments. 
Improving Diagnosis in Healthcare
Committee on Diagnostic Error in Health Care
Erin P. Balogh, Bryan T. Miller, and John R. Ball, Editors 
Board on Health Care Services, Institute of Medicine
The National Academies Press, [2015]

“In reviewing the evidence, the committee concluded that most people will experience at least one diagnostic error in their lifetime, sometimes with devastating consequences. Despite the pervasiveness of diagnostic errors and the risk for serious patient harm, diagnostic errors have been largely unappreciated within the quality and patient safety movements in health care. Without a dedicated focus on improving diagnosis, these errors will likely worsen as the delivery of health care and the diagnostic process continue to increase in complexity.”
Improving Diagnosis in Healthcare
Committee on Diagnostic Error in Health Care
Erin P. Balogh, Bryan T. Miller, and John R. Ball, Editors 
Board on Health Care Services, Institute of Medicine
The National Academies Press, [2015]

“Perceptual or cognitive errors made by radiologists are a source of diagnostic error. In addition, incomplete or incorrect patient information, as well as insufficient sharing of patient information, may lead to the use of an inadequate imaging protocol, an incorrect interpretation of imaging results, or the selection of an inappropriate imaging test by a referring clinician. Referring clinicians often struggle with selecting the appropriate imaging test, in part because of the large number of available imaging options and gaps in the teaching of radiology in medical schools.” 
Improving Diagnosis in Healthcare
Committee on Diagnostic Error in Health Care
Erin P. Balogh, Bryan T. Miller, and John R. Ball, Editors 
Board on Health Care Services, Institute of Medicine
The National Academies Press, [2015]

“ In the daily radiology practice, the rate of interpretation error is between 3% and 4%; however, of the radiology studies that contain abnormalities, the error rate is even higher, averaging in the 30% range.”
Fool Me Twice: Delayed Diagnoses in Radiology With Emphasis on Perpetuated Errors
Kim YW, Mansfield LT
AJR 2014;202:465-470

“ In our study, the majority of errors made were errors of underreading (42%), where the finding was simply missed.”
Fool Me Twice: Delayed Diagnoses in Radiology With Emphasis on Perpetuated Errors
Kim YW, Mansfield LT
AJR 2014;202:465-470

“Missed findings rather than misinterpretations of detected abnormalities were the most common reason for abdominopelvic CT report addenda. Awareness of the most common misses by anatomic location may help guide quality assurance initiatives. A wide variety of contributing factors were identified. Informatics and workflow optimization may be warranted to facilitate radiologists’ access to all available patient-related data, as well as communication with other physicians, and thereby help reduce diagnostic errors.” 
Diagnostic errors in abdominopelvic CT interpretation: characterization based on report addenda 
Andrew B. Rosenkrantz, Neil K. Bansal 
Abdom Radiol (2016) 41:1793–1799

“The purpose of the study was to determine if increasing radiologist reading speed results in more misses and interpretation errors.”
The Effect of Faster Reporting Speed for Imaging Studies on the Number of Misses and Interpretation Errors: A Pilot Study.
Sokolovskaya E et al.
J Am Coll Radiol. 2015 Jul;12(7):683-8. doi: 10.1016/j.jacr.

“ Reading at the faster speed resulted in more major misses for 4 of the 5 radiologists. The total number of major misses for the 5 radiologists, when they reported at the faster speed, was 16 of 60 reported cases, versus 6 of 60 reported cases at normal speed; P = .032. The average interpretation error rate of major misses among the 5 radiologists reporting at the faster speed was 26.6%, compared with 10% at normal speed.”
The Effect of Faster Reporting Speed for Imaging Studies on the Number of Misses and Interpretation Errors: A Pilot Study.
Sokolovskaya E et al.
J Am Coll Radiol. 2015 Jul;12(7):683-8. doi: 10.1016/j.jacr.

“Machine learning will displace much of the work of radiologists and anatomical pathologists. These physicians focus largely on interpreting digitized images, which can easily be fed directly to algorithms instead. Massive imaging data sets, com- bined with recent advances in computer vision, will drive rapid improvements in performance, and machine accuracy will soon exceed that of humans. Indeed, radiology is already partway there: algorithms can replace a second radiologist reading mammograms and will soon exceed human accuracy.”
Predicting the Future — Big Data, Machine Learning, and Clinical Medicine
Obermeyer Z, Emanuel EJ
N Engl J Med 375;13 September 29, 2016

“The patient- safety movement will increasingly advocate the use of algorithms over humans — after all, algorithms need no sleep, and their vigilance is the same at 2 a.m. as at 9 a.m. Algorithms will also monitor and interpret streaming physiological data, replacing aspects of anesthesiology and criti- cal care. The time scale for these disruptions is years, not decades.”
Predicting the Future — Big Data, Machine Learning, and Clinical Medicine
Obermeyer Z, Emanuel EJ
N Engl J Med 375;13 September 29, 2016

“I think that if you work as a radiologist you are like Wile E. Coyote in the cartoon,” Hinton told me. “You’re already over the edge of the cliff, but you haven’t yet looked down. There’s no ground underneath.” Deep-learning systems for breast and heart imaging have already been developed commercially. “It’s just completely obvious that in five years deep learning is going to do better than radiologists,” he went on. “It might be ten years. I said this at a hospital. It did not go down too well.”
Geoffrey Hinton
University of Toronto

“It’s hard to predict the future, and what immensely complicates predictions over seemingly promising technologies like gene therapy or AI is how their complex construction will interface with other equally complex and dynamic technologies, all of which operate in an environment of unceasing economic and institutional flux. It remains anyone’s guess as to how AI applications will be affected by their integration with PACS, how liability trends or regulatory efforts will affect AI, whether reimbursement for AI will justify its use, how mergers and acquisitions will affect AI implementation, and how well AI models will accommodate ethical requirements related to informed consent, privacy, and patient access.”
AI Hype and Radiology: A Plea for Realism and Accuracy 
Banja J et al.
Radiology: Artificial Intelligence 2020; 2(4):e190223

”It is important to keep the evolution of the AI/ML technology in context so as not to become overly enthusiastic about the current capabilities and simultaneously not to become overly pessimistic about future developments. Though the promised delivery date of fully self-driving cars has continuously been pushed back for the past decade, it is undeniable that drivers in semiautonomous vehicles are safer than unassisted drivers. Similarly, there are tangible patient care and cost benefits to be obtained through staged development of AL/ML systems even if fully autonomous MD systems are not on the horizon.”
Applications of artificial intelligence in the emergency department 
Supratik K. Moulik, Nina Kotter, Elliot K. Fishman
Emergency Radiology (2020) 27:355–358

“The main challenge of EM involves the timely provision of medical triage defined as the process of sorting patients according to urgency and severity. This is required owing to the unpredictable nature of emergencies and conditions present where resources (e.g. staffing/beds) are sometimes limited and stretched. Relevant literature on department crowding and patient flow have shown impacts to quality of patient care. To a grimmer extent, such instances are also linked to increased mortality levels. The use of machine learning and deep learning can potentially help discern patterns of data gathered over the years and shed insights for improvement in ED processes.”
Artificial Intelligence and Machine Learning in Emergency Medicine 
Tang KJW et al.
Biocybernetics and Biomedical Engineering 41 (2021) 156–172

“These four main applications are: 
(1) Pre-hospital emergency management, 
(2) Patient acuity, triage and disposition, 
(3) Prediction of medical ailments and conditions, and 
(4) Emergency department management.” 
Artificial Intelligence and Machine Learning in Emergency Medicine 
Tang KJW et al.
Biocybernetics and Biomedical Engineering 41 (2021) 156–172

Approval of artificial intelligence and machine learning based medical devices in the USA and Europe (2015–20): a comparative analysis
Urs J Muehlematter, Paola Daniore, Kerstin N Vokinger
Lancet Digit Health 2021; 3: e195–203

“The FDA strategy of approving programs that do not change the radiologist’s final read of a study protects the patient and provides a comfort zone for radiologists. Various articles have suggested that radiologists are concerned about the potential impact of AI on the job market. Many junior people, including medical students and residents, are clearly worried that the use of AI may reduce employment or yield decreased reimbursement. While the future is impossible to predict, AI is unlikely to exert such impacts in the short term.”
The first use of artificial intelligence (AI) in the ER: triage not diagnosis 
Edmund M. Weisberg, Linda C. Chu, Elliot K. Fishman
Emergency Radiology (2020) 27:361–366

”As we have maintained in another venue, though, radiologists will continue to be crucial in rendering complex ideas intelligible and interpreting the results of advancing technologies such as AI and machine learning. The first wave of AI applications is not replacing radiologists. Rather, the innovative software is improving throughput, contributing to the timeliness in which radiologists can get to read abnormal scans, and possibly enhances radiologists’ accuracy. As for what the future brings, time will tell.”
The first use of artificial intelligence (AI) in the ER: triage not diagnosis 
Edmund M. Weisberg, Linda C. Chu, Elliot K. Fishman
Emergency Radiology (2020) 27:361–366

“The FDA approval process to date has focused on applications (apps) that affect patient triage and not necessarily apps that have the computer serve as the only or final reader. We have chosen a select group of apps to provide the reader with a sense of the current state of AI use in the ER setting. Because adoption of new technology and FDA approval are always works in progress, it is not our intention here to be comprehensive. For a more thorough review of approved AI applications, please see the American College of Radiology record available here (https:// www.acrdsi.org/DSI-Services/FDA-Cleared-AI-Algorithms).” 
The first use of artificial intelligence (AI) in the ER: triage not diagnosis 
Edmund M. Weisberg, Linda C. Chu, Elliot K. Fishman
Emergency Radiology (2020) 27:361–366