• Satomi Kawamoto
• Seyoun Park
• Linda Chu
• Alejandra Blanco
• Shahab Shayesteh
• Saeed Ghandili
• Daniel Fadaei Fouladi
• Alan Yuille
• Ralph Hruban
• Kenneth Kinzler
• Bert Vogelstein
• Elliot K. Fishman

• To review roles of radiologists in development of artificial intelligence (AI) applications for early detection of pancreatic cancer on CT as a part of multidisciplinary team
• To review tasks involving development of AI applications and challenges of each task.
• To review roles of radiologist in each task involving development of AI applications

• Recent rapid growth of artificial intelligence (AI) in radiology has been mostly driven by computer scientists, engineers, and commercial companies. Radiologists generally have much less direct participation.
• However, radiologists can play a critical role in designing study, clinical applications and development of AI in medical imaging as a member of multidisciplinary team.
• Multidisciplinary approach has advantages in development of AI by working closely with expertise of different fields and fine tuning the process in real time.

• Radiologists play an important role in designing study and clinical applications and provide expertise in assuring accuracy of image segmentation highlighting critical imaging features, data acquisition, data collection, reviewing the results by algorithm, providing feedback to improve deep learning algorithms, and determining evaluation metrics in multidisciplinary team.
• We will review our experience of multidisciplinary collaboration in the development of successful AI program with the primary goal of early detection of pancreatic cancer.

• Artificial intelligence (AI) offers the opportunity to transform image interpretation from a purely qualitative and subjective task to effortless quantifiable and reproducible task.
• Deep learning (DL), a subset of AI, uses training data and multiple layers of equations to develop a mathematical model that fits the data.
• Convolutional neural network (CNN), a typical representative of deep learning, has been validated as an effective approach for radiological image classification in various studies.

• Trained: to create mathematical models that can be trained to produce useful outputs when input data are provided.
• Tuned: to produce accurate predictions for the training data by an optimization algorithm
• Generalized: to deliver correct predictions for new unseen data from their learned experience.

• Proper data collection of large-set images is critical to the successful training of a deep convolutional neural network.
• Radiology practice and is different in each institution and may change over time, e.g., patient populations, imaging equipment, imaging protocols.
• These factors may change how well an AI algorithm performs on the images.
• AI algorithm needs to be refined when data from different protocols, venders, and institutions are used to ensure its general applicability.
• Radiologists need to assure the data were acquired properly.

• One of the key elements of the success of CNN applied to natural images is the availability of very large-scale annotated datasets (e.g., 1.4 millions of images in ImageNet)
• As data sets get bigger and computers become more powerful, the results achieved by deep learning will get better.
• However, in medical images, the datasets are often very limited compared to natural images.
• To overcome the problem of insufficient data, fine-tuned DNN from natural images (i.e., data augmentation by scaling, rotating, or elastic deformation, and data synthesis) may be used to increase the number of training sets.

• Segmentation of the internal structures, like body organs, in medical images is an essential task for many clinical applications.
• Deep learning has largely relied on supervised learning approaches, i.e., large amounts of training data that composed of input images and ground truth annotations.
• Radiologists play a critical role for accuracy of segmentation of target structures and annotation of pathology.
• More than one radiologist could be engaged to review the images to have a better ground truth.

• Software tools can greatly speed up the segmentation process.
• The ideal software functionality would not be limited to basic two-dimensional (2D) drawings, but would include efficient patient-list management, visualization of annotated results, convenient graphic user interfaces for segmentation including advanced 3D segmentation, and automatic backup. 

• Example of manual segmentation and DNN prediction of normal abdominal organs in 3D rendering.
• In our series using 236 normal CT data, deep network prediction for the pancreas showed high fidelity with Dice similarity coefficient of 87.8 ± 3.1% and mean surface distances of 1.05 ± 0.65 mm.

• We developed a data collection and annotation process of for pancreatic ductal adenocarcinoma (PDAC) and Pancreatic neuroendocrine tumor (PNET) using volumetric CT.
• In this case, small PDAC in the uncinated process (allow) along with other abdominal organs were segmented.
• Radiologist plays a critical role for accuracy of segmentation of target structures and annotation of pathology.

• Accurate segmentation for supervised learning, especially by manual segmentation, requires experienced radiologists and is tedious and time consuming.
• As the size of the imaging datasets has increased, the time and cost required for the labor intensive process of manual segmentation has become more difficult.
• Interobserver and intraobserver variability are also an issue for manual segmentation.
• Semi-supervised or unsupervised learning approaches have been used to circumvent the need for large number of fully annotated datasets.

• Semi-supervised learning uses weakly annotated data, such as a bounding box of the target region. 
• Unsupervised learning employs a combination of detection and segmentation from raw images in the absence of any manual annotation. 
• However, semi-supervised and unsupervised approaches still require some annotated data for training. In addition, learning accuracy and convergences are clearly superior with supervised learning than with semi-supervised or unsupervised methods.
• To maximize the possibility and utility of deep learning, it is therefore optimal to have high-quality annotated data.

• After algorithm is tested and the results are reviewed by multidisciplinary team.
• Radiologists review the cases and provide feedback to the team to improve algorithms using their expertise.
  • To confirm if deep network prediction is correct.
  • To find out true lesion in false negative cases
  • To find out potential reasons for false positive prediction
  • To find out any potential annotation errors
• From the beginning of this project, we had weekly meeting to constantly review the annotation, data collection and algorithms to fine tune our processes in real time.

• Choose appropriate evaluation metric for the algorithm.
• Commonly used metrics are sensitivity, specificity, positive and negative predictive values.
• A reliable gold standard (ground truth) is critical.

• Receiver operating characteristic (ROC) curve: curvilinear graph generated by plotting true positive ratio as a function of false negative ratio.
• A given AI algorithm has a particular ROC curve.
• A particular point on ROC curve can be selected by AI developer to determine the output of the AL algorithm.

• Need to consider clinical scenarios for which AI algorithms are used.
• A performance threshold can be adjusted based on clinical scenario.
  • For example, to determine if high sensitivity (cancer detection) or specificity (ruling out an abnormality) is most important depending on the clinical scenarios
  • To determine the tolerance of the number of false positives and false negatives.
  • There is a trade-offs in performance metrics for AI algorithm.

• Although generally, radiologists currently have little formal role in the development of AI tools other than being the targeted consumers, radiologists can have a active role in development of AI tools in medical imaging as a member of multidisciplinary team.
• Multidisciplinary collaboration with expertise of different fields working closely has advantages in development of AI tools for early detection of pancreatic cancer.

• Radiologists provide expertise in assuring accuracy of image segmentation and annotation, and highlighting critical imaging features.
• Radiologists also play an important role in designing study and clinical applications, data acquisition, data collection, reviewing the results and providing feedback to improve deep learning algorithms.

• Radiologist can have an active role in guiding and facilitating the development and implementation of AI tools in diagnostic radiology.
• We believe that a multidisciplinary team approach is key to the success of this effort.

• Rubin DL. Artificial intelligence in imaging: The radiologist’s Role. J Am Coll Radiol 2019;16:1309-1317.
• Wang Y, Zhou Y, Shen W, Park S, Fishman EK, Yuille AL. Abdominal multi-organ segmentation with organ-attention networks and statistical fusion. http://arxiv.org/abs/1804.08414
• Hinton G. Deep learning – A technology with the potential to transform health care. JAMA 2018;320:1101-1102
• Deng J, et al. ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009, IEEE Conference on. IEEE. 2009;248-255.
• Lundervold AS, et al. An overview of deep learning in medical imaging focusing on MRI. Z Med Phys 2019;29:102-127.
• Park S, et al. Annotated normal CT data of the abdomen for deep learning. Diagnostic and Interventional Imaging 2019 [Epub ahead of print]
• Shin H, et al. Deep convolutional neural networks for computer-aided detection : CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 2016;35:1285–98.
• Chu L, et al. Application of deep learning to pancreatic cancer detection lessons learned from our initial experience JACR 2019;9:1338
• Zhu Z, Fishman EK, Yuille AL, et al. Multi-scale coarse-to-fine segmentation for screening pancreatic ductal adenocarcinoma. https://arxiv.org/pdf/1807.02941.
• Liu F, Fishman EK, Yuille AL, et al. Joint Shape Representation and Classification for Detecting PDAC in abdominal CT scans. https://arxiv.org/pdf/1804.10684.

Acknowledgements

• Satomi Kawamoto, M.D.
• Seyoun Park, Ph.D.
• Linda Chu, MD
• Alejandra Blanco, M.D.
• Shahab Shayesteh, M.D.
• Saeed Ghandili, M.D.
• Daniel Fadaei Fouladi, M.D.
• Alan Yuille, Ph.D.
• Ralph Hruban, M.D.
• Kenneth Kinzler, Ph.D.
• Bert Vogelstein, M.D.
• Elliot K. Fishman, M.D.