Oren Avram, Berkin Durmus, Nadav Rakocz, Giulia Corradetti, Ulzee An, Muneeswar G Nittala, Prerit Terway, Akos Rudas, Zeyuan Johnson Chen, Yu Wakatsuki, Kazutaka Hirabayashi, Swetha Velaga, Liran Tiosano, Federico Corvi, Aditya Verma, Ayesha Karamat, Sophiana Lindenberg, Deniz Oncel, Louay Almidani, Victoria Hull, Sohaib Fasih-Ahmad, Houri Esmaeilkhanian, Maxime Cannesson, Charles C Wykoff, Elior Rahmani, Corey W Arnold, Bolei Zhou, Noah Zaitlen, Ilan Gronau, Sriram Sankararaman, Jeffrey N Chiang, Srinivas R Sadda, Eran Halperin
Nat Biomed Eng . 2024 Oct 1. doi: 10.1038/s41551-024-01257-9. Online ahead of print.
The application of machine learning to tasks involving volumetric biomedical imaging is constrained by the limited availability of annotated datasets of three-dimensional (3D) scans for model training. Here we report a deep-learning model pre-trained on 2D scans (for which annotated data are relatively abundant) that accurately predicts disease-risk factors from 3D medical-scan modalities. The model, which we named SLIViT (for 'slice integration by vision transformer'), preprocesses a given volumetric scan into 2D images, extracts their feature map and integrates it into a single prediction. We evaluated the model in eight different learning tasks, including classification and regression for six datasets involving four volumetric imaging modalities (computed tomography, magnetic resonance imaging, optical coherence tomography and ultrasound). SLIViT consistently outperformed domain-specific state-of-the-art models and was typically as accurate as clinical specialists who had spent considerable time manually annotating the analysed scans. Automating diagnosis tasks involving volumetric scans may save valuable clinician hours, reduce data acquisition costs and duration, and help expedite medical research and clinical applications.