Medical Imaging with Deep Learning in the Time of Coronavirus
Montréal was the venue for Medical Imaging with Deep Learning 2020 (MIDL 2020) that took place on 6-9 July 2020. Due to the COVID-19 crisis, like many meetings this year (such as the ISMB conference we attend every year), the new normal has been to make it a virtual conference. GigaScience Data Scientist Chris Armit sat through the talks as they were streamed, and was deeply impressed at how Deep Learning is being used to support analysis of clinical imaging data. If you don’t have time to watch the archived content yourselves Chris has provided his write-up (following his many other imaging conference reports) and curated some videos of his favorite talks here.
Deep Learning: Predicting Short-Term Outcome of COVID-19
As a keynote speaker, Nikos Paragios of CentraleSupélec delivered a highly topical and fascinating talk on “AI-driven quantification, staging outcome prediction of COVID-19 pneumonia”. Nikos explained that there do exist AI-based tools to quantify interstitial lung diseases that share common CT features with COVID-19, but that there is a need for a validated model that is able to predict short-term outcome for COVID-19 lung disease including the need for intubation. The high dimensional radiomic features utilised by Nikos in his analysis included: shape features, first-order statistics that describe individual voxel values, and second-order statistics that describe textural features calculated between neighbouring voxels. Nikos compared the ability of 2D AtlasNet, which involves spatially mapping CT image data of the lung to a template, with 3D U-Net that captures the 3D texture of the diseased lung and reported that, in this comparison of Deep Learning ensembles, 2D AtlasNet performed slightly better for predicting COVID-19 outcome. In addition, Nikos detailed the Holistic Multi-Omics Selection that was used to determine the most informative features in the clinical setting, and a highly significant finding was the importance of biological features, such as age, sex, blood pressure, CRP, diabetes, and fat index, in predicting COVID-19 outcome. Indeed, by using an ablation study, whereby biological features and/or imaging data of the heart and lung were removed from the analysis to find if they are necessary, Nikos highlighted that imaging was not the most important feature. This highlights the need for clinical metadata to be included in the analysis of COVID-19 pneumonia. As we’ve encouraged community annotation of COVID-19 research, imaging is another area where the richness of curation and metadata is extremely important.
Machine Learning: The Dangers of Multisite Analysis
In her keynote talk entitled “Machine Learning: A New Approach to Drug Discovery”, Daphne Koller – the CEO and Founder of insitro – highlighted the importance of data quality by using a highly illustrative example of machine learning applied to X-ray images and that was originally reported by Badgeley et al. On initial inspection, this Machine Learning model offered a clean separation of X-ray images with fracture and those without fracture. However, on closer inspection the underlying predictive model had picked up on differences between the various scanners used in different hospitals, and the Machine Learning algorithm was simply finding the hospital scanner that had been used to scan a higher incidence of fractures. Consequently, the device used to capture the radiograph was, in this instance, the main source of variation.
As a means of addressing this issue, Daniel Moyer of USC presented an insightful “Overview of Scanner Invariant Representations”. As Daniel explained, whereas supervised problems are often of the form data x (image) predicts label y (diagnoses, outcomes), multi-site analyses can be a problem as “multi-site analyses have varying site signals”. The task then becomes how to remove scanner bias prior to performing the supervised problem. At the conference, Daniel reported that there is value in using a Deep Learning method based on variational auto-encoders (VAE) to construct scanner invariant encodings of the imaging data. A key feature of this approach is that it uses a conditional auto-encoder that utilises adversarial image-space losses to remove scanner bias. In a poster session, Daniel showcased how this approach has been successfully applied to the 2018 MICCAI Computational Diffusion MRI (CDMRI) Challenge Harmonization dataset.
Predicting Age-Related Changes in the Brain
Marianne Rakic of MIT and ETH Zürich showcased “Anatomical Predictions using Subject-Specific Medical Data”. Given a baseline MRI brain scan and subject specific information, Marianne questioned whether it is possible to predict the follow-up MRI brain scan at some later time? This would be incredibly useful as it could help guide clinical treatment and may also provide scientific insight. To test this hypothesis, Marianne captured the difference between a baseline MRI brain scan and a follow-up MRI brain scan by using a deformation field. A Convolutional Neural Network (CNN) was then used to predict the deformation field. Marianne reports that this approach performs better on brain ventricles than on smaller structures, but that it is further improved by the inclusion of non-image information, such as knowing the age of the patient. As with the aforementioned keynote lecture by Nikos Paragios on COVID-19 pneumonia, a key message was that there is great value in clinical metadata in these Deep Learning approaches.
Digital Pathology – When CNNs Fail
Jasper Linmans of Radboud UMC, working with Jeroen van der Laak and Geert Litjens, reported on “Efficient Out-of-Distribution Detection in Digital Pathology Using Multi-Head Convolutional Neural Networks”. Jeroen van der Laak and Geert Litjens had previously published the 1399 H&E-stained sentinel lymph node sections of breast cancer patients used in the CAMELYON study in GigaScience, and so I was intrigued to find out about their recent research on Convolutional Neural Networks (CNNs). As Jasper explained, “CNNs fail silently” and there is a need for Out-of-Distribution (OOD) detection as a means of delivering an uncertainty score. For detecting OOD, most popular approaches measure entropy value using Deep Ensembles or Mc-Dropout. However, Jasper instead proposes using Multi-Head CNNs that are both computationally and memory efficient. As Jasper explained, “the biggest advantage of being more memory efficient is the ability to train the entire model at once and to promote diversity during training”. Jasper showcased this method on the CAMELYON lymphoma data, and highlighted how Multi-Head CNNs offer efficient OOD detection, and can be used to correct Machine Learning model false-positive predictions.
Generative Adversarial Networks and Tensor Networks in Digital Pathology
In the field of Digital Pathology, there is a continuing focus on Machine Learning as a tool for automated diagnosis. However, I was deeply impressed by the presentation by Adalberto Claudio Quiros of the University of Glasgow who reported on “Pathology GAN: Learning deep representations of cancer tissue”. Adalberto explained that there are limitations on supervised learning as “it cannot provide unknown information about the data” and that consequently there is need for a generative model that can find fundamental morphological characteristics of tissue and that can be used to “identify and reproduce the different types of tissue”. Adalberto utilised Generative Adversarial Networks (GAN), whereby two neural networks contest with each other to deliver high-level features of histology image data, and applied this approach to deep representation of the entire tissue architecture of cancer tissue. Whereas other existing GAN-based tools – such as BigGAN – randomly place vector samples in latent space, a key advantage of Pathology GAN is that it shows structure in the latent space thus making the image that is generated interpretable. Adalberto anticipates that the disentangled representations delivered by Pathology GAN will provide “further understanding on phenotype diversity between and within tumours.”
I was additionally impressed by the presentation by Raghavendra Selvan, who reported on “Tensor Networks for Medical Image Classification”. Raghavendra suggested that, rather than flatten images into a one-dimensional vector, an alternative approach is to create a Locally orderless Tensor Network that can be shown as a series of layers. A tensor is a multidimensional array of numerical values that can be used to describe an image, and in tensor notation a zero edge tensor is a scalar, a one-edge tensor is a 1D vector, a two-edge tensor is a 2D matrix, and a three-edge tensor is a 3D matrix. Raghavendra’s approach is to first flatten a 2D image into a vector, but then to use tensor products to aggregate at multiple resolutions to create a high dimensional feature map. Raghavendra reported that this “squeeze operation” helps retain structure. The tensor network approach has the added value that it leads to a massive reduction in GPU utilisation, which is a known issue when processing very large images.
Medical Imaging with Deep Learning 2020 was a virtual conference and as we are rapidly learning these can be something of a challenge. However, the organisers are to be commended for delivering an exceptional online experience, and for providing a much-needed forum for deep learning researchers and clinicians at the intersection of machine learning and medical image analysis. One advantage of online conferences is that it’s easier to archive the video content, and the replay videos of day 1, day 2 and day3 are all available on youtube.
MIDL 2020 additionally highlighted the need for high-level curation of medical imaging data that is used for Machine Learning and Deep Learning. The International Society of Biocuration has touched on this subject previously, and have explicitly stated how important it is to annotate imaging data and “to properly attribute and track data provenance”. Collaboration between ISB and MISL 2020 may help to further address this need, and to raise awareness of the needs of the Deep Learning community amongst biocurators.
