Weidi Xie

Microscopy cell counting and detection with fully convolutional regression networks

This paper concerns automated cell counting and detection in microscopy images. The approach we take is to use convolutional neural networks (CNNs) to regress a cell spatial density map across the image. This is applicable to situations where traditional single-cell segmentation-based methods do not work well due to cell clumping or overlaps. We make the following contributions: (i) we develop and compare architectures for two fully convolutional regression networks (FCRNs) for this task; (ii) since the networks are fully convolutional, they can predict a density map for an input image of arbitrary size, and we exploit this to improve efficiency by end-to-end training on image patches; (iii) we show that FCRNs trained entirely on synthetic data are able to give excellent predictions on microscopy images from real biological experiments without fine-tuning, and that the performance can be further improved by fine-tuning on these real images. Finally, (iv) by inverting feature representations, we show to what extent the information from an input image has been encoded by feature responses in different layers. We set a new state-of-the-art performance for cell counting on standard synthetic image benchmarks and show that the FCRNs trained entirely with synthetic data can generalise well to real microscopy images both for cell counting and detections for the case of overlapping cells.

Fully-automated alignment of 3D fetal brain ultrasound to a canonical reference space using multi-task learning.

HIGHLIGHTSWe propose a FCN to automatically co‐align 3D fetal neurosonography images.The multi‐task FCN predicts skull boundaries, eye location, and 3D brain orientation.Our proposed brain alignment method is invariant to fetal size and gestational age.Structural and anatomical correspondence was achieved in 88% of 140 tested volumes. ABSTRACT Methods for aligning 3D fetal neurosonography images must be robust to (i) intensity variations, (ii) anatomical and age‐specific differences within the fetal population, and (iii) the variations in fetal position. To this end, we propose a multi‐task fully convolutional neural network (FCN) architecture to address the problem of 3D fetal brain localization, structural segmentation, and alignment to a referential coordinate system. Instead of treating these tasks as independent problems, we optimize the network by simultaneously learning features shared within the input data pertaining to the correlated tasks, and later branching out into task‐specific output streams. Brain alignment is achieved by defining a parametric coordinate system based on skull boundaries, location of the eye sockets, and head pose, as predicted from intracranial structures. This information is used to estimate an affine transformation to align a volumetric image to the skull‐based coordinate system. Co‐alignment of 140 fetal ultrasound volumes (age range: 26.0±4.4 weeks) was achieved with high brain overlap and low eye localization error, regardless of gestational age or head size. The automatically co‐aligned volumes show good structural correspondence between fetal anatomies.

Robust Regression of Brain Maturation from 3D Fetal Neurosonography Using CRNs

We propose a fully three-dimensional Convolutional Regression Network (CRN) for the task of predicting fetal brain maturation from 3D ultrasound (US) data. Anatomical development is modelled as the sonographic patterns visible in the brain at a given gestational age, which are aggregated by the model into a single value: the brain maturation (BM) score. These patterns are learned from 589 3D fetal volumes, and the model is applied to 3D US images of 146 fetal subjects acquired at multiple, ethnically diverse sites, spanning an age range of 18 to 36 gestational weeks. Achieving a mean error of 7.7 days between ground-truth and estimated maturational scores, our method outperforms the current state-of-art for automated BM estimation from 3D US images.

VP-Nets : Efficient automatic localization of key brain structures in 3D fetal neurosonography

HighlightsWe propose VP‐Nets for brain structure localization in 3D fetal neurosonography.The proposed 2.5D CNN is memory efficient, producing a high resolution 3D output.Our model can handle structure overlap while keeping contextual information.Visualization of the trained VP‐Nets is described.VP‐Nets is compared with Random Forests and 3D U‐Nets. Graphical abstract Figure. No caption available. ABSTRACT Three‐dimensional (3D) fetal neurosonography is used clinically to detect cerebral abnormalities and to assess growth in the developing brain. However, manual identification of key brain structures in 3D ultrasound images requires expertise to perform and even then is tedious. Inspired by how sonographers view and interact with volumes during real‐time clinical scanning, we propose an efficient automatic method to simultaneously localize multiple brain structures in 3D fetal neurosonography. The proposed View‐based Projection Networks (VP‐Nets), uses three view‐based Convolutional Neural Networks (CNNs), to simplify 3D localizations by directly predicting 2D projections of the key structures onto three anatomical views. While designed for efficient use of data and GPU memory, the proposed VP‐Nets allows for full‐resolution 3D prediction. We investigated parameters that influence the performance of VP‐Nets, e.g. depth and number of feature channels. Moreover, we demonstrate that the model can pinpoint the structure in 3D space by visualizing the trained VP‐Nets, despite only 2D supervision being provided for a single stream during training. For comparison, we implemented two other baseline solutions based on Random Forest and 3D U‐Nets. In the reported experiments, VP‐Nets consistently outperformed other methods on localization. To test the importance of loss function, two identical models are trained with binary corss‐entropy and dice coefficient loss respectively. Our best VP‐Net model achieved prediction center deviation: 1.8 ± 1.4 mm, size difference: 1.9 ± 1.5 mm, and 3D Intersection Over Union (IOU): 63.2 ± 14.7% when compared to the ground truth. To make the whole pipeline intervention free, we also implement a skull‐stripping tool using 3D CNN, which achieves high segmentation accuracy. As a result, the proposed processing pipeline takes a raw ultrasound brain image as input, and output a skull‐stripped image with five detected key brain structures.

Feature tracking cardiac magnetic resonance via deep learning and spline optimization

Feature tracking Cardiac Magnetic Resonance (CMR) has recently emerged as an area of interest for quantification of regional cardiac function from balanced, steady state free precession (SSFP) cine sequences. However, currently available techniques lack full automation, limiting reproducibility. We propose a fully automated technique whereby a CMR image sequence is first segmented with a deep, fully convolutional neural network (CNN) architecture, and quadratic basis splines are fitted simultaneously across all cardiac frames using least squares optimization. Experiments are performed using data from 42 patients with hypertrophic cardiomyopathy (HCM) and 21 healthy control subjects. In terms of segmentation, we compared state-of-the-art CNN frameworks, U-Net and dilated convolution architectures, with and without temporal context, using cross validation with three folds. Performance relative to expert manual segmentation was similar across all networks: pixel accuracy was \(\sim 97\%\), intersection-over-union (IoU) across all classes was \(\sim 87\%\), and IoU across foreground classes only was \(\sim 85\%\). Endocardial left ventricular circumferential strain calculated from the proposed pipeline was significantly different in control and disease subjects (\(-25.3{}\%\) vs \(-29.1{}\%\), \(p=0.006{}\)), in agreement with the current clinical literature.

Ω-Net (Omega-Net): Fully automatic, multi-view cardiac MR detection, orientation, and segmentation with deep neural networks

HighlightsThe authors propose Omega‐Net: A novel convolutional neural network architecture for the detection, orientation, and segmentation of cardiac MR images.Three modules comprise the network: a coarse‐grained segmentation module, an attention module, and a fine‐grained segmentation module.The network is trained end‐to‐end from scratch using three‐fold crossvalidation in 63 subjects (42 with hypertrophic cardiomyopathy, 21 healthy).Performance of the Omega‐Net is substantively improved compared with UNet alone.In addition, to be comparable with other works, Omega‐Net was retrained from scratch using five‐fold cross‐validation on the publicly available 2017 MICCAI Automated Cardiac Diagnosis Challenge (ACDC) dataset, achieving state‐of‐the‐art performance in two of three segmentation classes. Graphical abstract Figure. No caption available. ABSTRACT Pixelwise segmentation of the left ventricular (LV) myocardium and the four cardiac chambers in 2‐D steady state free precession (SSFP) cine sequences is an essential preprocessing step for a wide range of analyses. Variability in contrast, appearance, orientation, and placement of the heart between patients, clinical views, scanners, and protocols makes fully automatic semantic segmentation a notoriously difficult problem. Here, we present &OHgr;‐Net (Omega‐Net): A novel convolutional neural network (CNN) architecture for simultaneous localization, transformation into a canonical orientation, and semantic segmentation. First, an initial segmentation is performed on the input image; second, the features learned during this initial segmentation are used to predict the parameters needed to transform the input image into a canonical orientation; and third, a final segmentation is performed on the transformed image. In this work, &OHgr;‐Nets of varying depths were trained to detect five foreground classes in any of three clinical views (short axis, SA; four‐chamber, 4C; two‐chamber, 2C), without prior knowledge of the view being segmented. This constitutes a substantially more challenging problem compared with prior work. The architecture was trained using three‐fold cross‐validation on a cohort of patients with hypertrophic cardiomyopathy (HCM, Symbol) and healthy control subjects (Symbol). Network performance, as measured by weighted foreground intersection‐over‐union (IoU), was substantially improved for the best‐performing &OHgr;‐Net compared with U‐Net segmentation without localization or orientation (0.858 vs 0.834). In addition, to be comparable with other works, &OHgr;‐Net was retrained from scratch using five‐fold cross‐validation on the publicly available 2017 MICCAI Automated Cardiac Diagnosis Challenge (ACDC) dataset. The &OHgr;‐Net outperformed the state‐of‐the‐art method in segmentation of the LV and RV bloodpools, and performed slightly worse in segmentation of the LV myocardium. We conclude that this architecture represents a substantive advancement over prior approaches, with implications for biomedical image segmentation more generally. Symbol. No caption available. Symbol. No caption available.

Can Dilated Convolutions Capture Ultrasound Video Dynamics

Automated analysis of free-hand ultrasound video sweeps is an important topic in diagnostic and interventional imaging, however, it is a notoriously challenging task for detecting the standard planes, due to the low-quality data, variability in contrast, appearance and placement of the structures. Conventionally, sequential data is usually modelled with heavy Recurrent Neural Networks (RNNs). In this paper, we propose to apply a convolutional architecture (CNNs) for the standard plane detection in free-hand ultrasound videos. Our contributions are twofolds, firstly, we show a simple convolutional architecture can be applied to characterize the long range dependencies in the challenging ultrasound video sequences, and outperform the canonical LSTMs and the recently proposed two-stream spatial ConvNet by a large margin (89% versus 83% and 84% respectively). Secondly, to get an understanding of what evidences have been used by the model for decision making, we experimented with the soft-attention layers for feature pooling, and trained the entire model end-to-end with only standard classification losses. As a result, we find the input-dependent attention maps can not only boost the network’s performance, but also indicate useful patterns of the data that are deemed important for certain structure, therefore provide interpretation while deploying the models.