Christian Ledig

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation

HIGHLIGHTSAn efficient 11‐layers deep, multi‐scale, 3D CNN architecture.A novel training strategy that significantly boosts performance.The first employment of a 3D fully connected CRF for post‐processing.State‐of‐the‐art performance on three challenging lesion segmentation tasks.New insights into the automatically learned intermediate representations. ABSTRACT We propose a dual pathway, 11‐layers deep, three‐dimensional Convolutional Neural Network for the challenging task of brain lesion segmentation. The devised architecture is the result of an in‐depth analysis of the limitations of current networks proposed for similar applications. To overcome the computational burden of processing 3D medical scans, we have devised an efficient and effective dense training scheme which joins the processing of adjacent image patches into one pass through the network while automatically adapting to the inherent class imbalance present in the data. Further, we analyze the development of deeper, thus more discriminative 3D CNNs. In order to incorporate both local and larger contextual information, we employ a dual pathway architecture that processes the input images at multiple scales simultaneously. For post‐processing of the networks soft segmentation, we use a 3D fully connected Conditional Random Field which effectively removes false positives. Our pipeline is extensively evaluated on three challenging tasks of lesion segmentation in multi‐channel MRI patient data with traumatic brain injuries, brain tumours, and ischemic stroke. We improve on the state‐of‐the‐art for all three applications, with top ranking performance on the public benchmarks BRATS 2015 and ISLES 2015. Our method is computationally efficient, which allows its adoption in a variety of research and clinical settings. The source code of our implementation is made publicly available.

Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: The CADDementia challenge

Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimers disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimers Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.

Automatic Whole Brain MRI Segmentation of the Developing Neonatal Brain

Magnetic resonance (MR) imaging is increasingly being used to assess brain growth and development in infants. Such studies are often based on quantitative analysis of anatomical segmentations of brain MR images. However, the large changes in brain shape and appearance associated with development, the lower signal to noise ratio and partial volume effects in the neonatal brain present challenges for automatic segmentation of neonatal MR imaging data. In this study, we propose a framework for accurate intensity-based segmentation of the developing neonatal brain, from the early preterm period to term-equivalent age, into 50 brain regions. We present a novel segmentation algorithm that models the intensities across the whole brain by introducing a structural hierarchy and anatomical constraints. The proposed method is compared to standard atlas-based techniques and improves label overlaps with respect to manual reference segmentations. We demonstrate that the proposed technique achieves highly accurate results and is very robust across a wide range of gestational ages, from 24 weeks gestational age to term-equivalent age.

Automatic quantification of normal cortical folding patterns from fetal brain MRI

We automatically quantify patterns of normal cortical folding in the developing fetus from in utero MR images (N=80) over a wide gestational age (GA) range (21.7 to 38.9weeks). This work on data from healthy subjects represents a first step towards characterising abnormal folding that may be related to pathology, facilitating earlier diagnosis and intervention. The cortical boundary was delineated by automatically segmenting the brain MR image into a number of key structures. This utilised a spatio-temporal atlas as tissue priors in an expectation-maximization approach with second order Markov random field (MRF) regularization to improve the accuracy of the cortical boundary estimate. An implicit high resolution surface was then used to compute cortical folding measures. We validated the automated segmentations with manual delineations and the average surface discrepancy was of the order of 1mm. Eight curvature-based folding measures were computed for each fetal cortex and used to give summary shape descriptors. These were strongly correlated with GA (R(2)=0.99) confirming the close link between neurological development and cortical convolution. This allowed an age-dependent non-linear model to be accurately fitted to the folding measures. The model supports visual observations that, after a slow initial start, cortical folding increases rapidly between 25 and 30weeks and subsequently slows near birth. The model allows the accurate prediction of fetal age from an observed folding measure with a smaller error where growth is fastest. We also analysed regional patterns in folding by parcellating each fetal cortex using a nine-region anatomical atlas and found that Gompertz models fitted the change in lobar regions. Regional differences in growth rate were detected, with the parietal and posterior temporal lobes exhibiting the fastest growth, while the cingulate, frontal and medial temporal lobes developed more slowly.

ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI

&NA; Ischemic stroke is the most common cerebrovascular disease, and its diagnosis, treatment, and study relies on non‐invasive imaging. Algorithms for stroke lesion segmentation from magnetic resonance imaging (MRI) volumes are intensely researched, but the reported results are largely incomparable due to different datasets and evaluation schemes. We approached this urgent problem of comparability with the Ischemic Stroke Lesion Segmentation (ISLES) challenge organized in conjunction with the MICCAI 2015 conference. In this paper we propose a common evaluation framework, describe the publicly available datasets, and present the results of the two sub‐challenges: Sub‐Acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES). A total of 16 research groups participated with a wide range of state‐of‐the‐art automatic segmentation algorithms. A thorough analysis of the obtained data enables a critical evaluation of the current state‐of‐the‐art, recommendations for further developments, and the identification of remaining challenges. The segmentation of acute perfusion lesions addressed in SPES was found to be feasible. However, algorithms applied to sub‐acute lesion segmentation in SISS still lack accuracy. Overall, no algorithmic characteristic of any method was found to perform superior to the others. Instead, the characteristics of stroke lesion appearances, their evolution, and the observed challenges should be studied in detail. The annotated ISLES image datasets continue to be publicly available through an online evaluation system to serve as an ongoing benchmarking resource (www.isles‐challenge.org). HighlightsEvaluation framework for automatic stroke lesion segmentation from MRIPublic multi‐center, multi‐vendor, multi‐protocol databases releasedOngoing fair and automated benchmark with expert created ground truth setsComparison of 14+7 groups who responded to an open challenge in MICCAISegmentation feasible in acute and unsolved in sub‐acute cases Graphical abstract Figure. No caption available.

Cardiac Image Super-Resolution with Global Correspondence Using Multi-Atlas PatchMatch

The accurate measurement of 3D cardiac function is an important task in the analysis of cardiac magnetic resonance (MR) images. However, short-axis image acquisitions with thick slices are commonly used in clinical practice due to constraints of acquisition time, signal-to-noise ratio and patient compliance. In this situation, the estimation of a high-resolution image can provide an approximation of the underlaying 3D measurements. In this paper, we develop a novel algorithm for the estimation of high-resolution cardiac MR images from single short-axis cardiac MR image stacks. First, we propose to use a novel approximate global search approach to find patch correspondence between the short-axis MR image and a set of atlases. Then, we propose an innovative super-resolution model which does not require explicit motion estimation. Finally, we build an expectation-maximization framework to optimize the model. We validate the proposed approach using images from 19 subjects with 200 atlases and show that the proposed algorithm significantly outperforms conventional interpolation such as linear or B-spline interpolation. In addition, we show that the super-resolved images can be used for the reproducible estimation of 3D cardiac functional indices.

Unsupervised Domain Adaptation in Brain Lesion Segmentation with Adversarial Networks

Significant advances have been made towards building accurate automatic segmentation systems for a variety of biomedical applications using machine learning. However, the performance of these systems often degrades when they are applied on new data that differ from the training data, for example, due to variations in imaging protocols. Manually annotating new data for each test domain is not a feasible solution. In this work we investigate unsupervised domain adaptation using adversarial neural networks to train a segmentation method which is more robust to differences in the input data, and which does not require any annotations on the test domain. Specifically, we derive domain-invariant features by learning to counter an adversarial network, which attempts to classify the domain of the input data by observing the activations of the segmentation network. Furthermore, we propose a multi-connected domain discriminator for improved adversarial training. Our system is evaluated using two MR databases of subjects with traumatic brain injuries, acquired using different scanners and imaging protocols. Using our unsupervised approach, we obtain segmentation accuracies which are close to the upper bound of supervised domain adaptation.

Dynamic Changes in White Matter Abnormalities Correlate With Late Improvement and Deterioration Following TBI: A Diffusion Tensor Imaging Study

Objective. Traumatic brain injury (TBI) is not a single insult with monophasic resolution, but a chronic disease, with dynamic processes that remain active for years. We aimed to assess patient trajectories over the entire disease narrative, from ictus to late outcome. Methods. Twelve patients with moderate-to-severe TBI underwent magnetic resonance imaging in the acute phase (within 1 week of injury) and twice in the chronic phase of injury (median 7 and 21 months), with some undergoing imaging at up to 2 additional time points. Longitudinal imaging changes were assessed using structural volumetry, deterministic tractography, voxel-based diffusion tensor analysis, and region of interest analyses (including corpus callosum, parasagittal white matter, and thalamus). Imaging changes were related to behavior. Results. Changes in structural volumes, fractional anisotropy, and mean diffusivity continued for months to years postictus. Changes in diffusion tensor imaging were driven by increases in both axial and radial diffusivity except for the earliest time point, and were associated with changes in reaction time and performance in a visual memory and learning task (paired associates learning). Dynamic structural changes after TBI can be detected using diffusion tensor imaging and could explain changes in behavior. Conclusions. These data can provide further insight into early and late pathophysiology, and begin to provide a framework that allows magnetic resonance imaging to be used as an imaging biomarker of therapy response. Knowledge of the temporal pattern of changes in TBI patient populations also provides a contextual framework for assessing imaging changes in individuals at any given time point.

Differential diagnosis of neurodegenerative diseases using structural MRI data.

Different neurodegenerative diseases can cause memory disorders and other cognitive impairments. The early detection and the stratification of patients according to the underlying disease are essential for an efficient approach to this healthcare challenge. This emphasizes the importance of differential diagnostics. Most studies compare patients and controls, or Alzheimers disease with one other type of dementia. Such a bilateral comparison does not resemble clinical practice, where a clinician is faced with a number of different possible types of dementia. Here we studied which features in structural magnetic resonance imaging (MRI) scans could best distinguish four types of dementia, Alzheimers disease, frontotemporal dementia, vascular dementia, and dementia with Lewy bodies, and control subjects. We extracted an extensive set of features quantifying volumetric and morphometric characteristics from T1 images, and vascular characteristics from FLAIR images. Classification was performed using a multi-class classifier based on Disease State Index methodology. The classifier provided continuous probability indices for each disease to support clinical decision making. A dataset of 504 individuals was used for evaluation. The cross-validated classification accuracy was 70.6% and balanced accuracy was 69.1% for the five disease groups using only automatically determined MRI features. Vascular dementia patients could be detected with high sensitivity (96%) using features from FLAIR images. Controls (sensitivity 82%) and Alzheimers disease patients (sensitivity 74%) could be accurately classified using T1-based features, whereas the most difficult group was the dementia with Lewy bodies (sensitivity 32%). These results were notable better than the classification accuracies obtained with visual MRI ratings (accuracy 44.6%, balanced accuracy 51.6%). Different quantification methods provided complementary information, and consequently, the best results were obtained by utilizing several quantification methods. The results prove that automatic quantification methods and computerized decision support methods are feasible for clinical practice and provide comprehensive information that may help clinicians in the diagnosis making.