Christian Payer

Regressing Heatmaps for Multiple Landmark Localization Using CNNs

We explore the applicability of deep convolutional neural networks (CNNs) for multiple landmark localization in medical image data. Exploiting the idea of regressing heatmaps for individual landmark locations, we investigate several fully convolutional 2D and 3D CNN architectures by training them in an end-to-end manner. We further propose a novel SpatialConfiguration-Net architecture that effectively combines accurate local appearance responses with spatial landmark configurations that model anatomical variation. Evaluation of our different architectures on 2D and 3D hand image datasets show that heatmap regression based on CNNs achieves state-of-the-art landmark localization performance, with SpatialConfiguration-Net being robust even in case of limited amounts of training data.

Automated Age Estimation from Hand MRI Volumes Using Deep Learning

Biological age (BA) estimation from radiologic data is an important topic in clinical medicine, e.g. in determining endocrinological diseases or planning paediatric orthopaedic surgeries, while in legal medicine it is employed to approximate chronological age. In this work, we propose the use of deep convolutional neural networks (DCNN) for automatic BA estimation from hand MRI volumes, inspired by the way radiologists visually perform age estimation using established staging schemes that follow physical maturation. In our results we outperform the state of the art automatic BA estimation method, achieving a mean error between estimated and ground truth BA of \(0.36\,\pm \,0.30\) years, which is in line with radiologists doing visual BA estimation.

Automated integer programming based separation of arteries and veins from thoracic CT images

Automated computer-aided analysis of lung vessels has shown to yield promising results for non-invasive diagnosis of lung diseases. To detect vascular changes which affect pulmonary arteries and veins differently, both compartments need to be identified. We present a novel, fully automatic method that separates arteries and veins in thoracic computed tomography images, by combining local as well as global properties of pulmonary vessels. We split the problem into two parts: the extraction of multiple distinct vessel subtrees, and their subsequent labeling into arteries and veins. Subtree extraction is performed with an integer program (IP), based on local vessel geometry. As naively solving this IP is time-consuming, we show how to drastically reduce computational effort by reformulating it as a Markov Random Field. Afterwards, each subtree is labeled as either arterial or venous by a second IP, using two anatomical properties of pulmonary vessels: the uniform distribution of arteries and veins, and the parallel configuration and close proximity of arteries and bronchi. We evaluate algorithm performance by comparing the results with 25 voxel-based manual reference segmentations. On this dataset, we show good performance of the subtree extraction, consisting of very few non-vascular structures (median value: 0.9%) and merged subtrees (median value: 0.6%). The resulting separation of arteries and veins achieves a median voxel-based overlap of 96.3% with the manual reference segmentations, outperforming a state-of-the-art interactive method. In conclusion, our novel approach provides an opportunity to become an integral part of computer aided pulmonary diagnosis, where artery/vein separation is important.

Multi-factorial Age Estimation from Skeletal and Dental MRI Volumes

Age estimation from radiologic data is an important topic in forensic medicine to assess chronological age or to discriminate minors from adults, e.g. asylum seekers lacking valid identification documents. In this work we propose automatic multi-factorial age estimation methods based on MRI data to extend the maximal age range from 19 years, as commonly used for age assessment based on hand bones, up to 25 years, when combined with wisdom teeth and clavicles. Mimicking how radiologists perform age estimation, our proposed method based on deep convolutional neural networks achieves a result of \(1.14 \pm 0.96\) years of mean absolute error in predicting chronological age. Further, when fine-tuning the same network for majority age classification, we show an improvement in sensitivity of the multi-factorial system compared to solely relying on the hand.

Pulmonary Lobe Segmentation in CT Images using Alpha-Expansion.

Fully-automatic lung lobe segmentation in pathological lungs is still a challenging task. A new approach for automatic lung lobe segmentation is presented based on airways, vessels, fissures and prior knowledge on lobar shape. The anatomical information and prior knowledge are combined into an energy equation, which is minimized via graph cuts to yield an optimal segmentation. The algorithm is quantitatively validated on an in-house dataset of 25 scans and on the LObe and Lung Analysis 2011 (LOLA11) dataset, which contains a range of different challenging lungs (total of 55) with respect to lobe segmentation. Both experiments achieved solid results including a median absolute distance from manually set fissure markers of 1.04mm (interquartile range: 0.88-1.09mm) on the in-house dataset and a score of 0.866 on the LOLA11 dataset. We conclude that our proposed method is robust even in case of pathologies.

Multi-label Whole Heart Segmentation Using CNNs and Anatomical Label Configurations

We propose a pipeline of two fully convolutional networks for automatic multi-label whole heart segmentation from CT and MRI volumes. At first, a convolutional neural network (CNN) localizes the center of the bounding box around all heart structures, such that the subsequent segmentation CNN can focus on this region. Trained in an end-to-end manner, the segmentation CNN transforms intermediate label predictions to positions of other labels. Thus, the network learns from the relative positions among labels and focuses on anatomically feasible configurations. Results on the MICCAI 2017 Multi-Modality Whole Heart Segmentation (MM-WHS) challenge show that the proposed architecture performs well on the provided CT and MRI training volumes, delivering in a three-fold cross validation an average Dice Similarity Coefficient over all heart substructures of 88.9% and 79.0%, respectively. Moreover, on the MM-WHS challenge test data we rank first for CT and second for MRI with a whole heart segmentation Dice score of 90.8% and 87%, respectively, leading to an overall first ranking among all participants.

Volumetric Reconstruction from a Limited Number of Digitally Reconstructed Radiographs Using CNNs

We propose a method for 3D computed tomography (CT) image reconstruction from 3D digitally reconstructed radiographs (DRR). The 3D DRR images are generated from 2D projection images of the 3D CT image from different angles and used to train a convolutional neural network (CNN). Evaluating with a different number of input DRR images, we compare our resulting 3D CT reconstruction to those of the filtered backprojection (FBP), which represents the standard method for CT image reconstruction. The evaluation shows that our CNN based method is able to decrease the number of projection images necessary to reconstruct the original image without a significant reduction in image quality. This indicates the potential for accurate 3D reconstruction from a lower number of projection images leading to a reduced amount of ionizing radiation exposure during CT image acquisition.

Automatic Artery-Vein Separation from Thoracic CT Images Using Integer Programming

Automated computer-aided analysis of lung vessels has shown to yield promising results for non-invasive diagnosis of lung diseases. In order to detect vascular changes affecting arteries and veins differently, an algorithm capable of identifying these two compartments is needed. We propose a fully automatic algorithm that separates arteries and veins in thoracic computed tomography (CT) images based on two integer programs. The first extracts multiple subtrees inside a graph of vessel paths. The second labels each tree as either artery or vein by maximizing both, the contact surface in their Voronoi diagram, and a measure based on closeness to accompanying bronchi. We evaluate the performance of our automatic algorithm on 10 manual segmentations of arterial and venous trees from patients with and without pulmonary vascular disease, achieving an average voxel based overlap of 94.1% (range: 85.0% – 98.7%), outperforming a recent state-of-the-art interactive method.

Instance Segmentation and Tracking with Cosine Embeddings and Recurrent Hourglass Networks

Different to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time. The network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal video information. Furthermore, we train the network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos. Afterwards, these embeddings are clustered among subsequent video frames to create the final tracked instance segmentations. We evaluate the recurrent hourglass network by segmenting left ventricles in MR videos of the heart, where it outperforms a network that does not incorporate video information. Furthermore, we show applicability of the cosine embedding loss for segmenting leaf instances on still images of plants. Finally, we evaluate the framework for instance segmentation and tracking on six datasets of the ISBI celltracking challenge, where it shows state-of-the-art performance.

Reducing acquisition time for MRI-based forensic age estimation

Radiology-based estimation of a living person’s unknown age has recently attracted increasing attention due to large numbers of undocumented immigrants entering Europe. To avoid the application of X-ray-based imaging techniques, magnetic resonance imaging (MRI) has been suggested as an alternative imaging modality. Unfortunately, MRI requires prolonged acquisition times, which potentially represents an additional stressor for young refugees. To eliminate this shortcoming, we investigated the degree of reduction in acquisition time that still led to reliable age estimates. Two radiologists randomly assessed original images and two sets of retrospectively undersampled data of 15 volunteers (N = 45 data sets) applying an established radiological age estimation method to images of the hand and wrist. Additionally, a neural network-based age estimation method analyzed four sets of further undersampled images from the 15 volunteers (N = 105 data sets). Furthermore, we compared retrospectively undersampled and acquired undersampled data for three volunteers. To assess reliability with increasing degree of undersampling, intra-rater and inter-rater agreement were analyzed computing signed differences and intra-class correlation. While our findings have to be confirmed by a larger prospective study, the results from both radiological and automatic age estimation showed that reliable age estimation was still possible for acquisition times of 15 seconds.