Derek R. Magee

A Nonlinear Mapping Approach to Stain Normalization in Digital Histopathology Images Using Image-Specific Color Deconvolution

Histopathology diagnosis is based on visual examination of the morphology of histological sections under a microscope. With the increasing popularity of digital slide scanners, decision support systems based on the analysis of digital pathology images are in high demand. However, computerized decision support systems are fraught with problems that stem from color variations in tissue appearance due to variation in tissue preparation, variation in stain reactivity from different manufacturers/batches, user or protocol variation, and the use of scanners from different manufacturers. In this paper, we present a novel approach to stain normalization in histopathology images. The method is based on nonlinear mapping of a source image to a target image using a representation derived from color deconvolution. Color deconvolution is a method to obtain stain concentration values when the stain matrix, describing how the color is affected by the stain concentration, is given. Rather than relying on standard stain matrices, which may be inappropriate for a given image, we propose the use of a color-based classifier that incorporates a novel stain color descriptor to calculate image-specific stain matrix. In order to demonstrate the efficacy of the proposed stain matrix estimation and stain normalization methods, they are applied to the problem of tumor segmentation in breast histopathology images. The experimental results suggest that the paradigm of color normalization, as a preprocessing step, can significantly help histological image analysis algorithms to demonstrate stable performance which is insensitive to imaging conditions in general and scanner variations in particular.

Toward routine use of 3D histopathology as a research tool.

Three-dimensional (3D) reconstruction and examination of tissue at microscopic resolution have significant potential to enhance the study of both normal and disease processes, particularly those involving structural changes or those in which the spatial relationship of disease features is important. Although other methods exist for studying tissue in 3D, using conventional histopathological features has significant advantages because it allows for conventional histopathological staining and interpretation techniques. Until now, its use has not been routine in research because of the technical difficulty in constructing 3D tissue models. We describe a novel system for 3D histological reconstruction, integrating whole-slide imaging (virtual slides), image serving, registration, and visualization into one user-friendly package. It produces high-resolution 3D reconstructions with minimal user interaction and can be used in a histopathological laboratory without input from computing specialists. It uses a novel method for slice-to-slice image registration using automatic registration algorithms custom designed for both virtual slides and histopathological images. This system has been applied to >300 separate 3D volumes from eight different tissue types, using a total of 5500 virtual slides comprising 1.45 TB of primary image data. Qualitative and quantitative metrics for the accuracy of 3D reconstruction are provided, with measured registration accuracy approaching 120 μm for a 1-cm piece of tissue. Both 3D tissue volumes and generated 3D models are presented for four demonstrator cases.

An augmented reality simulator for ultrasound guided needle placement training

Details are presented of a low cost augmented-reality system for the simulation of ultrasound guided needle insertion procedures (tissue biopsy, abscess drainage, nephrostomy etc.) for interventional radiology education and training. The system comprises physical elements; a mannequin, a mock ultrasound probe and a needle, and software elements; generating virtual ultrasound anatomy and allowing data collection. These two elements are linked by a pair of magnetic 3D position sensors. Virtual anatomic images are generated based on anatomic data derived from full body CT scans of live humans. Details of the novel aspects of this system are presented including; image generation, registration and calibration.

Detecting lameness using ‘Re-sampling Condensation’ and ‘multi-stream cyclic hidden Markov models’

A system for the tracking and classification of livestock movements is presented. The combined ‘tracker-classifier’ scheme is based on a variant of Isard and Blakes ‘Condensation’ algorithm [Int. J. Comput. Vision (1998) 5] known as ‘Re-sampling Condensation’ in which a second set of samples is taken from each image in the input sequence based on the results of the initial Condensation sampling. This is analogous to a single iteration of a genetic algorithm and serves to incorporate image information in sample location. Re-sampling condensation relies on the variation within the spatial (shape) model being separated into pseudo-independent components (analogous to genes). In the system, a hierarchical spatial model based on a variant of the point distribution model [Proc. Br. Mach. Vision Conf. (1992) 9] is used to model shape variation accurately. Results are presented that show this algorithm gives improved tracking performance, with no computational overhead, over Condensation alone. Separate cyclic hidden Markov models are used to model ‘healthy’ and ‘lame’ movements within the Condensation framework in a competitive manner such that the model best representing the data will be propagated through the image sequence. q 2002 Elsevier Science B.V. All rights reserved.

Towards an architecture for cognitive vision using qualitative spatio-temporal representations and abduction

In recent years there has been increasing interest in constructing cognitive vision systems capable of interpreting the high level semantics of dynamic scenes. Purely quantitative approaches to the task of constructing such systems have met with some success. However, qualitative analysis of dynamic scenes has the advantage of allowing easier generalisation of classes of different behaviours and guarding against the propagation of errors caused by uncertainty and noise in the quantitative data. Our aim is to integrate quantitative and qualitative modes of representation and reasoning for the analysis of dynamic scenes. In particular, in this paper we outline an approach for constructing cognitive vision systems using qualitative spatial-temporal representations including prototypical spatial relations and spatio-temporal event descriptors automatically inferred from input data. The overall architecture relies on abduction: the system searches for explanations, phrased in terms of the learned spatio-temporal event descriptors, to account for the video data.

Protocols from perceptual observations

This paper presents a cognitive vision system capable of autonomously learning protocols from perceptual observations of dynamic scenes. The work is motivated by the aim of creating a synthetic agent that can observe a scene containing interactions between unknown objects and agents, and learn models of these sufficient to act in accordance with the implicit protocols present in the scene. Discrete concepts (utterances and object properties), and temporal protocols involving these concepts, are learned in an unsupervised manner from continuous sensor input alone. Crucial to this learning process are methods for spatio-temporal attention applied to the audio and visual sensor data. These identify subsets of the sensor data relating to discrete concepts. Clustering within continuous feature spaces is used to learn object property and utterance models from processed sensor data, forming a symbolic description. The progol Inductive Logic Programming system is subsequently used to learn symbolic models of the temporal protocols presented in the presence of noise and over-representation in the symbolic data input to it. The models learned are used to drive a synthetic agent that can interact with the world in a semi-natural way. The system has been evaluated in the domain of table-top game playing and has been shown to be successful at learning protocol behaviours in such real-world audio-visual environments.

3D reconstruction of multiple stained histology images.

Context: Three dimensional (3D) tissue reconstructions from the histology images with different stains allows the spatial alignment of structural and functional elements highlighted by different stains for quantitative study of many physiological and pathological phenomena. This has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, and enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications). Methods: This paper evaluates three strategies for 3D reconstruction from sets of two dimensional (2D) histological sections with different stains, by combining methods of 2D multi-stain registration and 3D volumetric reconstruction from same stain sections. Setting and Design: The different strategies have been evaluated on two liver specimens (80 sections in total) stained with Hematoxylin and Eosin (H and E), Sirius Red, and Cytokeratin (CK) 7. Results and Conclusion: A strategy of using multi-stain registration to align images of a second stain to a volume reconstructed by same-stain registration results in the lowest overall error, although an interlaced image registration approach may be more robust to poor section quality.

Investigation of uncertainties in image registration of cone beam CT to CT on an image-guided radiotherapy system

Methods of measuring uncertainties in rigid body image registration of fan beam computed tomography (FBCT) to cone beam CT (CBCT) have been developed for automatic image registration algorithms in a commercial image guidance system (Synergy, Elekta, UK). The relationships between image registration uncertainty and both imaging dose and image resolution have been investigated with an anthropomorphic skull phantom and further measurements performed with patient images of the head. A new metric of target registration error is proposed. The metric calculates the mean distance traversed by a set of equi-spaced points on the surface of a 5 cm sphere, centred at the isocentre when transformed by the residual error of registration. Studies aimed at giving practical guidance on the use of the Synergy automated image registration, including choice of algorithm and use of the Clipbox are reported. The chamfer-matching algorithm was found to be highly robust to the increased noise induced by low-dose acquisitions. This would allow the imaging dose to be reduced from the current clinical norm of 2 mGy to 0.2 mGy without a clinically significant loss of accuracy. A study of the effect of FBCT slice thickness/spacing and CBCT voxel size showed that 2.5 mm and 1 mm, respectively, gave acceptable image registration performance. Registration failures were highly infrequent if the misalignment was typical of normal clinical set-up errors and these were easily identified. The standard deviation of translational registration errors, measured with patient images, was 0.5 mm on the surface of a 5 cm sphere centred on the treatment centre. The chamfer algorithm is suitable for routine clinical use with minimal need for close inspection of image misalignment.

Personalizing Human Video Pose Estimation

We propose a personalized ConvNet pose estimator that automatically adapts itself to the uniqueness of a persons appearance to improve pose estimation in long videos. We make the following contributions: (i) we show that given a few high-precision pose annotations, e.g. from a generic ConvNet pose estimator, additional annotations can be generated throughout the video using a combination of image-based matching for temporally distant frames, and dense optical flow for temporally local frames, (ii) we develop an occlusion aware self-evaluation model that is able to automatically select the high-quality and reject the erroneous additional annotations, and (iii) we demonstrate that these high-quality annotations can be used to fine-tune a ConvNet pose estimator and thereby personalize it to lock on to key discriminative features of the persons appearance. The outcome is a substantial improvement in the pose estimates for the target video using the personalized ConvNet compared to the original generic ConvNet. Our method outperforms the state of the art (including top ConvNet methods) by a large margin on three standard benchmarks, as well as on a new challenging YouTube video dataset. Furthermore, we show that training from the automatically generated annotations can be used to improve the performance of a generic ConvNet on other benchmarks.

Cognitive Vision: Integrating Symbolic Qualitative Representations with Computer Vision

We describe the challenge of combining continuous computer vision techniques and qualitative, symbolic methods to achieve a system capable of cognitive vision. Key to a truly cognitive system, is the ability to learn: to be able to build and use models constructed autonomously from sensory input. In this paper we overview a number of steps we have taken along the route to the construction of such a system, and discuss some remaining challenges.