Jens Rittscher

Shape and Appearance Context Modeling

In this work we develop appearance models for computing the similarity between image regions containing deformable objects of a given class in realtime. We introduce the concept of shape and appearance context. The main idea is to model the spatial distribution of the appearance relative to each of the object parts. Estimating the model entails computing occurrence matrices. We introduce a generalization of the integral image and integral histogram frameworks, and prove that it can be used to dramatically speed up occurrence computation. We demonstrate the ability of this framework to recognize an individual walking across a network of cameras. Finally, we show that the proposed approach outperforms several other methods.

A Probabilistic Background Model for Tracking

A new probabilistic background model based on a Hidden Markov Model is presented. The hidden states of the model enable discrimination between foreground, background and shadow. This model functions as a low level process for a car tracker. A particle filter is employed as a stochastic filter for the car tracker. The use of a particle filter allows the incorporation of the information from the low level process via importance sampling. A novel observation density for the particle filter which models the statistical dependence of neighboring pixels based on a Markov random field is presented. The effectiveness of both the low level process and the observation likelihood are demonstrated.

Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue

Limitations on the number of unique protein and DNA molecules that can be characterized microscopically in a single tissue specimen impede advances in understanding the biological basis of health and disease. Here we present a multiplexed fluorescence microscopy method (MxIF) for quantitative, single-cell, and subcellular characterization of multiple analytes in formalin-fixed paraffin-embedded tissue. Chemical inactivation of fluorescent dyes after each image acquisition round allows reuse of common dyes in iterative staining and imaging cycles. The mild inactivation chemistry is compatible with total and phosphoprotein detection, as well as DNA FISH. Accurate computational registration of sequential images is achieved by aligning nuclear counterstain-derived fiducial points. Individual cells, plasma membrane, cytoplasm, nucleus, tumor, and stromal regions are segmented to achieve cellular and subcellular quantification of multiplexed targets. In a comparison of pathologist scoring of diaminobenzidine staining of serial sections and automated MxIF scoring of a single section, human epidermal growth factor receptor 2, estrogen receptor, p53, and androgen receptor staining by diaminobenzidine and MxIF methods yielded similar results. Single-cell staining patterns of 61 protein antigens by MxIF in 747 colorectal cancer subjects reveals extensive tumor heterogeneity, and cluster analysis of divergent signaling through ERK1/2, S6 kinase 1, and 4E binding protein 1 provides insights into the spatial organization of mechanistic target of rapamycin and MAPK signal transduction. Our results suggest MxIF should be broadly applicable to problems in the fields of basic biological research, drug discovery and development, and clinical diagnostics.

Appearance-based person reidentification in camera networks: problem overview and current approaches

Recent advances in visual tracking methods allow following a given object or individual in presence of significant clutter or partial occlusions in a single or a set of overlapping camera views. The question of when person detections in different views or at different time instants can be linked to the same individual is of fundamental importance to the video analysis in large-scale network of cameras. This is the person reidentification problem. The paper focuses on algorithms that use the overall appearance of an individual as opposed to passive biometrics such as face and gait. Methods that effectively address the challenges associated with changes in illumination, pose, and clothing appearance variation are discussed. More specifically, the development of a set of models that capture the overall appearance of an individual and can effectively be used for information retrieval are reviewed. Some of them provide a holistic description of a person, and some others require an intermediate step where specific body parts need to be identified. Some are designed to extract appearance features over time, and some others can operate reliably also on single images. The paper discusses algorithms for speeding up the computation of signatures. In particular it describes very fast procedures for computing co-occurrence matrices by leveraging a generalization of the integral representation of images. The algorithms are deployed and tested in a camera network comprising of three cameras with non-overlapping field of views, where a multi-camera multi-target tracker links the tracks in different cameras by reidentifying the same people appearing in different views.

Simultaneous estimation of segmentation and shape

The main focus of this work is the integration of feature grouping and model based segmentation into one consistent framework. The algorithm is based on partitioning a given set of image features using a likelihood function that is parameterized on the shape and location of potential individuals in the scene. Using a variant of the EM formulation, maximum likelihood estimates of both the model parameters and the grouping are obtained simultaneously. The resulting algorithm performs global optimization and generates accurate results even when decisions can not be made using local context alone. An important feature of the algorithm is that the number of people in the scene is not modeled explicitly. As a result no prior knowledge or assumed distributions are required. The approach is shown to be robust with respect to partial occlusion, shadows, clutter, and can operate over a large range of challenging view angles including those that are parallel to the ground plane. Comparisons with existing crowd segmentation systems are made and the utility of coupling crowd segmentation with a temporal tracking system is demonstrated.

Spatio-temporal cell cycle phase analysis using level sets and fast marching methods

Enabled by novel molecular markers, fluorescence microscopy enables the monitoring of multiple cellular functions using live cell assays. Automated image analysis is necessary to monitor such model systems in a high-throughput and high-content environment. Here, we demonstrate the ability to simultaneously track cell cycle phase and cell motion at the single cell level. Using a recently introduced cell cycle marker, we present a set of image analysis tools for automated cell phase analysis of live cells over extended time periods. Our model-based approach enables the characterization of the four phases of the cell cycle G1, S, G2, and M, which enables the study of the effect of inhibitor compounds that are designed to block the replication of cancerous cells in any of the phases. We approach the tracking problem as a spatio-temporal volume segmentation task, where the 2D slices are stacked into a volume with time as the z dimension. The segmentation of the G2 and S phases is accomplished using level sets, and we designed a model-based shape/size constraint to control the evolution of the level set. Our main contribution is the design of a speed function coupled with a fast marching path planning approach for tracking cells across the G1 phase based on the appearance change of the nuclei. The viability of our approach is demonstrated by presenting quantitative results on both controls and cases in which cells are treated with a cell cycle inhibitor.

Coupled minimum-cost flow cell tracking for high-throughput quantitative analysis.

A growing number of screening applications require the automated monitoring of cell populations in a high-throughput, high-content environment. These applications depend on accurate cell tracking of individual cells that display various behaviors including mitosis, merging, rapid movement, and entering and leaving the field of view. Many approaches to cell tracking have been developed in the past, but most are quite complex, require extensive post-processing, and are parameter intensive. To overcome such issues, we present a general, consistent, and extensible tracking approach that explicitly models cell behaviors in a graph-theoretic framework. We introduce a way of extending the standard minimum-cost flow algorithm to account for mitosis and merging events through a coupling operation on particular edges. We then show how the resulting graph can be efficiently solved using algorithms such as linear programming to choose the edges of the graph that observe the constraints while leading to the lowest overall cost. This tracking algorithm relies on accurate denoising and segmentation steps for which we use a wavelet-based approach that is able to accurately segment cells even in images with very low contrast-to-noise. In addition, the framework is able to measure and correct for microscope defocusing and stage shift. We applied the algorithms on nearly 6000 images of 400,000 cells representing 32,000 tracks taken from five separate datasets, each composed of multiple wells. Our algorithm was able to segment and track cells and detect different cell behaviors with an accuracy of over 99%. This overall framework enables accurate quantitative analysis of cell events and provides a valuable tool for high-throughput biological studies.

An HMM-based segmentation method for traffic monitoring movies

Shadows of moving objects often obstruct robust visual tracking. We propose an HMM-based segmentation method which classifies in real time each pixel or region into three categories: shadows, foreground, and background objects. In the case of traffic monitoring movies, the effectiveness of the proposed method has been proven through experimental results.

Guiding random particles by deterministic search

Among the algorithms developed towards the goal of robust and efficient tracking, two approaches which stand out due to their success are those based on particle filtering and variational approaches. The Bayesian approach led to the development of the particle filter, which performs a random search guided by a stochastic motion model. On the other hand, localising an object can be based on minimising a cost function. This minimum can be found using variational methods. The search paradigms differ in these two methods. One is stochastic and model-driven while the other is deterministic and data-driven. This paper presents a new algorithm to incorporate the strengths of both approaches into one consistent framework. To allow this fusion a smooth, wide likelihood function is constructed, based on a sum-of-squares distance measure and an appropriate sampling scheme is introduced. Based on low-level information this scheme automatically mixes the two methods of search and adapts the computational demands of the algorithm to the difficulty of the problem at hand. The ability to effectively track complex motions without the need for finely tuned motion models is demonstrated.

Detecting and counting people in surveillance applications

A number of surveillance scenarios require the detection and tracking of people. Although person detection and counting systems are commercially available today, there is need for further research to address the challenges of real world scenarios. The focus of this work is the segmentation of groups of people into individuals. One relevant application of this algorithm is people counting. Experiments document that the presented approach leads to robust people counts.