Elisa Drelie Gelasca

A biosegmentation benchmark for evaluation of bioimage analysis methods

BackgroundWe present a biosegmentation benchmark that includes infrastructure, datasets with associated ground truth, and validation methods for biological image analysis. The primary motivation for creating this resource comes from the fact that it is very difficult, if not impossible, for an end-user to choose from a wide range of segmentation methods available in the literature for a particular bioimaging problem. No single algorithm is likely to be equally effective on diverse set of images and each method has its own strengths and limitations. We hope that our benchmark resource would be of considerable help to both the bioimaging researchers looking for novel image processing methods and image processing researchers exploring application of their methods to biology.ResultsOur benchmark consists of different classes of images and ground truth data, ranging in scale from subcellular, cellular to tissue level, each of which pose their own set of challenges to image analysis. The associated ground truth data can be used to evaluate the effectiveness of different methods, to improve methods and to compare results. Standard evaluation methods and some analysis tools are integrated into a database framework that is available online at http://bioimage.ucsb.edu/biosegmentation/.ConclusionThis online benchmark will facilitate integration and comparison of image analysis methods for bioimages. While the primary focus is on biological images, we believe that the dataset and infrastructure will be of interest to researchers and developers working with biological image analysis, image segmentation and object tracking in general.

Evaluation and benchmark for biological image segmentation

This paper describes ongoing work on creating a benchmarking and validation dataset for biological image segmentation. While the primary target is biological images, we believe that the dataset would be of help to researchers working in image segmentation and tracking in general. The motivation for creating this resource comes from the observation that while there are a large number of effective segmentation methods available in the research literature, it is difficult for the application scientists to make an informed choice as to what methods would work for her particular problem. No one single tool exists that is effective on a diverse set of application contexts and different methods have their own strengths and limitations. We describe below three different classes of data, ranging in scale from subcellular to cellular to tissue level images, each of which pose their own set of challenges to image analysis. Of particular value to the image processing researchers is that the data comes with associated ground truth information that can be used to evaluate the effectiveness of different methods. The analysis and evaluation are also integrated into a database framework that is available online at http://dough.ece.ucsb.edu.

Objective evaluation of segmentation quality using spatio-temporal context

In this paper, we propose an automatic method for the objective evaluation of segmentation results. The method is based on computing the deviation of the segmentation results from a reference segmentation. The discrepancy between two results is weighted based on spatial and temporal contextual information, by taking into account the way humans perceive visual information. The metric is useful for applications where the final judge of the quality is a human observer or the results of segmentation are otherwise processed in a human-like fashion. The proposed evaluation has been applied both to automatically provide a ranking among different segmentation algorithms and to optimally set the parameters of a given algorithm.

Objective evaluation of the perceptual quality of 3D watermarking

In this paper an objective metric to measure the perceptual quality of watermarked 3D meshes is presented. The metric, which is based on a black-box approach, relies on the measurement of the roughness of 3D meshes before and after the insertion of the watermark. To calibrate the metric and to validate it, a set of psychovisual experiments has been carried out. Due to the lack of prior work in this field, a new methodology for the subjective evaluation of the quality of watermarked 3D objects is introduced. The validity of the proposed metric has been tested against a number of different 3D watermarking algorithms, showing an excellent match with the subjective evaluation of the quality stemming from the psychovisual experiments.

Towards Perceptually Driven Segmentation Evaluation Metrics

To be reliable, an automatic segmentation evaluation metric has to be validated by subjective tests. In this paper, a formal protocol for subjective tests for segmentation quality assessment is presented. The most common artifacts produced by segmentation algorithms are identified and an extensive analysis of their effects on the perceived quality is performed. A psychophysical experiment was performed to assess the quality of video with segmentation errors. The results show how an objective segmentation evaluation metric can be defined as a function of various error types.

Perceptual quality assessment for video watermarking

The reliable evaluation of the performance of watermarking algorithms is difficult. An important aspect in this process is the assessment of the visibility of the watermark. We address this issue and propose a methodology for evaluating the visual quality of watermarked video. Using a software tool that measures different types of perceptual video artifacts, we determine the most relevant impairments and design the corresponding objective metrics. We demonstrate their performance through subjective experiments on several different watermarking algorithms and video sequences.

On Evaluating Video Object Segmentation Quality: A Perceptually Driven Objective Metric

The task of extracting objects in video sequences emerges in many applications such as object-based video coding (e.g., MPEG-4) and content-based video indexing and retrieval (e.g., MPEG-7). The MPEG-4 standard provides specifications for the coding of video objects, but does not address the problem of how to extract foreground objects in image sequences. Therefore, for specific applications, evaluating the quality of foreground/background segmentation results is necessary to allow for an appropriate selection of segmentation algorithms and for tuning their parameters for optimal performance. Many segmentation algorithms have been proposed along with a number of evaluation criteria. Nevertheless, formal psychophysical experiments evaluating the quality of different video foreground object segmentation results have not yet been conducted. In this paper, a generic framework for both subjective and objective segmentation quality evaluation is presented. An objective quality assessment method for segmentation evaluation is derived on the basis of perceptual factors through subjective experiments. The performance of the proposed method is shown on different state-of-the-art foreground/background segmentation algorithms and our method is compared to other objective methods which do not include perceptual factors. Moreover, on the basis of subjective results, weighting strategies are introduced into the proposed metric to meet the specificity of different segmentation applications e.g., video compression, video surveillance and mixed reality. Experimental results confirm the efficiency of the proposed approach.

A Framework for Evaluating Video Object Segmentation Algorithms

Segmentation of moving objects in image sequences plays an important role in video processing and analysis. Evaluating the quality of segmentation results is necessary to allow the appropriate selection of segmentation algorithms and to tune their parameters for optimal performance. Many segmentation algorithms have been proposed along with a number of evaluation criteria. Nevertheless, no psychophysical experiments evaluating the quality of different video object segmentation results have been conducted. In this paper, a generic framework for segmentation quality evaluation is presented. A perceptually driven automatic method for segmentation evaluation is proposed and compared against an existing approach. Moreover, on the basis of subjective results, perceptual factors are introduced into the novel objective metric to meet the specificity of different segmentation applications such as video compression. Experimental results confirm the efficiency of the proposed evaluation criteria.

Intuitive strategy for parameter setting in video segmentation

In this paper, we propose an original framework for an intuitive tuning of parameters in image and video segmentation algorithms. The proposed framework is very flexible and generic and does not depend on a specific segmentation algorithm, a particular evaluation metric, or a specific optimization approach, which are the three main components of its block diagram. This framework requires a manual segmentation input provided by a human operator as he/she would have performed intuitively. This input allows the framework to search for the optimal set of parameters which will provide results similar to those obtained by manual segmentation. On one hand, this allows researchers and designers to quickly and automatically find the best parameters in the segmentation algorithms they have developed. It helps them to better understand the degree of importance of each parameters value on the final segmentation result. It also identifies the potential of the segmentation algorithm under study in terms of best possible performance level. On the other hand, users and operators of systems with segmentation components, can efficiently identify the optimal sets of parameters for different classes of images or video sequences. In a large extent, this optimization can be performed without a deep understanding of the underlying algorithm, which would facilitate the exploitations and optimizations in real applications by non-experts in segmentation. A specific implementation of the proposed framework was obtained by adopting a video segmentation algorithm invariant to shadows as segmentation component, a full reference segmentation quality metric based on a perceptually motivated spatial context, as the evaluation component, and a down-hill simplex method, as optimization component. Simulation results on various test sequences, covering a representative set of indoor and ourdoor video, show that optimal set of parameters can be obtained efficiently and largely improve the results obtained when compared to a simple implementation of the same segmentation algorithm with ad-hoc parameter setting strategy.

Annoyance of spatio-temporal artifacts in segmentation quality assessment [video sequences]

This paper describes the results of a series of subjective experiments that investigated the annoyance caused by the most common artifacts present in segmented video sequences. Various types of artifacts were inserted into a reference segmented video, considered as ideal, and shown to our test subjects. The artifacts varied in their location, size, appearance and duration. Annoyance of segmentation artifacts are found to be tied up with their intrinsic characteristics (e.g., size, position) but only weakly related to the video content. The results identify the characteristics that should be taken into account in the design of a perceptually driven objective metric.