Bastiaan Johannes Boom

Special issue on animal and insect behaviour understanding in image sequences

Imaging systems are, nowadays, used increasingly in a range of ecological monitoring applications, in particular for biological, fishery, geological and physical surveys. These technologies have improved radically the ability to capture high-resolution images in challenging environments and consequently to manage effectively natural resources. Unfortunately, advances in imaging devices have not been followed by improvements in automated analysis systems, necessary because of the need for timeconsuming and expensive inputs by human observers. This analytical ‘bottleneck’ greatly limits the potentialities of these technologies and increases demand for automatic content analysis approaches to enable proactive provision of analytical information. On the other side, the study of the behaviour by processing visual data has become an active research area in computer vision. The visual information gathered from image sequences is extremely useful to understand the behaviour of the different objects in the scene, as well as how they interact with each other or with the surrounding environment. However, whilst a large number of video analysis techniques have been developed specifically for investigating events and behaviour in human-centred applications, very little attention has been paid to the understanding of other live organisms, such as animals and insects, although a huge amount of video data are routinely recorded, e.g. the Fish4Knowledge project (www. fish4knowledge.eu) or the wide range of nest cams (http:// watch.birds.cornell.edu/nestcams/home/index) continuously monitor, respectively, underwater reef and bird nests (there exist also variants focusing on wolves, badgers, foxes etc.). The automated analysis of visual data in real-life environments for animal and insect behaviour understanding poses several challenges for computer vision researchers

Underwater live fish recognition using a balance-guaranteed optimized tree

Live fish recognition in the open sea is a challenging multi-class classification task. We propose a novel method to recognize fish in an unrestricted natural environment recorded by underwater cameras. This method extracts 66 types of features, which are a combination of color, shape and texture properties from different parts of the fish and reduce the feature dimensions with forward sequential feature selection (FSFS) procedure. The selected features of the FSFS are used by an SVM. We present a Balance-Guaranteed Optimized Tree (BGOT) to control the error accumulation in hierarchical classification and, therefore, achieve better performance. A BGOT of 10 fish species is automatically constructed using the inter-class similarities and a heuristic method. The proposed BGOT-based hierarchical classification method achieves about 4% better accuracy compared to state-of-the-art techniques on a live fish image dataset.

Understanding fish behavior during typhoon events in real-life underwater environments

The study of fish populations in their own natural environment is a task that has usually been tackled in invasive ways which inevitably influenced the behavior of the fish under observation. Recent projects involving the installation of permanent underwater cameras (e.g. the Fish4Knowledge (F4K) project, for the observation of Taiwan’s coral reefs) allow to gather huge quantities of video data, without interfering with the observed environment, but at the same time require the development of automatic processing tools, since manual analysis would be impractical for such amounts of videos. Event detection is one of the most interesting aspects from the biologists’ point of view, since it allows the analysis of fish activity during particular events, such as typhoons. In order to achieve this goal, in this paper we present an automatic video analysis approach for fish behavior understanding during typhoon events. The first step of the proposed system, therefore, involves the detection of “typhoon” events and it is based on video texture analysis and on classification by means of Support Vector Machines (SVM). As part of our behavior understanding efforts, trajectory extraction and clustering have been performed to study the differences in behavior when disruptive events happen. The integration of event detection with fish behavior understanding surpasses the idea of simply detecting events by low-level features analysis, as it supports the full semantic comprehension of interesting events.

Hierarchical classification with reject option for live fish recognition

A live fish recognition system is needed in application scenarios where manual annotation is too expensive, i.e. too many underwater videos. We present a novel balance-enforced optimized tree with reject option (BEOTR) for live fish recognition. It recognizes the top 15 common species of fish and detects new species in an unrestricted natural environment recorded by underwater cameras. The three main contributions of the paper are: (1) a novel hierarchical classification method suited for greatly unbalanced classes, (2) a novel classification-rejection method to clear up decisions and reject unknown classes, (3) an application of the classification method to free swimming fish. This system assists ecological surveillance research, e.g. fish population statistics in the open sea. BEOTR is automatically constructed based on inter-class similarities. Afterwards, trajectory voting is used to eliminate accumulated errors during hierarchical classification and, therefore, achieves better performance. We apply a Gaussian mixture model and Bayes rule as a reject option after the hierarchical classification to evaluate the posterior probability of being a certain species to filter less confident decisions. The proposed BEOTR-based hierarchical classification method achieves significant improvements compared to state-of-the-art techniques on a live fish image dataset of 24,150 manually labelled images from South Taiwan Sea.

Point Light Source Estimation based on Scenes Recorded by a RGB-D camera.

Bastiaan J. Boom1 http://homepages.inf.ed.ac.uk/bboom/ Sergio Orts-Escolano2 http://www.dtic.ua.es/~sorts/ Xin X. Ning1 avener1446@hotmail.com Steven McDonagh1 http://homepages.inf.ed.ac.uk/s0458953/ Peter Sandilands1 http://homepages.inf.ed.ac.uk/s0569500/ Robert B. Fisher1 http://homepages.inf.ed.ac.uk/rbf/ 1 Institute of Perception, Action and Behaviour University of Edinburgh Edinburgh, UK 2 Computer Technology Department University of Alicante Alicante, Spain

Uncertainty-aware estimation of population abundance using machine learning

Machine learning is widely used for mining collections, such as images, sounds, or texts, by classifying their elements into categories. Automatic classification based on supervised learning requires groundtruth datasets for modeling the elements to classify, and for testing the quality of the classification. Because collecting groundtruth is tedious, a method for estimating the potential errors in large datasets based on limited groundtruth is needed. We propose a method that improves classification quality by using limited groundtruth data to extrapolate the potential errors in larger datasets. It significantly improves the counting of elements per class. We further propose visualization designs for understanding and evaluating the classification uncertainty. They support end-users in considering the impact of potential misclassifications for interpreting the classification output. This work was developed to address the needs of ecologists studying fish population abundance using computer vision, but generalizes to a larger range of applications. Our method is largely applicable for a variety of Machine learning technologies, and our visualizations further support their transfer to end-users.

MTAP special issue on methods and tools for ground truth collection in multimedia applications

The importance of having high quality ground truth annotations for a variety of multimedia applications is widely recognised. Indeed, one of the most time-consuming steps in methods’ development is represented by the generation of accurate truth and comparing this truth to the output of applications to provide evidence that the devised methods are performing well in the targeted domain. However, the cost of creating labeled data, which implies asking a human to examine multimedia data thoroughly and provide labels, becomes impractical as datasets to be labeled grow. This can lead to the creation of disparate datasets which are often too small for both learning and evaluating the underlining data distribution. To build up large scale datasets, recently, methods exploiting the collaborative effort of a large population of users annotators (e.g. Labelme, CalTech, Pascal VOC, Trecvid) have been devised. Nevertheless, the creation of a common and large scale ground truth data to train, test and evaluate algorithms for multimedia processing is still a major concern. In particular, the research in ground truth labelling still lacks both in developing user-oriented tools and in automatic methods for supporting annotators in accomplishing their labelling tasks. In fact, tools for ground truth annotation must be user-oriented, providing visual interfaces and methods that are able to guide and speed-up the process of ground truth creation. Under this scenario, multimedia processing methods and collaborative methods play a crucial role. Further, setting up requirements and standards for the creation of multimedia dataset allows other researchers in the field to continue efforts and to contribute to the creation and annotation of multimedia data. This allows researchers to share and extend each others’ work, which is beneficial for the research community.

GMM improves the reject option in hierarchical classification for fish recognition

A reject option in classification is useful to filter less confident decisions of known classes or to detect and remove untrained classes. This paper presents a novel rejection system in a hierarchical classification method for fish species recognition. Since hierarchical methods accumulate errors along the decision path, the rejection system provides an alternative channel to discover misclassified samples at the leaves of the classification hierarchy. This is also applied to probe test samples from new classes. We apply a Gaussian Mixture Model (GMM) to evaluate the posterior probability of testing samples. 2626 dimensions of features, e.g. color and shape and texture properties, from different parts of the fish are computed and normalized. We use forward sequential feature selection (FSFS), which utilizes SVM as a classifier, to select a subset of effective features that distinguishes samples of a given class from others. After learning the mixture models, the reject function is integrated with a Balance-Guaranteed Optimized Tree (BGOT) hierarchical method. We compare three rejection methods. The experimental results demonstrate a reduction in the accumulated errors from hierarchical classification and an improvement in discovering unknown classes.

Adaptive deblurring of surveillance video sequences that deteriorate over time

We present a method for restoring the recordings obtained from surveillance cameras whose quality deteriorates due to dirt or water that gathers on the cameras lens. The method is designed to operate in the surveillance setting and makes use of good quality frames from the beginning of the recorded sequence to remove the blur at later stages caused by the dirty lens. A background subtraction method allows us to obtain a stable background of the scene. Based on this background, a multiframe blind deconvolution algorithm is used to estimate the Point Spread Function (PSF) of the blur. Once the PSF is obtained it can be used to deblur the entire scene. This restoration method was tested on both synthetic and real data with improvements of 15 dB in PSNR being achieved by using clean frames from the beginning of the recorded sequence.

Approximate nearest neighbor search to support manual image annotation of large domain-specific datasets

The annotation of large datasets containing domain-specific images is both time-consuming and difficult. However, currently computer vision and machine learning methods have to deal with ever increasing amounts of data, where annotation of this data is essential. The annotated images allow these kind of methods to learn the variation in large datasets and evaluate methods based on large datasets. This paper presents a method for annotation of domain-specific (fish species) images using approximate nearest neighbor search to retrieve similar fish species in a large set (216,501) of images. The approximate nearest neighbor search allows us to find a ranked set of images in large datasets. Presenting similar images to users allows them to annotate images much more efficiently. In this case, our user interface present these images in such a way that the user does not need to have knowledge of a specific domain to contribute in the annotation of images.