Emma Beauxis-Aussalet

A case study of trust issues in scientific video collections

In-situ video recording of underwater ecosystems is able to provide valuable information for biology research and natural resources management, e.g. changes in species abundance. Searching the videos manually, however, requires costly human effort. Our video analysis tool supports the key task of counting different species of fish, allowing marine biologists to query the video collection without watching the videos. To be suitable for scientific research on changes in species abundance, the video data must include data provenance information that reflects the potential biases introduced through the video processing.In order to trust the analyses made by the system, we need to provide expert users with sufficient information to allow them to interpret these potential biases. We conducted two user studies to design a user interface that includes data provenance information. Our qualitative analysis discusses the support for understanding the reliability of video analysis, and trusting the results it produces. Our main finding is that disclosing details about the video processing and provenance data allows biologists to compare the results with their traditional statistical methods, thus increasing their trust in the results.

Uncertainty-aware estimation of population abundance using machine learning

Machine learning is widely used for mining collections, such as images, sounds, or texts, by classifying their elements into categories. Automatic classification based on supervised learning requires groundtruth datasets for modeling the elements to classify, and for testing the quality of the classification. Because collecting groundtruth is tedious, a method for estimating the potential errors in large datasets based on limited groundtruth is needed. We propose a method that improves classification quality by using limited groundtruth data to extrapolate the potential errors in larger datasets. It significantly improves the counting of elements per class. We further propose visualization designs for understanding and evaluating the classification uncertainty. They support end-users in considering the impact of potential misclassifications for interpreting the classification output. This work was developed to address the needs of ecologists studying fish population abundance using computer vision, but generalizes to a larger range of applications. Our method is largely applicable for a variety of Machine learning technologies, and our visualizations further support their transfer to end-users.

Ergonomic Considerations For The Design And The Evaluation Of Uncertain Data Visualizations

Uncertainty impacts many crucial issues the world is facing today – from climate change prediction, to scientific modelling, to the interpretation of medical data. Decisions typically rely on data which can be aggregated from different sources and further transformed using a variety of algorithms and models. Such data processing pipelines involve different types of uncertainty. As visual data representations are able to mediate between human cognition and computational models, a trustworthy conveyance of data characteristics requires effective representations of uncertainty which take productivity and cognitive abilities, as important human factors, into account. We summarize findings resulting from prior work on interactive uncertainty visualizations. Subsequently, an evaluation study is presented which investigates the effect of different visualizations of uncertain data on users’ efficiency (time, error rate) and subjectively perceived cognitive load. A table, a static graphic, and an interactive graphic containing uncertain data were compared. The results of an online study (N = 146) showed a significant difference in the task completion time between the visualization type, while there are no significant differences in error rate. A non-parametric K-W test found a significant difference in subjective cognitive load [H (2) = 7.39, p < 0.05]. Subjectively perceived cognitive load was lower for static and interactive graphs than for the numerical table. Given that the shortest task completion time was produced by a static graphic representation, we recommend this for use cases in which uncertain data are to be used time-efficiently.

Understanding Uncertainty Issues in the Exploration of Fish Counts

Several data analysis steps are required for understanding computer vision results and drawing conclusions about the actual trends in the fish populations. Particular attention must be drawn to the potential errors that can impact the scientific validity of end-results. This chapter discusses the means for ecologists to investigate the uncertainty in computer vision results. We address a set of uncertainty factors identified by interviewing both ecology and computer vision experts, as discussed in Chap. 2. We investigate state-of-the-art methods to specify these uncertainty factors. We identify issues with conveying the results of ground-truth evaluation methods to end-users who are not familiar with computer vision technology, and we present a novel visualization design addressing these issues. Finally, we discuss the uncertainty factors for which evaluation methods require further research.

User Information Needs

Computer vision technology has been considered in marine ecology research as a innovative, promising data collection method. It contrasts with traditional practices in the information that is collected, and its inherent errors and biases. Ecology research is based on the analysis of biological characteristics (e.g., species, size, age, distribution, density, behaviors), while computer vision focuses on visual characteristics that are not necessarily related to biological concepts (e.g., contours, contrasts, color histograms, background model). It is challenging for ecologists to assess the scientific validity of surveys performed on the basis of image analysis. User information needs may not be fully addressed by image features, or may not be reliable enough. We gathered user requirements for supporting ecology research based on computer vision technologies, and identified those we can address within the Fish4Knowledge project. We particularly investigated the uncertainty inherent to computer vision technology, and the means to support users in considering uncertainty when interpreting information on fish populations. We introduce potential biases and uncertainty factors that can impact the scientific validity of interpretations drawn from computer vision results. We conclude by introducing potential approaches for providing users with evaluations of the uncertainties introduced at each information processing step.

Multifactorial uncertainty assessment for monitoring population abundance using computer vision

Computer vision enables in-situ monitoring of animal populations at a lower cost and with less ecosystem disturbance than with human observers. However, computer vision uncertainty may not be fully understood by end-users, and the uncertainty assessments performed by technology experts may not fully address end-user needs. This knowledge gap can yield misinterpretations of computer vision data, and trust issues impeding the transfer of valuable technologies. We bridge this gap with a user-centered analysis of the uncertainty issues. Key uncertainty factors, and their interactions, are identified from the perspective of a core task in ecology research and beyond: counting individuals from different classes. We highlight factors for which uncertainty assessment methods are currently unavailable. The remaining uncertainty assessment methods are not interoperable. Hence it is currently difficult to assess the combined results of multiple uncertainty factors, and their impact on end-user counting tasks. We propose a framework for assessing the multifactorial uncertainty propagation along the data processing pipeline. It integrates methods from both computer vision and ecology domains, and aims at supporting the statistical analysis of abundance trends for population monitoring. Our typology of uncertainty factors and our assessment methods were drawn from interviews with marine ecology and computer vision experts, and from prior work for a fish monitoring application. Our findings contribute to enabling scientific research based on computer vision.

Multi-Purpose Exploration of Uncertain Data for the Video Monitoring of Ecosystems

Computer Vision is a promising technique for in-situ monitoring of ecosystems. It is non-intrusive and costeffective compared to sending human observers. Automatic animal detection and species recognition support the study of population dynamics and species composition, i.e., the evolution of species populations’ size. Fixed cameras support continuous data collection, which can serve a variety of ecology research. Prior to in-depth data analysis, ecologists need to familiarise with the dataset, and with the limitations of video technologies. We propose an interactive visualization system for exploring the video data. It addresses user needs for i) eliciting information of interest for specific studies; and ii) identifying the uncertainty factors inherent to video technologies. We discuss generalisable interaction principes and illustrate them with screenshots of an online prototype.