Diane H. Theriault

A protocol and calibration method for accurate multi-camera field videography

Stereo videography is a powerful technique for quantifying the kinematics and behavior of animals, but it can be challenging to use in an outdoor field setting. We here present a workflow and associated software for performing calibration of cameras placed in a field setting and estimating the accuracy of the resulting stereoscopic reconstructions. We demonstrate the workflow through example stereoscopic reconstructions of bat and bird flight. We provide software tools for planning experiments and processing the resulting calibrations that other researchers may use to calibrate their own cameras. Our field protocol can be deployed in a single afternoon, requiring only short video clips of light, portable calibration objects.

Seasonal variation in colony size of Brazilian free-tailed bats at Carlsbad Cavern based on thermal imaging

Abstract The colony of Brazilian free-tailed bats (Tadarida brasiliensis) at Carlsbad Cavern, New Mexico, is a well-known example of this highly gregarious and conspicuous species in North America. For nearly a century researchers have tried to estimate the size of this colony, but different census methods and lack of repeatability have resulted in questionable estimates that have given rise to poorly understood but highly popularized, long-term population trends for this migratory species. In this study we present accurate seasonal estimates of colony size based on a recently developed census method—thermal infrared imaging and computer vision analysis. The size of the colony was estimated several times monthly from March through October 2005. Our estimates range from 67,602 to 793,838 bats, values that are orders of magnitude lower than the largest historic estimates. Consecutive estimates of nightly emergences show fluctuations of as many as 291,000 individuals, indicating that colony composition is considerably more dynamic than previously thought. Our results, combined with a quantitative analysis of emergence behavior, question the validity of early historic estimates that millions of bats once roosted in this cave and suggest that the long-term pattern of decline reported for this species might not be as severe as currently thought.

A Thermal Infrared Video Benchmark for Visual Analysis

We hereby publish a new thermal infrared video benchmark, called TIV, for various visual analysis tasks, which include single object tracking in clutter, multi-object tracking in single or multiple views, analyzing motion patterns of large groups, and censusing wild animals in flight. Our data describe real world scenarios, such as bats emerging from their caves in large numbers, a crowded street view during a marathon competition, and students walking through an atrium during class break. We also introduce baseline methods and evaluation protocols for these tasks. Our TIV benchmark enriches and diversifies video data sets available to the research community with thermal infrared footage, which poses new and challenging video analysis problems. We hope the TIV benchmark will help the community to better understand these interesting problems, generate new ideas, and value it as a testbed to compare solutions.

How to Collect Segmentations for Biomedical Images? A Benchmark Evaluating the Performance of Experts, Crowdsourced Non-experts, and Algorithms

Analyses of biomedical images often rely on demarcating the boundaries of biological structures (segmentation). While numerous approaches are adopted to address the segmentation problem including collecting annotations from domain-experts and automated algorithms, the lack of comparative benchmarking makes it challenging to determine the current state-of-art, recognize limitations of existing approaches, and identify relevant future research directions. To provide practical guidance, we evaluated and compared the performance of trained experts, crowd sourced non-experts, and algorithms for annotating 305 objects coming from six datasets that include phase contrast, fluorescence, and magnetic resonance images. Compared to the gold standard established by expert consensus, we found the best annotators were experts, followed by non-experts, and then algorithms. This analysis revealed that online paid crowd sourced workers without domain-specific backgrounds are reliable annotators to use as part of the laboratory protocol for segmenting biomedical images. We also found that fusing the segmentations created by crowd sourced internet workers and algorithms yielded improved segmentation results over segmentations created by single crowd sourced or algorithm annotations respectively. We invite extensions of our work by sharing our data sets and associated segmentation annotations (http://www.cs.bu.edu/~betke/Biomedical Image Segmentation).

Cell morphology classification and clutter mitigation in phase-contrast microscopy images using machine learning

We propose using machine learning techniques to analyze the shape of living cells in phase-contrast microscopy images. Large scale studies of cell shape are needed to understand the response of cells to their environment. Manual analysis of thousands of microscopy images, however, is time-consuming and error-prone and necessitates automated tools. We show how a combination of shape-based and appearance-based features of fibroblast cells can be used to classify their morphological state, using the Adaboost algorithm. The classification accuracy of our method approaches the agreement between two expert observers. We also address the important issue of clutter mitigation by developing a machine learning approach to distinguish between clutter and cells in time-lapse microscopy image sequences.

Study of bat flight behavior by combining thermal image analysis with a LiDAR forest reconstruction

The nature of forest structure plays an important role in the study of foraging behaviors of bats. In this study, we demonstrate a new combined methodology that uses both thermal imaging technology and a ground-based LiDAR system to record and reconstruct Eptesicus fuscus (big brown bats) flight trajectories in three-dimensional (3-D) space. The combination of the two 3-D datasets provided a fine-scale reconstruction of the flight characteristics adjacent to and within the forests. A 3-D forest reconstruction, assembled from nine Echidna Validation Instrument LiDAR scans over the 1 ha site area, provided the essential environmental variables for the study of bat foraging behaviors, such as the canopy height, terrain, location of the obstacles, and canopy openness at a bat roosting and maternity site in Petersham, Massachusetts. Flight trajectories of 24 bats were recorded over the 25 m × 37.5 m region within the LiDAR forest reconstruction area. The trajectories were reconstructed using imaging data from multiple FLIR ThermoVision SC8000 cameras and were co-registered to the 3-D forest reconstruction. Twenty-four of these flight trajectories were categorized into four different behavior groups according to velocity and altitude analysis of the flight trajectories. Initial results showed that although all bats were guided by echolocation and avoided hitting a tree that was in all of their flight paths, different bats chose different flight routes. This study is an initial demonstration of the power of coupling thermal image analysis and LiDAR forest reconstructions. Our goal was to break ground for future ecological studies, where more extensive flight trajectories of bats can be coupled with the canopy reconstructions to better establish responses of bats to different habitat characteristics and clutter, which includes both static (trees) and dynamic (other bats) obstacles.

Collision avoidance in biological systems using collision cones

The focus of the work in this paper is the comparison of a mathematical deconfliction algorithm to biological data in a range of species that demonstrate agile flight beyond the current capabilities of engineered systems. The algorithm was tailored to two coordinate systems, global and body relative, and two velocity changing criteria, constant and variable speed. Three species of animals were considered: fish, birds and bats. Overall, strong correlations were found between the data and the algorithm in two of the species with data indicating a bias toward a body-fixed coordinate system with variable speed maneuvering. Results also suggested future development of a fully three dimensional algorithm rather than the planar version considered here.

Perceptual modalities guiding bat flight in a native habitat

Flying animals accomplish high-speed navigation through fields of obstacles using a suite of sensory modalities that blend spatial memory with input from vision, tactile sensing, and, in the case of most bats and some other animals, echolocation. Although a good deal of previous research has been focused on the role of individual modes of sensing in animal locomotion, our understanding of sensory integration and the interplay among modalities is still meager. To understand how bats integrate sensory input from echolocation, vision, and spatial memory, we conducted an experiment in which bats flying in their natural habitat were challenged over the course of several evening emergences with a novel obstacle placed in their flight path. Our analysis of reconstructed flight data suggests that vision, echolocation, and spatial memory together with the possible exercise of an ability in using predictive navigation are mutually reinforcing aspects of a composite perceptual system that guides flight. Together with the recent development in robotics, our paper points to the possible interpretation that while each stream of sensory information plays an important role in bat navigation, it is the emergent effects of combining modalities that enable bats to fly through complex spaces.

Using collision cones to assess biological deconfliction methods

Biological systems consistently outperform autonomous systems governed by engineered algorithms in their ability to reactively avoid collisions. To better understand this discrepancy, a collision avoidance algorithm was applied to frames of digitized video trajectory data from bats, swallows and fish (Myotis velifer, Petrochelidon pyrrhonota and Danio aequipinnatus). Information available from visual cues, specifically relative position and velocity, was provided to the algorithm which used this information to define collision cones that allowed the algorithm to find a safe velocity requiring minimal deviation from the original velocity. The subset of obstacles provided to the algorithm was determined by the animals sensing range in terms of metric and topological distance. The algorithmic calculated velocities showed good agreement with observed biological velocities, indicating that the algorithm was an informative basis for comparison with the three species and could potentially be improved for engineered applications with further study.

Perception and Steering Control in Paired Bat Flight

Animals within groups need to coordinate their reactions to perceived environmental features and to each other in order to safely move from one point to another. This paper extends our previously published work on the flight patterns of Myotis velifer that have been observed in a habitat near Johnson City, Texas. Each evening, these bats emerge from a cave in sequences of small groups that typically contain no more than three or four individuals, and they thus provide ideal subjects for studying leader-follower behaviors. By analyzing the flight paths of a group of M. velifer, the data show that the flight behavior of a follower bat is influenced by the flight behavior of a leader bat in a way that is not well explained by existing pursuit laws, such as classical pursuit, constant bearing and motion camouflage. Thus we propose an alternative steering law based on virtual loom, a concept we introduce to capture the geometrical configuration of the leader-follower pair. It is shown that this law may be integrated with our previously proposed vision-enabled steering laws to synthesize trajectories, the statistics of which fit with those of the bats in our data set. The results suggest that bats use perceived information of both the environment and their neighbors for navigation.