Tahir Nawaz

A Protocol for Evaluating Video Trackers Under Real-World Conditions

The absence of a commonly adopted performance evaluation framework is hampering advances in the design of effective video trackers. In this paper, we present a single-score evaluation measure and a protocol to objectively compare trackers. The proposed measure evaluates tracking accuracy and failure, and combines them for both summative and formative performance assessment. The proposed protocol is composed of a set of trials that evaluate the robustness of trackers on a range of test scenarios representing several real-world conditions. The protocol is validated on a set of sequences with a diversity of targets (head, vehicle and person) and challenges (occlusions, background clutter, pose changes and scale changes) using six state-of-the-art trackers, highlighting their strengths and weaknesses on more than 187000 frames. The software implementing the protocol and the evaluation results are made available online and new results can be included, thus facilitating the comparison of trackers.

Measures of Effective Video Tracking

To evaluate multitarget video tracking results, one needs to quantify the accuracy of the estimated target-size and the cardinality error as well as measure the frequency of occurrence of ID changes. In this paper, we survey existing multitarget tracking performance scores and, after discussing their limitations, we propose three parameter-independent measures for evaluating multitarget video tracking. The measures consider target-size variations, combine accuracy and cardinality errors, quantify long-term tracking accuracy at different accuracy levels, and evaluate ID changes relative to the duration of the track in which they occur. We conduct an extensive experimental validation of the proposed measures by comparing them with existing ones and by evaluating four state-of-the-art trackers on challenging real-world publicly-available data sets. The software implementing the proposed measures is made available online to facilitate their use by the research community.

Trajectory clustering for motion pattern extraction in aerial videos

We present an end-to-end approach for trajectory clustering from aerial videos that enables the extraction of motion patterns in urban scenes. Camera motion is first compensated by mapping object trajectories on a reference plane. Then clustering is performed based on statistics from the Discrete Wavelet Transform coefficients extracted from the trajectories. Finally, motion patterns are identified by distance minimization from the centroids of the trajectory clusters. The experimental validation on four datasets shows the effectiveness of the proposed approach in extracting trajectory clusters. We also make available two new real-world aerial video datasets together with the estimated object trajectories and ground-truth cluster labeling.

ViTBAT: Video tracking and behavior annotation tool

Reliable and repeatable evaluation of low-level (tracking) and high-level (behavior analysis) vision tasks require annotation of ground-truth information in videos. Depending on the scenarios, ground-truth annotation may be required for individual targets and/or groups of targets. Unlike the existing tools that generally allow an explicit annotation for individual targets only, we propose a tool that enables an explicit annotation both for individual targets and groups of targets for the tracking and behavior recognition tasks together with effective visualization features. Whether for individuals or groups, the tool allows labeling of their states and behaviors manually or semi-automatically through a simple and friendly user interface in a time-efficient manner. Based on a subjective assessment, the proposed tool is found to be more effective than the well-known ViPER tool on a series of defined criteria. A dedicated website makes the tool publicly available for the community.

Tracking performance evaluation on PETS 2015 Challenge datasets

This paper presents a quantitative evaluation of a tracking system on PETS 2015 Challenge datasets using well-established performance measures. Using the existing tools, the tracking system implements an end-to-end pipeline that include object detection, tracking and post-processing stages. The evaluation results are presented on the provided sequences of both ARENA and P5 datasets of PETS 2015 Challenge. The results show an encouraging performance of the tracker in terms of accuracy but a greater tendency of being prone to cardinality error and ID changes on both datasets. Moreover, the analysis show a better performance of the tracker on visible imagery than on thermal imagery.

User-centric, embedded vision-based human monitoring: A concept and a healthcare use case

In an Internet of Things (IoT) camera-based monitoring application the transmission of images away from the video sensors for processing poses security and privacy risks. Hence, there is a need for an advanced trusted user-centric monitoring system that pushes the application of security and privacy protection closer to the sensor itself and which enables an enhanced control on data privacy. To this end, this white paper proposes a new approach that involves sensor edge computing to enable sensor-level security and privacy protection and allows observed individuals to interact and control their data without impacting on the quality of the data for further processing. Overall, an IoT vision system is presented that employs a network of fixed embedded cameras in a highly trusted manner, possessing both privacy-protecting and data security features. As a potential application, we discuss an Ambient Assisted Living (AAL) healthcare use case demanding privacy and security for outpatients.

An annotation-free method for evaluating privacy protection techniques in videos

While several privacy protection techniques are presented in the literature, they are not complemented with an established objective evaluation method for their assessment and comparison. This paper proposes an annotation-free evaluation method that assesses the two key aspects of privacy protection that are privacy and utility. Unlike some existing methods, the proposed method does not rely on the use of subjective judgements and does not assume a specific target type in the image data. The privacy aspect is quantified as an appearance similarity and the utility aspect is measured as a structural similarity between the original raw image data and the privacy-protected image data. We performed an extensive experimentation using six challenging datasets (including two new ones) to demonstrate the effectiveness of the evaluation method by providing a performance comparison of four state-of-the-art privacy protection techniques.

Effective evaluation of privacy protection techniques in visible and thermal imagery

Abstract. Privacy protection may be defined as replacing the original content in an image region with a (less intrusive) content having modified target appearance information to make it less recognizable by applying a privacy protection technique. Indeed, the development of privacy protection techniques also needs to be complemented with an established objective evaluation method to facilitate their assessment and comparison. Generally, existing evaluation methods rely on the use of subjective judgments or assume a specific target type in image data and use target detection and recognition accuracies to assess privacy protection. An annotation-free evaluation method that is neither subjective nor assumes a specific target type is proposed. It assesses two key aspects of privacy protection: “protection” and “utility.” Protection is quantified as an appearance similarity, and utility is measured as a structural similarity between original and privacy-protected image regions. We performed an extensive experimentation using six challenging datasets (having 12 video sequences), including a new dataset (having six sequences) that contains visible and thermal imagery. The new dataset is made available online for the community. We demonstrate effectiveness of the proposed method by evaluating six image-based privacy protection techniques and also show comparisons of the proposed method over existing methods.

Temporally stable feature clusters for maritime object tracking in visible and thermal imagery

This paper describes a new approach to detect and track maritime objects in real time. The approach particularly addresses the highly dynamic maritime environment, panning cameras, target scale changes, and operates on both visible and thermal imagery. Object detection is based on agglomerative clustering of temporally stable features. Object extents are first determined based on persistence of detected features and their relative separation and motion attributes. An explicit cluster merging and splitting process handles object creation and separation. Stable object clusters are tracked frame-to-frame. The effectiveness of the approach is demonstrated on four challenging real-world public datasets.

Assessing tracking assessment measures

We propose a methodology to quantitatively compare the relative performance of tracking evaluation measures. The proposed methodology is based on determining the probabilistic agreement between tracking result decisions made by measures and those made by humans. We use tracking results on publicly available datasets with different target types and varying challenges, and collect the judgments of 90 skilled, semi-skilled and unskilled human subjects using a web-based performance assessment test. The analysis of the agreements allows us to highlight the variation in performance of the different measures and the most appropriate ones for the various stages of tracking performance evaluation.