Vinayak Elangovan

A survey of imagery techniques for semantic labeling of human-vehicle interactions in persistent surveillance systems

Understanding and semantic annotation of Human-Vehicle Interactions (HVI) facilitate fusion of Hard sensor (HS) and Human Intelligence (HUMINT) in a cohesive way. By characterization, classification, and discrimination of HVI patterns pertinent threats may be realized. Various Persistent Surveillance System (PSS) imagery techniques have been proposed in the past decade for identifying human interactions with various objects in the environment. Understanding of such interactions facilitates to discover human intentions and motives. However, without consideration of incidental context, reasoning and analysis of such behavioral activities is a very challenging and difficult task. This paper presents a current survey of related publications in the area of context-based Imagery techniques applied for HVI recognition, in particular, it discusses taxonomy and ontology of HVI and presents a summary of reported robust image processing techniques for spatiotemporal characterization and tracking of human targets in urban environments. The discussed techniques include model-based, shape-based and appearance-based techniques employed for identification and classification of objects. A detailed overview of major past research activities related to HVI in PSS with exploitation of spatiotemporal reasoning techniques applied to semantic labeling of the HVI is also presented.

Context-based semantic labeling of human-vehicle interactions in persistent surveillance systems

The improved Situational awareness in Persistent Surveillance Systems (PSS) is an ongoing research effort of the Department of Defense. Most PSS generate huge volume of raw data and they heavily rely on human operators to interpret and inference data in order to detect potential threats. Many outdoor apprehensive activities involve vehicles as their primary source of transportation to and from the scene where a plot is executed. Vehicles are employed to bring in and take out ammunitions, supplies, and personnel. Vehicles are also used as a disguise, hide-out, a meeting place to execute threat plots. Analysis of the Human-Vehicle Interactions (HVI) helps us to identify cohesive patterns of activities representing potential threats. Identification of such patterns can significantly improve situational awareness in PSS. In our approach, image processing technique is used as the primary source of sensing modality. We use HVI taxonomy as a means for recognizing different types of HVI activities. HVI taxonomy may comprise multiple threads of ontological patterns. By spatiotemporal linking of ontological patterns, a HVI pattern is hypothesized to pursue a potential threat situation. The proposed technique generates semantic messages describing ontology of HVI. This paper also discusses a vehicle zoning technique for HVI semantic labeling and demonstrates efficiency and effectiveness of the proposed technique for identifying HVI.

Localization strategies for large-scale airborne deployed wireless sensors

Localization is the process of finding the geometric locations of wireless sensor nodes according to some real or virtual coordinate system. It is an important task when direct measurements of the wireless sensor locations are not available. From the various techniques evolved in localizing sensor nodes, one approach is to use the received signal strength to predict the location of unknown sensing devices. In this paper, passive localization algorithms are developed, presented and tested. The algorithms perform region-based localization of stationary wireless sensors with respect to a frame of reference using received signal strength of the sensors. The reported work is conducted in two phases, theoretical development then simulation and hardware testing. In the first phase, localization algorithms were developed to predict the location of wireless sensor nodes. We categorized localization of sensors in three different classes. In class-I, localization is done for sensors that are in the communication range of at least three head nodes. In class -II, localization is done for sensors in the communication range of two head nodes, and in class-III, localization is done for sensors that are in the communication range of only one head node. In the second phase, the three different categories were tested by simulation then using hardware. A test-bed was established using the crossbow (MICAz) hardware and used to measure the sensors transmission signal strength. Then, the localization software provided with these signal strength as input to predict the location of each wireless sensor nodes. The algorithm developments, the simulation and hardware preliminary test results of the localization algorithms are presented in this paper.

Acoustic semantic labeling and fusion of human-vehicle interactions

Situational awareness in a Persistent Surveillance System (PSS) can be significantly improved by fusion of Data from physical (Hard) sensors and information provided by human observers (as Soft/biological sensors) from the field. One of the major limitations that this trend brings about is, however, the integration and fusion of the sensory data collected from hard sensors along with soft data gathered from human agents in a consistent and cohesive way. This paper presents a proposed approach for semantic labeling of vehicular non-stationary acoustic events in the context of PSS. Two techniques for feature extraction based on discrete wavelet and short-time Fourier transforms are described. A correlation-based classifier is proposed for classifying and semantic labeling of vehicular acoustic events. The presented result demonstrates the proposed solution is both reliable and effective, and can be extended to future PSS applications.

A multi-attribute based methodology for vehicle detection and identification

Robust vehicle detection and identification is required for the intelligent persistent surveillance systems. In this paper, we present a Multi-attribute Vehicle Detection and Identification technique (MVDI) for detection and classification of stationary vehicles. The proposed model uses a supervised Hamming Neural Network (HNN) for taxonomy of shape of the vehicle. Vehicles silhouette features are employed for the training of the HNN from a large array of training vehicle samples in different type, scale, and color variation. Invariant vehicle silhouette attributes are used as features for training of the HNN which is based on an internal Hamming Distance and shape features to determine degree of similarity of a test vehicle against those it’s selectively trained with. Upon detection of class of the vehicle, the other vehicle attributes such as: color and orientation are determined. For vehicle color detection, provincial regions of the vehicle body are used for matching color of the vehicle. For the vehicle orientation detection, the key structural features of the vehicle are extracted and subjected to classification based on color tune, geometrical shape, and tire region detection. The experimental results show the technique is promising and has robustness for detection and identification of vehicle based on their multi-attribute features. Furthermore this paper demonstrates the importance of the vehicle attributes detection towards the identification of Human-Vehicle Interaction events.

Knowledge discovery in group activities through sequential observation analysis

Understanding of Group Activities (GA) has significant applications in civilian and military domains. The process of understanding GA is typically involved with spatiotemporal analysis of multi-modality sensor data. Video imagery is one popular sensing modality that offers rich data, however, data associated with imagery source may become fragmented and discontinued due to a number of reasons (e.g., data transmission, or observation obstructions and occlusions). However, making sense out of video imagery is a real challenge. It requires a proper inference working model capable of analyzing video imagery frame by frame, extract and inference spatiotemporal information pertaining to observations while developing an incremental perception of the GA as they emerge overtime. In this paper, we propose an ontology based GA recognition where three inference Hidden Markov Models (HMM’s) are used for predicting group activities taking place in outdoor environments and different task operational taxonomy. The three competing models include: a concatenated HMM, a cascaded HMM, and a context-based HMM. The proposed ontology based GA-HMM was subjected to set of semantically annotated visual observations from outdoor group activity experiments. Experimental results from GA-HMM are presented with technical discussions on design of each model and their potential implication to Persistent Surveillance Systems (PSS).

A decision support system for fusion of hard and soft sensor information based on probabilistic latent semantic analysis technique

This paper presents an ongoing effort towards development of an intelligent Decision-Support System (iDSS) for fusion of information from multiple sources consisting of data from hard (physical sensors) and soft (textural sources. Primarily, this paper defines taxonomy of decision support systems for latent semantic data mining from heterogeneous data sources. A Probabilistic Latent Semantic Analysis (PLSA) approach is proposed for latent semantic concepts search from heterogeneous data sources. An architectural model for generating semantic annotation of multi-modality sensors in a modified Transducer Markup Language (TML) is described. A method for TML messages fusion is discussed for alignment and integration of spatiotemporally correlated and associated physical sensory observations. Lastly, the experimental results which exploit fusion of soft/hard sensor sources with support of iDSS are discussed.

Recognition of Human-Vehicle Interactions in Group Activities Via Multi-Attributed Semantic Message Generation

Improved Situational awareness is a vital ongoing research effort for the U.S. Homeland Security for the past recent years. Many outdoor anomalous activities involve vehicles as their primary source of transportation to and from the scene where a plot is executed. Analysis of dynamics of Human-Vehicle Interaction (HVI) helps to identify correlated patterns of activities representing potential threats. The objective of this paper is bi-folded. Primarily, we discuss a method for temporal HVI events detection and verification for generation of HVI hypotheses. To effectively recognize HVI events, a Multi-attribute Vehicle Detection and Identification technique (MVDI) for detection and classification of stationary vehicles is presented. Secondly, we describe a method for identification of pertinent anomalous behaviors through analysis of state transitions between two successively detected events. Finally, we present a technique for generation of HVI semantic messages and present our experimental results to demonstrate the effectiveness of semantic messages for discovery of HVI in group activities.

A multi-modality attributes representation scheme for Group Activity characterization and data fusion

Proper characterization of human Group Activity (GA) interactions can help to detect and prevent certain pertinent threats efficiently. In this paper, we present a model-based scheme for robust group activity characterization. The proposed approach takes advantage of synergy of multi-sensors data to track and identify key individual and group activity events based on fusion of imagery and acoustic sensors data. Each activity event is attributed by a set of tagged features. By matching and correlating attributes of events, the model attempts to associate sensory observations to a priori known ontology. The proposed model benefits from a fusion process that achieves perceptual grouping of activities by spatiotemporal correlation and association of fragmented perceptions extracted from attributed events. In this paper, we present the results of our experimental work and demonstrate the effective and robustness of the decision fusion technique in terms of properly classifying group activities and generating semantic messages describing dynamics of human group activities that, in turn, improves situational awareness.

A robust technique for semantic annotation of group activities based on recognition of extracted features in video streams

Recognition and understanding of group activities can significantly improve situational awareness in Surveillance Systems. To maximize reliability and effectiveness of Persistent Surveillance Systems, annotations of sequential images gathered from video streams (i.e. imagery and acoustic features) must be fused together to generate semantic messages describing group activities (GA). To facilitate efficient fusion of extracted features from any physical sensors a common structure will suffice to ease integration of processed data into new comprehension. In this paper, we describe a framework for extraction and management of pertinent features/attributes vital for annotation of group activities reliably. A robust technique is proposed for fusion of generated events and entities’ attributes from video streams. A modified Transducer Markup Language (TML) is introduced for semantic annotation of events and entities attributes. By aggregation of multi-attribute TML messages, we have demonstrated that salient group activities can be spatiotemporal can be reliable annotated. This paper discusses our experimental results; our analysis of a set of simulated group activities performed under different contexts and demonstrates the efficiency and effectiveness of the proposed modified TML data structure which facilitates seamless fusion of extracted information from video streams.