Vandana Pursnani Janeja

Neighborhood based detection of anomalies in high dimensional spatio-temporal sensor datasets

The behavior of spatial objects is under the influence of nearby spatial processes. Therefore in order to perform any type of spatial analysis we need to take into account not only the spatial relationships among objects but also the underlying spatial processes and other spatial features in the vicinity that influence the behavior of a given spatial object. In this paper, we address the outlier detection by refining the concept of a neighborhood of an object, which essentially characterizes similarly behaving objects into one neighborhood. This similarity is quantified in terms of the spatial relationships among the objects and other semantic relationships based on the spatial processes and spatial features in their vicinity. These spatial features could be natural such as a stream, and vegetation, or man-made such as a bridge, railroad, and chemical factory. The paper also addresses the identification of spatio-temporal outliers in high dimensions, in their neighborhood.

Random Walks to Identify Anomalous Free-Form Spatial Scan Windows

Often, it is required to identify anomalous windows over a spatial region that reflect unusual rate of occurrence of a specific event of interest. A spatial scan statistic-based approach essentially considers a scan window and computes the statistic of a parameter(s) of interest, and identifies anomalous windows by moving the scan window in the region. While this approach has been successfully employed in identifying anomalous windows, earlier proposals adopting spatial scan statistic suffer from two limitations: (1) In general, the scan window should be of a regular shape (e.g., circle, rectangle, cylinder). Thus, most approaches are capable of identifying anomalous windows of fixed shapes only. However, the region of anomaly, in general, is not necessarily of a regular shape. Recent proposals to identify windows of irregular shapes identify windows much larger than the true anomalies or penalize large-sized windows. (2) These techniques take into account autocorrelation among spatial data but not spatial heterogeneity. As a result, they often result in inaccurate anomalous windows. To address these limitations, in this paper, we propose a random-walk-based Free-Form Spatial Scan Statistic (FS3). We construct a weighted Delaunay nearest neighbor (WDNN) graph to capture both spatial autocorrelation and heterogeneity. We then use random walks to identify natural free-form scan windows that are not restricted to a predefined shape. We use spatial scan statistics to identify anomalous windows and prove that they are not random but indeed are formed as a result of an anomaly. Application of FS3 on real data sets has shown that it can identify more refined anomalous windows with better likelihood ratio of it being an anomaly than those identified by earlier spatial scan statistic approaches.

Predicting Hospital Length of Stay (PHLOS): A Multi-tiered Data Mining Approach

A model to predict the Length of Stay (LOS) for hospitalized patients can be an effective tool for healthcare providers. Such a model will enable early interventions to prevent complications and prolonged LOS and also enable more efficient utilization of manpower and facilities in hospitals. In this paper, we propose an approach for Predicting Hospital Length of Stay (PHLOS) using a multi-tiered data mining approach. In this paper we propose a methodology that employs clustering to create the training sets to train different classification algorithms. We compared the performance of different classifiers along several different performance measures and consistently found that using clustering as a precursor to form the training set gives better prediction results as compared to non-clustering based training sets. We have also found the accuracies to be consistently higher than some reported in the current literature for predicting individual patient LOS. The classification techniques used in this study are interpretable, enabling us to examine the details of the classification rules learned from the data. As a result, this study provides insight into the underlying factors that influence hospital length of stay. We also examine our results with domain expert insights.

LS 3 : a L inear S emantic S can S tatistic technique for detecting anomalous windows

Often, it is required to identify anomalous windows along a linear path that reflect unusual rate of occurrence of a specific event of interest. Such examples include: determination of places with high number of occurrences of road accidents along a highway, leaks in natural gas transmission pipelines, pedestrian fatalities on roads, etc. In this paper, we propose a <u>L</u>inear <u>S</u>emantic <u>S</u>can <u>S</u>tatistic (LS3) approach to identify such anomalous windows along a linear path. We assume that a linear path is comprised of one-dimensional spatial locations called markers, where each marker is associated with a set of structural and behavioral attributes. We divide the linear path into linear semantic segments such that each semantic segment contains markers associated with similar structural attributes. Our goal is to identify the windows within a semantic segment whose behavioral attributes are anomalous in some sense. We accomplish this by applying the scan statistic to the behavioral attributes of the markers. We have implemented our approach by considering the real datasets of certain highways in New Jersey, USA. Our results validate that LS3 is effective in identifying high traffic accident windows.

Spatial neighborhood based anomaly detection in sensor datasets

Success of anomaly detection, similar to other spatial data mining techniques, relies on neighborhood definition. In this paper, we argue that the anomalous behavior of spatial objects in a neighborhood can be truly captured when both (a) spatial autocorrelation (similar behavior of nearby objects due to proximity) and (b) spatial heterogeneity (distinct behavior of nearby objects due to difference in the underlying processes in the region) are taken into consideration for the neighborhood definition. Our approach begins by generating micro neighborhoods around spatial objects encompassing all the information about a spatial object. We selectively merge these based on spatial relationships accounting for autocorrelation and inferential relationships accounting for heterogeneity, forming macro neighborhoods. In such neighborhoods, we then identify (i) spatio-temporal outliers, where individual sensor readings are anomalous, (ii) spatial outliers, where the entire sensor is an anomaly, and (iii) spatio-temporally coalesced outliers, where a group of spatio-temporal outliers in the macro neighborhood are separated by a small time lag indicating the traversal of the anomaly. We demonstrate the effectiveness of our approach in neighborhood formation and anomaly detection with experimental results in (i) water monitoring and (ii) highway traffic monitoring sensor datasets. We also compare the results of our approach with an existing approach for spatial anomaly detection.

Using Semantic Networks and Context in Search for Relevant Software Engineering Artifacts

The discovery of relevant software artifacts can increase software reuse and reduce the cost of software development and maintenance. Furthermore, change requests, which are a leading cause of project failures, can be better classified and handled when all relevant artifacts are available to the decision makers. However, traditional full-text and similarity search techniques often fail to provide the full set of relevant documents because they do not take into consideration existing relationships between software artifacts. We propose a metadata approach with semantic networks which convey such relationships. Our approach reveals additional relevant artifacts that the user might have not been aware of. We also apply contextual information to filter out results unrelated to the user contexts, thus, improving the precision of the search results. Experimental results show that the combination of semantic networks and context significantly improve the precision and recall of the search results.

Anomalous window discovery through scan statistics for linear intersecting paths (SSLIP)

Anomalous windows are the contiguous groupings of data points. In this paper, we propose an approach for discovering anomalous windows using Scan Statistics for Linear Intersecting Paths (SSLIP). A linear path refers to a path represented by a line with a single dimensional spatial coordinate marking an observation point. Our approach for discovering anomalous windows along linear paths comprises of the following distinct steps: (a) Cross Path Discovery: where we identify a subset of intersecting paths to be considered, (b) Anomalous Window Discovery: where we outline three order invariant algorithms, namely SSLIP, Brute Force-SSLIP and Central Brute Force-SSLIP, for the traversal of the cross paths to identify varying size directional windows along the paths. For identifying an anomalous window we compute an unusualness metric, in the form of a likelihood ratio to indicate the degree of unusualness of this window with respect to the rest of the data. We identify the window with the highest likelihood ratio as our anomalous window, and (c) Monte Carlo Simulations: to ascertain whether this window is truly anomalous and not just a random occurrence we perform hypothesis testing by computing a p-value using Monte Carlo Simulations. We present extensive experimental results in real world accident datasets for various highways with known issues(code and data available from [27], [21]). Our results show that our approach indeed is effective in identifying anomalous traffic accident windows along multiple intersecting highways.

Approach for Discovering and Handling Crisis in a Service-Oriented Environment

In an emergency situation failure to respond in a timely manner poses a significant threat. Data needed for timely response comes from various sources and sensors. These individual data streams when viewed in isolation may appear irrelevant, however, when analyzed collectively may identify potential threats. An effective and timely response also requires collaboration and information sharing among various government agencies at all levels. This collaboration information sharing among agencies can be achieved using service-oriented architecture, where agencies provide access to their information resources and applications using Web services. Each of these agencies has its own rules/policies for providing their services. It is therefore, important to verify the correctness of the emergency response processes with respect to the rules/policies of the collaborating agencies involved in the execution of such processes. In this paper we present an approach which addresses the above challenges. Specifically, the proposed approach: a) employs multi stream data mining for identification of potential threats and disambiguation of alarms; b) provides a methodology for the discovery and selection of relevant Web services; c) employs a timed automata based verification methodology for determining the correctness of emergency response processes with respect to the rules of the collaborating agencies. We provide an overview of the initial implementation of the proposed approach.

Spatiotemporal neighborhood discovery for sensor data

The focus of this paper is the discovery of spatiotemporal neighborhoods in sensor datasets where a time series of data is collected at many spatial locations. The purpose of the spatiotemporal neighborhoods is to provide regions in the data where knowledge discovery tasks such as outlier detection, can be focused. As building blocks for the spatiotemporal neighborhoods, we have developed a method to generate spatial neighborhoods and a method to discretize temporal intervals. These methods were tested on real life datasets including (a) sea surface temperature data from the Tropical Atmospheric Ocean Project (TAO) array in the Equatorial Pacific Ocean and (b)highway sensor network data archive. We have found encouraging results which are validated by real life phenomenon.

Spatial outlier detection in heterogeneous neighborhoods

Spatial outlier detection approaches identify outliers by first defining a spatial neighborhood. However, existing approaches suffer from two issues: (1) they primarily consider autocorrelation alone in forming the neighborhood, but ignore heterogeneity among spatial objects. (2) they do not consider interrelationships among the attributes for identifying how distinct the object is with respect to its neighbors, but consider them independently (either single or multiple). As a result, one may not identify truly unusual spatial objects and may also end up with frivolous outliers. In this paper, we revisit the computation of the spatial neighborhood and propose an approach to address the above two issues. We begin our approach with identifying a spatially related neighborhood, capturing autocorrelation. We then consider interrelationships between attributes and multiple, multilevel distributions within these attributes, thus considering autocorrelation and heterogeneity in various forms. Subsequently, we identify outliers in these neighborhoods. Our experimental results in various datasets (North Carolina SIDS data, New Mexico Leukemia data, etc.) indicate that our approach indeed correctly identifies outliers in heterogeneous neighborhoods.