Ricky J. Sethi

Making data analysis expertise broadly accessible through workflows

The demand for advanced skills in data analysis spans many areas of science, computing, and business analytics. This paper discusses how non-expert users reuse workflows created by experts and representing complex data mining processes for text analytics. They include workflows for document classification, document clustering, and topic detection, all assembled from components available in well-known text analytics software libraries. The workflows expose to non-experts expert-level knowledge on how these individual components need to be combined with data preparation and feature selection steps to make the underlying statistical learning algorithms most effective. The framework allows non-experts to easily experiment with different combinations of data analysis processes, represented as workflows of computations that they can easily reconfigure. We report on our experiences to date on having users with limited data analytic knowledge and even basic programming skills to apply workflows to their data.

Modeling and recognition of complex multi-person interactions in video

In this paper, we focus on the problem of searching for complex activities involving multiple, interacting objects in video. We examine the dynamics of formation and dispersal of groups as well as their interactions with other groups and individuals. In order to establish a general formalism, we examine activities using relative distances in phase space via pairwise analysis of all objects. This allows us to characterize interactions directly by modeling multi-object activities with the Multiple Objects, Pairwise Analysis (MOPA) feature vector, which is based upon physical models of complex interactions in phase space; specifically, we model paired motion as a damped oscillator in phase space. We model and recognize more complex interactions by characterizing pairs which are correlated in phase space as groups. We show how this model can be used for recognition of complex activities on the standard CAVIAR, VIVID, and UCR Videoweb datasets capturing a variety of problem settings.

Activity recognition by integrating the physics of motion with a Neuromorphic model of perception

In this paper, we propose a computational framework for integrating the physics of motion with the neurobiological basis of perception in order to model and recognize human actions and object activities. The essence, or gist, of an action is intrinsically related to the motion of the scenes objects. We define the Hamiltonian Energy Signature (HES) and derive the S-Metric to yield a global representation of the motion of the scenes objects in order to capture the gist of the activity. The HES is a scalar time-series that represents the motion of an object over the course of an activity and the S-Metric is a distance metric which characterizes the global motion of the object, or the entire scene, with a single, scalar value. The neurobiological aspect of activity recognition is handled by casting our analysis within a framework inspired by Neuromorphic Computing (NMC), in which we integrate a Motion Energy model with a Form/Shape model. We employ different Form/Shape representations depending on the video resolution but use our HES and S-Metric for the Motion Energy approach in either case. As the core of our Integration mechanism, we utilize variants of the latest neurobiological models of feature integration and biased competition, which we implement within a Multiple Hypothesis Testing (MHT) framework. Experimental validation of the theory is provided on standard datasets capturing a variety of problem settings: single agent actions (KTH), multi-agent actions, and aerial sequences (VIVID).

Curriculum Guidelines for Undergraduate Programs in Data Science

The Park City Math Institute (PCMI) 2016 Summer Undergraduate Faculty Program met for the purpose of composing guidelines for undergraduate programs in Data Science. The group consisted of 25 undergraduate faculty from a variety of institutions in the U.S., primarily from the disciplines of mathematics, statistics and computer science. These guidelines are meant to provide some structure for institutions planning for or revising a major in Data Science.

Physics-based activity modelling in phase space

In this paper, we employ ideas grounded in physics to examine activities in video. We build the Multi-Resolution Phase Space (MRPS) descriptor, which is a set of feature descriptors that is able to represent complex activities in multiple domains directly from tracks without the need for different heuristics. MRPS is used to do single- and multi-object activity modelling in phase space, which consists of all possible values of the coordinates. The MRPS contains the Sethi Metric (S-Metric), the Hamiltonian Energy Signature (HES), and the Multiple Objects, Pairwise Analysis (MOPA) descriptors: the S-Metric is a distance metric which characterizes the global motion of the object, or the entire scene, with a single, scalar value; the HES is a scalar or multi-dimensional time-series that represents the motion of an object over the course of an activity using either the Hamiltonian or the S-Metric; and the MOPA contains phase space features for paired activities, in which we develop physical models of complex interactions in phase space (specifically, we model paired motion as a damped oscillator in phase space). Finally, we show the S-Metric is a proper distance measure over a metric space and prove its additivity; this allows use of the S-Metric as a distance measure as well as its use in the HES. Experimental validation of the theory is provided on the standard VIVID and UCR Videoweb datasets capturing a variety of problem settings: single agent actions, multi-agent actions, and aerial sequences, including video search.

Wide Area Tracking in Single and Multiple Views

Maintaining the stability of tracks on multiple targets in video over extended time periods and wide areas remains a challenging problem. Basic trackers like the Kalman filter or particle filter deteriorate in performance as the complexity of the scene increases. A few methods have recently shown encouraging results in these application domains. They rely on learning context models, the availability of training data, or modeling the inter-relationships between the tracks. In this chapter, we provide an overview of research in the area of long-term tracking in video. We review some of the methods in the literature and analyze the common sources of errors which cause trackers to fail. We also discuss the limits of performance of the trackers as multiple objects come together to form groups and crowds. On multiple real-life video sequences obtained for a single camera as well as a camera network, we compare the performance of some of the methods.

A Review of Physics-based Methods for Group and Crowd Analysis in Computer Vision

Crowd analysis is a popular topic in computer vision, with important applications to video surveillance, social media analysis, and multimedia retrieval, to name just a few areas. In this paper, we review some of the physics-based methods for group and crowd analysis in computer vision. In particular, we examine approaches for physics-based analysis of groups, crowds, and the simulation of crowds. The purpose of this review is to categorize and delineate the various physics-based and physics-inspired approaches that have been applied to the examination of groups and crowds in video.

Modeling and Recognition of Complex Human Activities

Activity recognition is a field of computer vision which has shown great progress in the past decade. Starting from simple single person activities, research in activity recognition is moving toward more complex scenes involving multiple objects and natural environments. The main challenges in the task include being able to localize and recognize events in a video and deal with the large amount of variation in viewpoint, speed of movement and scale. This chapter gives the reader an overview of the work that has taken place in activity recognition, especially in the domain of complex activities involving multiple interacting objects. We begin with a description of the challenges in activity recognition and give a broad overview of the different approaches. We go into the details of some of the feature descriptors and classification strategies commonly recognized as being the state of the art in this field. We then move to more complex recognition systems, discussing the challenges in complex activity recognition and some of the work which has taken place in this respect. Finally, we provide some examples of recent work in complex activity recognition. The ability to recognize complex behaviors involving multiple interacting objects is a very challenging problem and future work needs to study its various aspects of features, recognition strategies, models, robustness issues, and context, to name a few.

Large-scale multimedia content analysis using scientific workflows

Analyzing web content, particularly multimedia content, for security applications is of great interest. However, it often requires deep expertise in data analytics that is not always accessible to non-experts. Our approach is to use scientific workflows that capture expert-level methods to examine web content. We use workflows to analyze the image and text components of multimedia web posts separately, as well as by a multimodal fusion of both image and text data. In particular, we re-purpose workflow fragments to do the multimedia analysis and create additional components for the fusion of the image and text modalities. In this paper, we present preliminary work which focuses on a Human Trafficking Detection task to help deter human trafficking of minors by thus fusing image and text content from the web. We also examine how workflow fragments save time and effort in multimedia content analysis while bringing together multiple areas of machine learning and computer vision. We further export these workflow fragments using linked data as web objects.

Towards defining groups and crowds in video using the atomic group actions dataset.

Understanding group activities is an essential step towards studying complex crowd behaviours in video. However, such research is often hampered by the lack of a formal definition of a group, as well as a dearth of datasets that concentrate specifically on Atomic Group Actions. 1 In this paper, we provide a quantitative definition of a group based on the Group Transition Ratio (Gtr); the Gtr helps determine when individuals transition to becoming a group (where the individuals can still be tracked) or a crowd (where tracking of individuals is lost). In addition, we introduce the Atomic Group Actions Dataset, a set of 200 videos that concentrate on the atomic group actions of objects in video, namely the group-group actions of formation, dispersal, and movement of a group, as well as the group-person actions of person joining and person leaving a group. We further incorporate a structured, end-to-end analysis methodology, based on workflows, to easily and automatically allow for standardized testing of new group action models against this dataset. We demonstrate the efficacy of the Gtr on the Atomic Group Actions Dataset and make the full dataset (the videos, along with their associated tracks and ground truth, and the exported workflows) publicly available to the research community for free use and extension at at http://research. sethi.org/ricky/datasets/.