Sanjay Purushotham

Recurrent Neural Networks for Multivariate Time Series with Missing Values

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provide useful insights for better understanding and utilization of missing values in time series analysis.

Picture-in-picture copy detection using spatial coding techniques

Picture-in-Picture (PiP) is a special video transformation where one or more videos is scaled and spatially embedded in a host video. PiP is a very useful service to watch two or more videos simultaneously, however it can be exploited to visually hide one video inside another video. Todays copy detection techniques can be easily fooled by PiP, which is reflected in the poor results in the yearly TRECVID competitions. Inspired by the promise of spatial coding in partial image matching, we propose a generalized spatial coding representation in which both the relative position and relative orientation is embedded in the spatial code. In this paper, we will provide novel formulation for spatial verification problem and introduce polynomial and non-polynomial algorithms to efficiently address the spatial verification problem. Our initial experiment results on TRECVID and MSRA datasets shows that our proposed spatial verification algorithms provide around 20% improvement over the classical hierarchical bag-of-words approach.

Measuring and Predicting Tag Importance for Image Retrieval

Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in todays Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval. To achieve this, we first propose a method to measure the relative importance of object and scene tags from image sentence descriptions. Using this as the ground truth, we present a tag importance prediction model to jointly exploit visual, semantic and context cues. The Structural Support Vector Machine (SSVM) formulation is adopted to ensure efficient training of the prediction model. Then, the Canonical Correlation Analysis (CCA) is employed to learn the relation between the image visual feature and tag importance to obtain robust retrieval performance. Experimental results on three real-world datasets show a significant performance improvement of the proposed MIR with Tag Importance Prediction (MIR/TIP) system over other MIR systems.

Factorized sparse learning models with interpretable high order feature interactions

Identifying interpretable discriminative high-order feature interactions given limited training data in high dimensions is challenging in both machine learning and data mining. In this paper, we propose a factorization based sparse learning framework termed FHIM for identifying high-order feature interactions in linear and logistic regression models, and study several optimization methods for solving them. Unlike previous sparse learning methods, our model FHIM recovers both the main effects and the interaction terms accurately without imposing tree-structured hierarchical constraints. Furthermore, we show that FHIM has oracle properties when extended to generalized linear regression models with pairwise interactions. Experiments on simulated data show that FHIM outperforms the state-of-the-art sparse lear-ning techniques. Further experiments on our experimentally generated data from patient blood samples using a novel SOMAmer (Slow Off-rate Modified Aptamer) technology show that, FHIM performs blood-based cancer diagnosis and bio-marker discovery for Renal Cell Carcinoma much better than other competing methods, and it identifies interpretable block-wise high-order gene interactions predictive of cancer stages of samples. A literature survey shows that the interactions identified by FHIM play important roles in cancer development.

Personalized Group Recommender Systems for Location- and Event-Based Social Networks

Location-Based Social Networks (LBSNs) such as Foursquare, Google+ Local, and so on, and Event-Based Social Networks (EBSNs) such as Meetup, Plancast, and so on, have become popular platforms for users to plan, organize, and attend social events with friends and acquaintances. These LBSNs and EBSNs provide rich content such as online and offline user interactions, location/event descriptions that can be leveraged for personalized group recommendations. In this article, we propose novel Collaborative Filtering-based Bayesian models to capture the location or event semantics and group dynamics such as user interactions, user group membership, user influence, and the like for personalized group recommendations. Empirical experiments on two large real-world datasets (Gowalla LBSN dataset and Meetup EBSN dataset) show that our models outperform the state-of-the-art group recommender systems. We discuss the group characteristics of our datasets and show that modeling of group dynamics learns better group preferences than aggregating individual user preferences. Moreover, our model provides human interpretable results that can be used to understand group participation behavior and location/event popularity.

Modeling Group Dynamics for Personalized Group-Event Recommendation

Event-Based Social Networks (EBSNs) such as Meetup, Plancast, etc., have become popular platforms for users to plan and organize social events with friends and acquaintances. These EBSNs provide rich online and offline user interactions, and rich event content information which can be leveraged for personalized group-event recommendations. In this paper, we propose collaborative-filtering based Bayesian models which captures group dynamics such as user interactions, user-group membership etc., for personalized group-event recommendations. We show that modeling group dynamics learns the group preferences better than aggregating individual user preferences, and that our approach out-performs popular state-of-the-art group recommender systems. Moreover, our model provides interpretable results which can be used to study the group participations and event popularity.

A pilot study in using deep learning to predict limited life expectancy in women with recurrent cervical cancer

OBJECTIVE: Cervical cancer remains the most common gynecological malignancy internationally, and nearly one third of patients succumb to this disease within 5 years from diagnosis. While patients with malignancy who have limited life expectancy benefit from less aggressive treatment intervention, there is currently little evidence to guide the prediction of the length of life expectancy. The aims of the study were (1) to examine predictors for survival by utilizing clinicolaboratory variables among women with recurrent cervical cancer and (2) to examine the utility of a new analytic approach using a deep-learning neural networks model.

Video Genre Inference Based on Camera Capturing Models

On-line video collection is getting larger nowadays. It becomes difficult for users to go through the whole collection to find the video of their interest. To allow efficient browsing, search and retrieval, one intuitive solution is to cluster video clips according to their genres automatically. Then, users’ choices can be narrowed down. Besides on-line video repositories, other applications include managing television broadcasting archives, video conferencing records, etc. The goal of video classification is to automatically place each video title in different categories, such as news, sports, etc. The classification process involves extracting the information from the video clips and classifying them into different classes. In this chapter, we first review related work in this field. Then, two novel features based on the camera shooting process is proposed for video genre classification. These new camera based features exploit the fact that a different genre tends to have different camera effects and user perception. Although a lot of work has been proposed with the consideration of cinematic principles, most extracted features are low-level features without much semantic information.We propose a feature that estimates the number of cameras used in a short time interval. Then, we propose another feature by calculating the distribution of the camera distance, which is approximated by the normalized foreground area of each frame. The block-based motion vector field is adopted to reduce the complexity involved in foreground/background modeling. Preliminary experiment results show that the proposed features capture additional genre-related information so that the video genre can be inferred from the proposed features well.

Association of tumor differentiation grade and survival of women with squamous cell carcinoma of the uterine cervix

Objective To examine the association between tumor grade and survival for women with squamous cervical cancer. Methods This retrospective observational study utilized the Surveillance, Epidemiology, and End Result program data between 1983 and 2013 to examine women with squamous cervical cancer with known tumor differentiation grade. Multivariable analyses were performed to assess independent associations between tumor differentiation grade and survival. Results A total of 31,536 women were identified including 15,175 (48.1%) with grade 3 tumors, 14,084 (44.7%) with grade 2 neoplasms and 2,277 (7.2%) with grade 1 tumors. Higher tumor grade was significantly associated with older age, higher stage disease, larger tumor size, and lymph node metastasis (all, p<0.001). In a multivariable analysis, grade 2 tumors (adjusted-hazard ratio [HR]=1.21; p<0.001) and grade 3 tumors (adjusted-HR=1.45; p<0.001) were independently associated with decreased cause-specific survival (CSS) compared to grade 1 tumors. Among the 7,429 women with stage II–III disease who received radiotherapy without surgical treatment, grade 3 tumors were independently associated with decreased CSS compared to grade 2 tumors (adjusted-HR=1.16; p<0.001). Among 4,045 women with node-negative stage I disease and tumor size ≤4 cm who underwent surgical treatment without radiotherapy, grade 2 tumors (adjusted-HR=2.54; p=0.028) and grade 3 tumors (adjusted-HR=4.48; p<0.001) were independently associated with decreased CSS compared to grade 1 tumors. Conclusion Our study suggests that tumor differentiation grade may be a prognostic factor in women with squamous cervical cancer, particularly in early-stage disease. Higher tumor grade was associated with poorer survival.

Time Series Feature Learning with Applications to Health Care

Exponential growth in mobile health devices and electronic health records has resulted in a surge of large-scale time series data, which demands effective and fast machine learning models for analysis and discovery. In this chapter, we discuss a novel framework based on deep learning which automatically performs feature learning from heterogeneous time series data. It is well-suited for healthcare applications, where available data have many sparse outputs (e.g., rare diagnoses) and exploitable structures (e.g., temporal order and relationships between labels). Furthermore, we introduce a simple yet effective knowledge-distillation approach to learn an interpretable model while achieving the prediction performance of deep models. We conduct experiments on several real-world datasets and show the empirical efficacy of our framework and the interpretability of the mimic models.