James Sharpnack

Recovering graph-structured activations using adaptive compressive measurements

We study the localization of a cluster of activated vertices in a graph, from adaptively designed compressive measurements. We propose a hierarchical partitioning of the graph that groups the activated vertices into few partitions, so that a top-down sensing procedure can identify these partitions, and hence the activations, using few measurements. By exploiting the cluster structure, we are able to provide localization guarantees at weaker signal-to-noise ratios than in the unstructured setting. We complement this performance guarantee with an information-theoretic lower bound, providing a necessary signal-to-noise ratio for any algorithm to successfully localize the cluster. We verify our analysis with some simulations, demonstrating the practicality of our algorithm.

Detecting Anomalous Activity on Networks With the Graph Fourier Scan Statistic

We consider the problem of deciding, based on a single noisy measurement at each vertex of a given graph, whether the underlying unknown signal is constant over the graph or there exists a cluster of vertices with anomalous activation. This problem is relevant to several applications such as surveillance, disease outbreak detection, biomedical imaging, environmental monitoring, etc. Since the activations in these problems often tend to be localized to small groups of vertices in the graphs, we model such activity by a class of signals that are elevated over a (possibly disconnected) cluster with low cut size relative to its size. We analyze the corresponding generalized likelihood ratio (GLR) statistics and relate it to the problem of finding a sparsest cut in the graph. We develop a convex relaxation of the GLR statistic based on spectral graph theory, which we call the graph Fourier scan statistic (GFSS). In our main theoretical result, we show that the performance of the GFSS depends explicitly on the spectral properties of the graph. To assess the optimality of the GFSS, we prove an information theoretic lower bound for the detection of anomalous activity on graphs. Because the GFSS requires the specification of a tuning parameter, we develop an adaptive version of the GFSS. Using these results, we are able to characterize in a very explicit form the performance of the GFSS on a few notable graph topologies. We demonstrate that the GFSS can efficiently detect a simulated Arsenic contamination in groundwater.

Near-optimal and computationally efficient detectors for weak and sparse graph-structured patterns

In this paper, we review our recent work on detecting weak patterns that are sparse and localized on a graph. This problem is relevant to many applications including detecting anomalies in sensor and computer networks, brain activity, co-expressions in gene networks, disease outbreaks etc. We characterize such a class of weak and sparse graph-structured patterns by small subsets of weakly activated nodes with a low cut in an underlying known graph. On one hand, the combinatorial nature of this class renders traditional detectors such as GLRT (aka scan statistic) computationally intractable for general graphs. On the other hand, attempts to develop feasible detectors such as fast subset scanning or averaging/thresholding sacrifice statistical efficiency. We describe and compare three detectors for weak graph-structured patterns that are developed using tools from graph theory, optimization and machine learning. These detectors are computationally efficient, applicable to graphs and patterns with general structures and come with precise theoretical guarantees, often achieving near-optimal statistical performance.

Proteogenomic Analysis of Surgically Resected Lung Adenocarcinoma

Introduction: Despite apparently complete surgical resection, approximately half of resected early‐stage lung cancer patients relapse and die of their disease. Adjuvant chemotherapy reduces this risk by only 5% to 8%. Thus, there is a need for better identifying who benefits from adjuvant therapy, the drivers of relapse, and novel targets in this setting. Methods: RNA sequencing and liquid chromatography/liquid chromatography–mass spectrometry proteomics data were generated from 51 surgically resected non–small cell lung tumors with known recurrence status. Results: We present a rationale and framework for the incorporation of high‐content RNA and protein measurements into integrative biomarkers and show the potential of this approach for predicting risk of recurrence in a group of lung adenocarcinomas. In addition, we characterize the relationship between mRNA and protein measurements in lung adenocarcinoma and show that it is outcome specific. Conclusions: Our results suggest that mRNA and protein data possess independent biological and clinical importance, which can be leveraged to create higher‐powered expression biomarkers.

Large-scale Collaborative Ranking in Near-Linear Time

In this paper, we consider the Collaborative Ranking (CR) problem for recommendation systems. Given a set of pairwise preferences between items for each user, collaborative ranking can be used to rank un-rated items for each user, and this ranking can be naturally used for recommendation. It is observed that collaborative ranking algorithms usually achieve better performance since they directly minimize the ranking loss; however, they are rarely used in practice due to the poor scalability. All the existing CR algorithms have time complexity at least O(|Ω|r) per iteration, where r is the target rank and |Ω| is number of pairs which grows quadratically with number of ratings per user. For example, the Netflix data contains totally 20 billion rating pairs, and at this scale all the current algorithms have to work with significant subsampling, resulting in poor prediction on testing data. In this paper, we propose a new collaborative ranking algorithm called Primal-CR that reduces the time complexity to O(|Ω|+d1 |d2 r), where d1 is number of users and |d2 is the averaged number of items rated by a user. Note that d1 |d2 is strictly smaller and often much smaller than |Ω|. Furthermore, by exploiting the fact that most data is in the form of numerical ratings instead of pairwise comparisons, we propose Primal-CR++ with O(d1|d2 (r+ log |d2)) time complexity. Both algorithms have better theoretical time complexity than existing approaches and also outperform existing approaches in terms of NDCG and pairwise error on real data sets. To the best of our knowledge, this is the first collaborative ranking algorithm capable of working on the full Netflix dataset using all the 20 billion rating pairs, and this leads to a model with much better recommendation compared with previous models trained on subsamples. Finally, compared with classical matrix factorization algorithm which also requires O(d1d2r) time, our algorithm has almost the same efficiency while making much better recommendations since we consider the ranking loss.

A path algorithm for localizing anomalous activity in graphs

The localization of anomalous activity in graphs is a statistical problem that arises in many applications, such as network surveillance, disease outbreak detection, and activity monitoring in social networks. We will address the localization of a cluster of activity in Gaussian noise in directed, weighted graphs. We develop a penalized likelihood estimator (we call the relaxed graph scan) as a relaxation of the NP-hard graph scan statistic. We review how the relaxed graph scan (RGS) can be solved using graph cuts, and outline the max-flow min-cut duality. We use this combinatorial duality to derive a path algorithm for the RGS by solving successive max flows. We demonstrate the effectiveness of the RGS on two simulations, over an undirected and directed graph.