Alexander G. Gray

High redshift detection of the integrated Sachs-Wolfe effect

We present evidence of a large angle correlation between the cosmic microwave background measured by WMAP and a catalog of photometrically detected quasars from the SDSS. The observed cross correlation is 0.30{+-}0.14 {mu}K at zero lag, with a shape consistent with that expected for correlations arising from the integrated Sachs-Wolfe effect. The photometric redshifts of the quasars are centered at z{approx}1.5, making this the deepest survey in which such a correlation has been observed. Assuming this correlation is due to the ISW effect, this constitutes the earliest evidence yet for dark energy and it can be used to constrain exotic dark energy models.

The three-point correlation function of luminous red galaxies in the Sloan Digital Sky Survey

We present measurements of the redshift-space three-point correlation function of 50 967 luminous red galaxies (LRGs) from Data Release 3 (DR3) of the Sloan Digital Sky Survey (SDSS). We have studied the shape dependence of the reduced three-point correlation function (Qz(s, q, θ )) on three different scales, s = 4, 7 and 10 h −1 Mpc, and over the range of 1 < q < 3 and 0 ◦ <θ <180 ◦ . On small scales (s = 4 h −1 Mpc), Qz is nearly constant, with little change as a function of q and θ. However, there is evidence for a shallow U-shaped behaviour (with θ ) which is expected from theoretical modelling of Qz(s, q, θ ). On larger scales (s = 7 and 10 h −1 Mpc), the U-shaped anisotropy in Qz (with θ) is more clearly detected. We compare this shape dependence in Qz(s, q, θ) with that seen in mock galaxy catalogues which were generated by populating the dark matter haloes in large N-body simulations with mock galaxies using various halo occupation distributions (HOD). We find that the combination of the observed number density of LRGs, the (redshift-space) two-point correlation function and Qz(s, q, θ ) provides a strong constraint on the allowed HOD parameters (Mmin, M1, α) and breaks key degeneracies between these parameters. For example, our observed Qz(s, q, θ) disfavours mock catalogues that overpopulate massive dark matter haloes with many LRG satellites. We also estimate the linear bias of LRGs to be b = 1.87 ± 0.07 in excellent agreement with other measurements.

Maximum inner-product search using cone trees

The problem of efficiently finding the best match for a query in a given set with respect to the Euclidean distance or the cosine similarity has been extensively studied. However, the closely related problem of efficiently finding the best match with respect to the inner-product has never been explored in the general setting to the best of our knowledge. In this paper we consider this problem and contrast it with the previous problems considered. First, we propose a general branch-and-bound algorithm based on a (single) tree data structure. Subsequently, we present a dual-tree algorithm for the case where there are multiple queries. Our proposed branch-and-bound algorithms are based on novel inner-product bounds. Finally we present a new data structure, the cone tree, for increasing the efficiency of the dual-tree algorithm. We evaluate our proposed algorithms on a variety of data sets from various applications, and exhibit up to five orders of magnitude improvement in query time over the naive search technique in some cases.

EIGHT-DIMENSIONAL MID-INFRARED/OPTICAL BAYESIAN QUASAR SELECTION

We explore the multidimensional, multiwavelength selection of quasars from mid-infrared (MIR) plus optical data, specifically from Spitzer-Infrared Array Camera (IRAC) and the Sloan Digital Sky Survey (SDSS). Traditionally, quasar selection relies on cuts in two-dimensional color space despite the fact that most modern surveys (optical and IR) are done in more than three bandpasses. In this paper, we apply modern statistical techniques to combined Spitzer MIR and SDSS optical data, allowing up to eight-dimensional (8-D) color selection of quasars. Using a Bayesian selection method, we catalog 5546 quasar candidates to an 8.0 μm depth of 56 μJy over an area of ~24 deg2. Roughly 70% of these candidates are not identified by applying the same Bayesian algorithm to 4-color SDSS optical data alone. The 8-D optical+MIR selection on this data set recovers 97.7% of known type 1 quasars in this area and greatly improves the effectiveness of identifying 3.5 < z < 5 quasars which are challenging to identify (without considerable contamination) using MIR data alone. We demonstrate that, even using only the two shortest wavelength IRAC bandpasses (3.6 and 4.5 μm), it is possible to use our Bayesian techniques to select quasars with 97% completeness and as little as 10% contamination (as compared to ~60% contamination using color cuts alone). We compute photometric redshifts for our sample; comparison with known objects suggests a photometric redshift accuracy of 93.6% (Δz ± 0.3), remaining roughly constant when the two reddest MIR bands are excluded. Despite the fact that our methods are designed to find type 1 (unobscured) quasars, as many as 1200 of the objects are type 2 (obscured) quasar candidates. Coupling deep optical imaging data, with deep MIR data, could enable selection of quasars in significant numbers past the peak of the quasar luminosity function (QLF) to at least z ~ 4. Such a sample would constrain the shape of the QLF both above and below the break luminosity (L* Q ) and enable quasar clustering studies over the largest range of redshift and luminosity to date, yielding significant gains in our understanding of the physics of quasars and their contribution to galaxy evolution.

Introduction to astroML: Machine learning for astrophysics

Astronomy and astrophysics are witnessing dramatic increases in data volume as detectors, telescopes and computers become ever more powerful. During the last decade, sky surveys across the electromagnetic spectrum have collected hundreds of terabytes of astronomical data for hundreds of millions of sources. Over the next decade, the data volume will enter the petabyte domain, and provide accurate measurements for billions of sources. Astronomy and physics students are not traditionally trained to handle such voluminous and complex data sets. In this paper we describe astroML; an initiative, based on python and scikit-learn, to develop a compendium of machine learning tools designed to address the statistical needs of the next generation of students and astronomical surveys. We introduce astroML and present a number of example applications that are enabled by this package.

Rapid Mass Spectrometric Metabolic Profiling of Blood Sera Detects Ovarian Cancer with High Accuracy

Background: Ovarian cancer diagnosis is problematic because the disease is typically asymptomatic, especially at the early stages of progression and/or recurrence. We report here the integration of a new mass spectrometric technology with a novel support vector machine computational method for use in cancer diagnostics, and describe the application of the method to ovarian cancer. Methods: We coupled a high-throughput ambient ionization technique for mass spectrometry (direct analysis in real-time mass spectrometry) to profile relative metabolite levels in sera from 44 women diagnosed with serous papillary ovarian cancer (stages I-IV) and 50 healthy women or women with benign conditions. The profiles were input to a customized functional support vector machine–based machine-learning algorithm for diagnostic classification. Performance was evaluated through a 64-30 split validation test and with a stringent series of leave-one-out cross-validations. Results: The assay distinguished between the cancer and control groups with an unprecedented 99% to 100% accuracy (100% sensitivity and 100% specificity by the 64-30 split validation test; 100% sensitivity and 98% specificity by leave-one-out cross-validations). Conclusion: The method has significant clinical potential as a cancer diagnostic tool. Because of the extremely low prevalence of ovarian cancer in the general population (∼0.04%), extensive prospective testing will be required to evaluate the tests potential utility in general screening applications. However, more immediate applications might be as a diagnostic tool in higher-risk groups or to monitor cancer recurrence after therapeutic treatment. Impact: The ability to accurately and inexpensively diagnose ovarian cancer will have a significant positive effect on ovarian cancer treatment and outcome. Cancer Epidemiol Biomarkers Prev; 19(9); 2262–71. ©2010 AACR.

On-line anomaly detection of deployed software: a statistical machine learning approach

This paper presents a new machine-learning technique that performs anomaly detection as software is executing in the field. The technique uses a fully observable Markov model where each state in the model emits a number of distinct observations according to a probability distribution, and estimates the model parameters using the Baum-Welch algorithm. The trained model is then deployed with the software to perform anomaly detection. By performing the anomaly detection as the software is executing, faults associated with anomalies can be located and fixed before they cause critical failures in the system, and developers time to debug deployed software can be reduced. This paper also presents a prototype implementation of our technique, along with a case study that shows, for the subjects we studied, the effectiveness of the technique.

Fast euclidean minimum spanning tree: algorithm, analysis, and applications

The Euclidean Minimum Spanning Tree problem has applications in a wide range of fields, and many efficient algorithms have been developed to solve it. We present a new, fast, general EMST algorithm, motivated by the clustering and analysis of astronomical data. Large-scale astronomical surveys, including the Sloan Digital Sky Survey, and large simulations of the early universe, such as the Millennium Simulation, can contain millions of points and fill terabytes of storage. Traditional EMST methods scale quadratically, and more advanced methods lack rigorous runtime guarantees. We present a new dual-tree algorithm for efficiently computing the EMST, use adaptive algorithm analysis to prove the tightest (and possibly optimal) runtime bound for the EMST problem to-date, and demonstrate the scalability of our method on astronomical data sets.

Automatic joint classification and segmentation of whole cell 3D images

We present a machine learning tool for automatic texton-based joint classification and segmentation of mitochondria in MNT-1 cells imaged using ion-abrasion scanning electron microscopy (IA-SEM). For diagnosing signatures that may be unique to cellular states such as cancer, automatic tools with minimal user intervention need to be developed for analysis and mining of high-throughput data from these large volume data sets (typically ~2GB/cell). Challenges for such a tool in 3D electron microscopy arise due to low contrast and signal-to-noise ratios (SNR) inherent to biological imaging. Our approach is based on block-wise classification of images into a trained list of regions. Given manually labeled images, our goal is to learn models that can localize novel instances of the regions in test datasets. Since datasets obtained using electron microscopes are intrinsically noisy, we improve the SNR of the data for automatic segmentation by implementing a 2D texture-preserving filter on each slice of the 3D dataset. We investigate texton-based region features in this work. Classification is performed by k-nearest neighbor (k-NN) classifier, support vector machines (SVMs), adaptive boosting (AdaBoost) and histogram matching using a NN classifier. In addition, we study the computational complexity vs. segmentation accuracy tradeoff of these classifiers. Segmentation results demonstrate that our approach using minimal training data performs close to semi-automatic methods using the variational level-set method and manual segmentation carried out by an experienced user. Using our method, which we show to have minimal user intervention and high classification accuracy, we investigate quantitative parameters such as volume of the cytoplasm occupied by mitochondria, differences between the surface area of inner and outer membranes and mean mitochondrial width which are quantities potentially relevant to distinguishing cancer cells from normal cells. To test the accuracy of our approach, these quantities are compared against manually computed counterparts. We also demonstrate extension of these methods to segment 3D images obtained using electron tomography.

Density estimation trees

In this paper we develop density estimation trees (DETs), the natural analog of classification trees and regression trees, for the task of density estimation. We consider the estimation of a joint probability density function of a d-dimensional random vector X and define a piecewise constant estimator structured as a decision tree. The integrated squared error is minimized to learn the tree. We show that the method is nonparametric: under standard conditions of nonparametric density estimation, DETs are shown to be asymptotically consistent. In addition, being decision trees, DETs perform automatic feature selection. They empirically exhibit the interpretability, adaptability and feature selection properties of supervised decision trees while incurring slight loss in accuracy over other nonparametric density estimators. Hence they might be able to avoid the curse of dimensionality if the true density is sparse in dimensions. We believe that density estimation trees provide a new tool for exploratory data analysis with unique capabilities.