Ruoqing Zhu

Reinforcement Learning Trees

In this article, we introduce a new type of tree-based method, reinforcement learning trees (RLT), which exhibits significantly improved performance over traditional methods such as random forests (Breiman 2001) under high-dimensional settings. The innovations are three-fold. First, the new method implements reinforcement learning at each selection of a splitting variable during the tree construction processes. By splitting on the variable that brings the greatest future improvement in later splits, rather than choosing the one with largest marginal effect from the immediate split, the constructed tree uses the available samples in a more efficient way. Moreover, such an approach enables linear combination cuts at little extra computational cost. Second, we propose a variable muting procedure that progressively eliminates noise variables during the construction of each individual tree. The muting procedure also takes advantage of reinforcement learning and prevents noise variables from being considered in the search for splitting rules, so that toward terminal nodes, where the sample size is small, the splitting rules are still constructed from only strong variables. Last, we investigate asymptotic properties of the proposed method under basic assumptions and discuss rationale in general settings. Supplementary materials for this article are available online.

Recursively Imputed Survival Trees

We propose recursively imputed survival tree (RIST) regression for right-censored data. This new nonparametric regression procedure uses a novel recursive imputation approach combined with extremely randomized trees that allows significantly better use of censored data than previous tree-based methods, yielding improved model fit and reduced prediction error. The proposed method can also be viewed as a type of Monte Carlo EM algorithm, which generates extra diversity in the tree-based fitting process. Simulation studies and data analyses demonstrate the superior performance of RIST compared with previous methods.

Identifying gene-environment and gene-gene interactions using a progressive penalization approach.

In genomic studies, identifying important gene–environment and gene–gene interactions is a challenging problem. In this study, we adopt the statistical modeling approach, where interactions are represented by product terms in regression models. For the identification of important interactions, we adopt penalization, which has been used in many genomic studies. Straightforward application of penalization does not respect the “main effect, interaction” hierarchical structure. A few recently proposed methods respect this structure by applying constrained penalization. However, they demand very complicated computational algorithms and can only accommodate a small number of genomic measurements. We propose a computationally fast penalization method that can identify important gene–environment and gene–gene interactions and respect a strong hierarchical structure. The method takes a stagewise approach and progressively expands its optimization domain to account for possible hierarchical interactions. It is applicable to multiple data types and models. A coordinate descent method is utilized to produce the entire regularized solution path. Simulation study demonstrates the superior performance of the proposed method. We analyze a lung cancer prognosis study with gene expression measurements and identify important gene–environment interactions.

Combining Biomarkers with EMR Data to Identify Patients in Different Phases of Sepsis

Sepsis is a leading cause of death and is the most expensive condition to treat in U.S. hospitals. Despite targeted efforts to automate earlier detection of sepsis, current techniques rely exclusively on using either standard clinical data or novel biomarker measurements. In this study, we apply machine learning techniques to assess the predictive power of combining multiple biomarker measurements from a single blood sample with electronic medical record data (EMR) for the identification of patients in the early to peak phase of sepsis in a large community hospital setting. Combining biomarkers and EMR data achieved an area under the receiver operating characteristic (ROC) curve (AUC) of 0.81, while EMR data alone achieved an AUC of 0.75. Furthermore, a single measurement of six biomarkers (IL-6, nCD64, IL-1ra, PCT, MCP1, and G-CSF) yielded the same predictive power as collecting an additional 16 hours of EMR data(AUC of 0.80), suggesting that the biomarkers may be useful for identifying these patients earlier. Ultimately, supervised learning using a subset of biomarker and EMR data as features may be capable of identifying patients in the early to peak phase of sepsis in a diverse population and may provide a tool for more timely identification and intervention.

Tree based weighted learning for estimating individualized treatment rules with censored data

Estimating individualized treatment rules is a central task for personalized medicine. [23] and [22] proposed outcome weighted learning to estimate individualized treatment rules directly through maximizing the expected outcome without modeling the response directly. In this paper, we extend the outcome weighted learning to right censored survival data without requiring either inverse probability of censoring weighting or semiparametric modeling of the censoring and failure times as done in [26]. To accomplish this, we take advantage of the tree based approach proposed in [28] to nonparametrically impute the survival time in two different ways. The first approach replaces the reward of each individual by the expected survival time, while in the second approach only the censored observations are imputed by their conditional expected failure times. We establish consistency and convergence rates for both estimators. In simulation studies, our estimators demonstrate improved performance compared to existing methods. We also illustrate the proposed method on a phase III clinical trial of non-small cell lung cancer.

Increasing Access to State Psychiatric Hospital Beds: Exploring Supply-Side Solutions

OBJECTIVE The objective of this study was to identify supply-side interventions to reduce state psychiatric hospital admission delays. METHODS Healthcare Enterprise Accounts Receivable Tracking System (HEARTS) data were collected for all patients admitted between July 1, 2010, and July 31, 2012, to one of North Carolinas three state-operated psychiatric hospitals (N=3,156). Additional information on hospital use was collected at nine meetings with hospital administrators and other local stakeholders. A discrete-event simulation model was built to simulate the flow of adult nonforensic patients through the hospital. Hypothetical scenarios were used to evaluate the effects of varying levels of increased capacity on annual number of admissions and average patient wait time prior to admission. RESULTS In the base case, the model closely approximated actual state hospital utilization, with an average of 1,251±65 annual admissions and a preadmission wait time of 3.3±.1 days across 50 simulations. Results from simulated expansion scenarios highlighted substantial capacity shortfalls in the current system. For example, opening an additional 24-bed unit was projected to decrease average wait time by only 6%. Capacity would need to be increased by 165% (356 beds) to reduce average wait time below 24 hours. CONCLUSIONS Without more robust community-based hospital and residential capacity, major increases in state psychiatric hospital inpatient capacity are necessary to ensure timely admission of people in crisis.

The effects of state psychiatric hospital waitlist policies on length of stay and time to readmission.

This study examined the effects of a waitlist policy for state psychiatric hospitals on length of stay and time to readmission using data from North Carolina for 2004–2010. Cox proportional hazards models tested the hypothesis that patients were discharged “quicker-but-sicker” post-waitlist, as hospitals struggled to manage admission delays and quickly admit waitlisted patients. Results refute this hypothesis, indicating that waitlists were associated with increased length of stay and time to readmission. Further research is needed to evaluate patients’ clinical outcomes directly and to examine the impact of state hospital waitlists in other areas, such as state hospital case mix, local emergency departments, and outpatient mental health agencies.

Greedy outcome weighted tree learning of optimal personalized treatment rules

We propose a subgroup identification approach for inferring optimal and interpretable personalized treatment rules with high-dimensional covariates. Our approach is based on a two-step greedy tree algorithm to pursue signals in a high-dimensional space. In the first step, we transform the treatment selection problem into a weighted classification problem that can utilize tree-based methods. In the second step, we adopt a newly proposed tree-based method, known as reinforcement learning trees, to detect features involved in the optimal treatment rules and to construct binary splitting rules. The method is further extended to right censored survival data by using the accelerated failure time model and introducing double weighting to the classification trees. The performance of the proposed method is demonstrated via simulation studies, as well as analyses of the Cancer Cell Line Encyclopedia (CCLE) data and the Tamoxifen breast cancer data.