Berk Ustun

The World Health Organization Adult Attention-Deficit/Hyperactivity Disorder Self-Report Screening Scale for DSM-5.

Importance Recognition that adult attention-deficit/hyperactivity disorder (ADHD) is common, seriously impairing, and usually undiagnosed has led to the development of adult ADHD screening scales for use in community, workplace, and primary care settings. However, these scales are all calibrated to DSM-IV criteria, which are narrower than the recently developed DSM-5 criteria. Objectives To update for DSM-5 criteria and improve the operating characteristics of the widely used World Health Organization Adult ADHD Self-Report Scale (ASRS) for screening. Design, Setting, and Participants Probability subsamples of participants in 2 general population surveys (2001-2003 household survey [n = 119] and 2004-2005 managed care subscriber survey [n = 218]) who completed the full 29-question self-report ASRS, with both subsamples over-sampling ASRS-screened positives, were blindly administered a semistructured research diagnostic interview for DSM-5 adult ADHD. In 2016, the Risk-Calibrated Supersparse Linear Integer Model, a novel machine-learning algorithm designed to create screening scales with optimal integer weights and limited numbers of screening questions, was applied to the pooled data to create a DSM-5 version of the ASRS screening scale. The accuracy of the new scale was then confirmed in an independent 2011-2012 clinical sample of patients seeking evaluation at the New York University Langone Medical Center Adult ADHD Program (NYU Langone) and 2015-2016 primary care controls (n = 300). Data analysis was conducted from April 4, 2016, to September 22, 2016. Main Outcomes and Measures The sensitivity, specificity, area under the curve (AUC), and positive predictive value (PPV) of the revised ASRS. Results Of the total 637 participants, 44 (37.0%) household survey respondents, 51 (23.4%) managed care respondents, and 173 (57.7%) NYU Langone respondents met DSM-5 criteria for adult ADHD in the semistructured diagnostic interview. Of the respondents who met DSM-5 criteria for adult ADHD, 123 were male (45.9%); mean (SD) age was 33.1 (11.4) years. A 6-question screening scale was found to be optimal in distinguishing cases from noncases in the first 2 samples. Operating characteristics were excellent at the diagnostic threshold in the weighted (to the 8.2% DSM-5/Adult ADHD Clinical Diagnostic Scale population prevalence) data (sensitivity, 91.4%; specificity, 96.0%; AUC, 0.94; PPV, 67.3%). Operating characteristics were similar despite a much higher prevalence (57.7%) when the scale was applied to the NYU Langone clinical sample (sensitivity, 91.9%; specificity, 74.0%; AUC, 0.83; PPV, 82.8%). Conclusions and Relevance The new ADHD screening scale is short, easily scored, detects the vast majority of general population cases at a threshold that also has high specificity and PPV, and could be used as a screening tool in specialty treatment settings.

Interpretable classification models for recidivism prediction

We investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent, and interpretable to use for decision-making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on probation, to allocating preventative social services. Each use case might have an objective other than classification accuracy, such as a desired true positive rate (TPR) or false positive rate (FPR). Each (TPR, FPR) pair is a point on the receiver operator characteristic (ROC) curve. We use popular machine learning methods to create models along the full ROC curve on a wide range of recidivism prediction problems. We show that many methods (SVM, Ridge Regression) produce equally accurate models along the full ROC curve. However, methods that designed for interpretability (CART, C5.0) cannot be tuned to produce models that are accurate and/or interpretable. To handle this shortcoming, we use a new method known as SLIM (Supersparse Linear Integer Models) to produce accurate, transparent, and interpretable models along the full ROC curve. These models can be used for decision-making for many different use cases, since they are just as accurate as the most powerful black-box machine learning models, but completely transparent, and highly interpretable.

Clinical Prediction Models for Sleep Apnea: The Importance of Medical History over Symptoms.

STUDY OBJECTIVE Obstructive sleep apnea (OSA) is a treatable contributor to morbidity and mortality. However, most patients with OSA remain undiagnosed. We used a new machine learning method known as SLIM (Supersparse Linear Integer Models) to test the hypothesis that a diagnostic screening tool based on routinely available medical information would be superior to one based solely on patient-reported sleep-related symptoms. METHODS We analyzed polysomnography (PSG) and self-reported clinical information from 1,922 patients tested in our clinical sleep laboratory. We used SLIM and 7 state-of-the-art classification methods to produce predictive models for OSA screening using features from: (i) self-reported symptoms; (ii) self-reported medical information that could, in principle, be extracted from electronic health records (demographics, comorbidities), or (iii) both. RESULTS For diagnosing OSA, we found that model performance using only medical history features was superior to model performance using symptoms alone, and similar to model performance using all features. Performance was similar to that reported for other widely used tools: sensitivity 64.2% and specificity 77%. SLIM accuracy was similar to state-of-the-art classification models applied to this dataset, but with the benefit of full transparency, allowing for hands-on prediction using yes/no answers to a small number of clinical queries. CONCLUSION To predict OSA, variables such as age, sex, BMI, and medical history are superior to the symptom variables we examined for predicting OSA. SLIM produces an actionable clinical tool that can be applied to data that is routinely available in modern electronic health records, which may facilitate automated, rather than manual, OSA screening. COMMENTARY A commentary on this article appears in this issue on page 159.

Optimized Risk Scores

Risk scores are simple classification models that let users quickly assess risk by adding, subtracting, and multiplying a few small numbers. Such models are widely used in healthcare and criminal justice, but are often built ad hoc. In this paper, we present a principled approach to learn risk scores that are fully optimized for feature selection, integer coefficients, and operational constraints. We formulate the risk score problem as a mixed integer nonlinear program, and present a new cutting plane algorithm to efficiently recover its optimal solution. Our approach can fit optimized risk scores in a way that scales linearly with the sample size of a dataset, provides a proof of optimality, and obeys complex constraints without parameter tuning. We illustrate these benefits through an extensive set of numerical experiments, and an application where we build a customized risk score for ICU seizure prediction.

Importance Sampling in Stochastic Programming: A Markov Chain Monte Carlo Approach

Stochastic programming models are large-scale optimization problems that are used to facilitate decision making under uncertainty. Optimization algorithms for such problems need to evaluate the expected future costs of current decisions, often referred to as the recourse function. In practice, this calculation is computationally difficult as it requires the evaluation of a multidimensional integral whose integrand is an optimization problem. In turn, the recourse function has to be estimated using techniques such as scenario trees or Monte Carlo methods, both of which require numerous functional evaluations to produce accurate results for large-scale problems with multiple periods and high-dimensional uncertainty. In this work, we introduce an importance sampling framework for stochastic programming that can produce accurate estimates of the recourse function using a small number of samples. Our framework combines Markov chain Monte Carlo methods with kernel density estimation algorithms to build a nonparametric importance sampling distribution, which can then be used to produce a lower-variance estimate of the recourse function. We demonstrate the increased accuracy and efficiency of our approach using variants of well-known multistage stochastic programming problems. Our numerical results show that our framework produces more accurate estimates of the optimal value of stochastic programming models, especially for problems with moderate variance, multimodal, or rare-event distributions.

Association of an Electroencephalography-Based Risk Score With Seizure Probability in Hospitalized Patients

Importance Continuous electroencephalography (EEG) use in critically ill patients is expanding. There is no validated method to combine risk factors and guide clinicians in assessing seizure risk. Objective To use seizure risk factors from EEG and clinical history to create a simple scoring system associated with the probability of seizures in patients with acute illness. Design, Setting, and Participants We used a prospective multicenter (Emory University Hospital, Brigham and Women’s Hospital, and Yale University Hospital) database containing clinical and electrographic variables on 5427 continuous EEG sessions from eligible patients if they had continuous EEG for clinical indications, excluding epilepsy monitoring unit admissions. We created a scoring system model to estimate seizure risk in acutely ill patients undergoing continuous EEG. The model was built using a new machine learning method (RiskSLIM) that is designed to produce accurate, risk-calibrated scoring systems with a limited number of variables and small integer weights. We validated the accuracy and risk calibration of our model using cross-validation and compared its performance with models built with state-of-the-art logistic regression methods. The database was developed by the Critical Care EEG Research Consortium and used data collected over 3 years. The EEG variables were interpreted using standardized terminology by certified reviewers. Exposures All patients had more than 6 hours of uninterrupted EEG recordings. Main Outcomes and Measures The main outcome was the average risk calibration error. Results There were 5427 continuous EEGs performed on 4772 participants (2868 men, 49.9%; median age, 61 years) performed at 3 institutions, without further demographic stratification. Our final model, 2HELPS2B, had an area under the curve of 0.819 and average calibration error of 2.7% (95% CI, 2.0%-3.6%). It included 6 variables with the following point assignments: (1) brief (ictal) rhythmic discharges (B[I]RDs) (2 points); (2) presence of lateralized periodic discharges, lateralized rhythmic delta activity, or bilateral independent periodic discharges (1 point); (3) prior seizure (1 point); (4) sporadic epileptiform discharges (1 point); (5) frequency greater than 2.0 Hz for any periodic or rhythmic pattern (1 point); and (6) presence of “plus” features (superimposed, rhythmic, sharp, or fast activity) (1 point). The probable seizure risk of each score was 5% for a score of 0, 12% for a score of 1, 27% for a score of 2, 50% for a score of 3, 73% for a score of 4, 88% for a score of 5, and greater than 95% for a score of 6 or 7. Conclusions and Relevance The 2HELPS2B model is a quick accurate tool to aid clinical judgment of the risk of seizures in critically ill patients.