Siddhartha R Dalal

Using information mining of the medical literature to improve drug safety.

OBJECTIVE Prescription drugs can be associated with adverse effects (AEs) that are unrecognized despite evidence in the medical literature, as shown by rofecoxibs late recall in 2004. We assessed whether applying information mining to PubMed could reveal major drug-AE associations if articles testing whether drugs cause AEs are over-represented in the literature. DESIGN MEDLINE citations published between 1949 and September 2009 were retrieved if they mentioned one of 38 drugs and one of 55 AEs. A statistical document classifier (using MeSH index terms) was constructed to remove irrelevant articles unlikely to test whether a drug caused an AE. The remaining relevant articles were analyzed using a disproportionality analysis that identified drug-AE associations (signals of disproportionate reporting) using step-up procedures developed to control the familywise type I error rate. MEASUREMENTS Sensitivity and positive predictive value (PPV) for empirical drug-AE associations as judged against drug-AE associations subject to FDA warnings. RESULTS In testing, the statistical document classifier identified relevant articles with 81% sensitivity and 87% PPV. Using data filtered by the statistical document classifier, base-case models showed 64.9% sensitivity and 42.4% PPV for detecting FDA warnings. Base-case models discovered 54% of all detected FDA warnings using literature published before warnings. For example, the rofecoxib-heart disease association was evident using literature published before 2002. Analyses incorporating literature mentioning AEs common to the drug class of interest yielded 71.4% sensitivity and 40.7% PPV. CONCLUSIONS Results from large-scale literature retrieval and analysis (literature mining) compared favorably with and could complement current drug safety methods.

How can we recognize continuous quality improvement

Objective Continuous quality improvement (CQI) methods are foundational approaches to improving healthcare delivery. Publications using the term CQI, however, are methodologically heterogeneous, and labels other than CQI are used to signify relevant approaches. Standards for identifying the use of CQI based on its key methodological features could enable more effective learning across quality improvement (QI) efforts. The objective was to identify essential methodological features for recognizing CQI. Design Previous work with a 12-member international expert panel identified reliably abstracted CQI methodological features. We tested which features met rigorous a priori standards as essential features of CQI using a three-phase online modified-Delphi process. Setting Primarily United States and Canada. Participants 119 QI experts randomly assigned into four on-line panels. Intervention(s) Participants rated CQI features and discussed their answers using online, anonymous and asynchronous discussion boards. We analyzed ratings quantitatively and discussion threads qualitatively. Main outcome measure(s) Panel consensus on definitional CQI features. Results Seventy-nine (66%) panelists completed the process. Thirty-three completers self-identified as QI researchers, 18 as QI practitioners and 28 as both equally. The features ‘systematic data guided activities,’ ‘designing with local conditions in mind’ and ‘iterative development and testing’ met a priori standards as essential CQI features. Qualitative analyses showed cross-cutting themes focused on differences between QI and CQI. Conclusions We found consensus among a broad group of CQI researchers and practitioners on three features as essential for identifying QI work more specifically as ‘CQI.’ All three features are needed as a minimum standard for recognizing CQI methods.

Improving scenario discovery using orthogonal rotations

Scenario discovery offers a new means to characterize and communicate the information in computer simulation models under conditions of deep uncertainty. The approach first defines scenarios as the future states of world where a proposed policy fails to meet its goals and then uses statistical algorithms to find concise descriptions of such regions in large databases of simulation model results. Current scenario discovery applications rely on the Patient Rule Induction Method (PRIM), a user-interactive bump-hunting algorithm that identifies hyper-rectangular regions in the input space of the simulation model. While often successful, scenario discovery applications have been limited because in general a policys vulnerabilities are not well described by the PRIMs hyper-rectangular regions. This study proposes and evaluates improved scenario discovery algorithms that address this challenge with a Principal Component Analysis (PCA)-based preprocessing step that transforms the original model input parameters so that PRIM can then identify high quality hyper-rectangular scenarios in the new rotated coordination system. We explore two versions. PCA-PRIM allows rotations among all uncertain model input parameters and CPCA-PRIM (for constrained PCA-PRIM) only allows rotations among parameters within user-specified domains. The latter may provide more useful information to users, who may find scenario axes described by linear combinations of related domain parameters more interpretable than combinations of dissimilar parameters. We run two sets of tests on the PCA-PRIM and CPCA-PRIM algorithms, the first using simulated test date and the second results from a model used in a previous RAND study of the cost-effectiveness of renewable energy portfolio standards. We find that the new algorithms produce higher quality scenarios than PRIM alone as evaluated by several important measures of merit. In the test data, PCA-PRIM produces improvements averaging 37 percent, and CPCA-PRIM averaging 14 percent, over PRIM alone. In the renewable energy policy case study, PCA-PRIM and CPCA-PRIM exhibit similar improvements of about 16 percent over PRIM, and CPCA-PRIM generates scenarios interpretable by, and that provide useful information to, decision makers.

A Pilot Study Using Machine Learning and Domain Knowledge To Facilitate Comparative Effectiveness Review Updating

Background. Comparative effectiveness and systematic reviews require frequent and time-consuming updating. Results of earlier screening should be useful in reducing the effort needed to screen relevant articles. Methods. We collected 16,707 PubMed citation classification decisions from 2 comparative effectiveness reviews: interventions to prevent fractures in low bone density (LBD) and off-label uses of atypical antipsychotic drugs (AAP). We used previously written search strategies to guide extraction of a limited number of explanatory variables pertaining to the intervention, outcome, and study design. We empirically derived statistical models (based on a sparse generalized linear model with convex penalties [GLMnet] and a gradient boosting machine [GBM]) that predicted article relevance. We evaluated model sensitivity, positive predictive value (PPV), and screening workload reductions using 11,003 PubMed citations retrieved for the LBD and AAP updates. Results. GLMnet-based models performed slightly better than GBM-based models. When attempting to maximize sensitivity for all relevant articles, GLMnet-based models achieved high sensitivities (0.99 and 1.0 for AAP and LBD, respectively) while reducing projected screening by 55.4% and 63.2%. The GLMnet-based model yielded sensitivities of 0.921 and 0.905 and PPVs of 0.185 and 0.102 when predicting articles relevant to the AAP and LBD efficacy/effectiveness analyses, respectively (using a threshold of P ≥ 0.02). GLMnet performed better when identifying adverse effect relevant articles for the AAP review (sensitivity = 0.981) than for the LBD review (0.685). The system currently requires MEDLINE-indexed articles. Conclusions. We evaluated statistical classifiers that used previous classification decisions and explanatory variables derived from MEDLINE indexing terms to predict inclusion decisions. This pilot system reduced workload associated with screening 2 simulated comparative effectiveness review updates by more than 50% with minimal loss of relevant articles.

Detection of radioactive material entering national ports: A Bayesian approach to radiation portal data

Given the potential for illicit nuclear material being used for terrorism, most ports now inspect a large number of goods entering national borders for radioactive cargo. The U.S. Department of Homeland Security is moving toward one hundred percent inspection of all containers entering the U.S. at various ports of entry for nuclear material. We propose a Bayesian classification approach for the real-time data collected by the inline Polyvinyl Toluene radiation portal monitors. We study the computational and asymptotic properties of the proposed method and demonstrate its efficacy in simulations. Given data available to the authorities, it should be feasible to implement this approach in practice.

Collaborative learning framework for online stakeholder engagement

Public and stakeholder engagement can improve the quality of both research and policy decision making. However, such engagement poses significant methodological challenges in terms of collecting and analysing input from large, diverse groups.

Machine learning for the automatic identification of terrorist incidents in worldwide news media

The RAND Database of Worldwide Terrorism Incidents (RDWTI) seeks to index information about all terrorist incidents that occur and are mentioned in worldwide news media, providing a useful resource for policy researchers and decision makers. We examined automated classification methods that could be used to identify news articles about terrorist incidents, thus enabling analysts to read a smaller number of news articles and maintain the database with less effort and cost. The support vector machine (SVM) and Lasso methods were only modestly successful, but a classifier based on the gradient boosting method (GBM) appeared to be very successful, correctly ranking 80% of the relevant articles at the “top of the pile” for examination by a human analyst.

Simultaneous One-Sided Tests with Application to Education Evaluation Systems

There is widespread interest in using various statistical inference tools as a part of the evaluations for individual teachers and schools. Evaluation systems typically involve classifying hundreds or even thousands of teachers or schools according to their estimated performance. Many current evaluations are largely based on individual estimates and hypothesis tests, with little or no consideration of controlling simultaneous error rates and their potential effect on the whole educational evaluation system. In this article, we discuss controlling simultaneous errors in classification of teachers or schools by a decision-theoretic approach. We first develop a β-mixture model to estimate the local false discovery rate (local fdr) by the classic p-values from one-sided tests. We discuss a few decision rules based on standard loss functions. We also construct a controversy loss function to further accommodate the incoherence between effect size and local fdr. We apply the proposed approach to evaluate adequate yearly progress (AYP) in math proficiency of Pennsylvania schools.

Combinatorial enlargement of ground-truth datasets and efficient evaluation of segmentation algorithms

We propose a method to exponentially enlarge a small dataset of domain specific ground truth segmentation labels to evaluate the performance of segmentation algorithms. Furthermore, we adapt ideas from combinatorial software testing to efficiently infer statistics of segmentation performance by evaluating performance on only a certain subset of the combinatorially generated images. Extensions of this work to optimal sequence for performance testing and algorithm selection are also suggested.