Lester Reich

Data Mining in Pharmacovigilance

Data mining is receiving considerable attention as a tool for pharmacovigilance and is generating many perspectives on its uses. This paper presents four concepts that have appeared in various professional venues and represent potential sources of misunderstanding and/or entail extended discussions: (i) data mining algorithms are unvalidated; (ii) data mining algorithms allow data miners to objectively screen spontaneous report data; (iii) mathematically more complex Bayesian algorithms are superior to frequentist algorithms; and (iv) data mining algorithms are not just for hypothesis generation. Key points for a balanced perspective are that: (i) validation exercises have been done but lack a gold standard for comparison and are complicated by numerous nuances and pitfalls in the deployment of data mining algorithms. Their performance is likely to be highly situation dependent; (ii) the subjective nature of data mining is often underappreciated; (iii) simpler data mining models can be supplemented with ‘clinical shrinkage’, preserving sensitivity; and (iv) applications of data mining beyond hypothesis generation are risky, given the limitations of the data. These extended applications tend to ‘creep’, not pounce, into the public domain, leading to potential overconfidence in their results. Most importantly, in the enthusiasm generated by the promise of data mining tools, users must keep in mind the limitations of the data and the importance of clinical judgment and context, regardless of statistical arithmetic. In conclusion, we agree that contemporary data mining algorithms are promising additions to the pharmacovigilance toolkit, but the level of verification required should be commensurate with the nature and extent of the claimed applications.

Safety Related Drug-Labelling Changes Findings from Two Data Mining Algorithms

AbstractIntroduction: With increasing volumes of postmarketing safety surveillance data, data mining algorithms (DMAs) have been developed to search large spontaneous reporting system (SRS) databases for disproportional statistical dependencies between drugs and events. A crucial question is the proper deployment of such techniques within the universe of methods historically used for signal detection. One question of interest is comparative performance of algorithms based on simple forms of disproportionality analysis versus those incorporating Bayesian modelling. A potential benefit of Bayesian methods is a reduced volume of signals, including false-positive signals. Objective: To compare performance of two well described DMAs (proportional reporting ratios [PRRs] and an empirical Bayesian algorithm known as multi-item gamma Poisson shrinker [MGPS]) using commonly recommended thresholds on a diverse data set of adverse events that triggered drug labelling changes. Methods: PRRs and MGPS were retrospectively applied to a diverse sample of drug-event combinations (DECs) identified on a government Internet site for a 7-month period. Metrics for this comparative analysis included the number and proportion of these DECs that generated signals of disproportionate reporting with PRRs, MGPS, both or neither method, differential timing of signal generation between the two methods, and clinical nature of events that generated signals with only one, both or neither method. Results: There were 136 relevant DECs that triggered safety-related labelling changes for 39 drugs during a 7-month period. PRRs generated a signal of disproportionate reporting with almost twice as many DECs as MGPS (77 vs 40). No DECs were flagged by MGPS only. PRRs highlighted DECs in advance of MGPS (1–15 years) and a label change (1–30 years). For 59 DECs, there was no signal with either DMA. DECs generating signals of disproportionate reporting with only PRRs were both medically serious and non-serious. Discussion/conclusion: In most instances in which a DEC generated a signal of disproportionate reporting with both DMAs (almost twice as many with PRRs), the signal was generated using PRRs in advance of MGPS. No medically important events were signalled only by MGPS. It is likely that the incremental utility of DMAs are highly situation-dependent. It is clear, however, that the volume of signals generated by itself is an inadequate criterion for comparison and that clinical nature of signalled events and differential timing of signals needs to be considered. Accepting commonly recommended threshold criteria for DMAs examined in this study as universal benchmarks for signal detection is not justified.

Potential Use of Data-Mining Algorithms for the Detection of 'Surprise' Adverse Drug Reactions

AbstractBackground and objective: Various data mining algorithms (DMAs) that perform disporportionality analysis on spontaneous reporting system (SRS) data are being heavily promoted to improve drug safety surveillance. The incremental value of DMAs is ultimately related to their ability to detect truly unexpected associations that would have escaped traditional surveillance and/or their ability to identify the same associations as traditional methods but with greater scientific efficiency. As to the former potential benefit, in the course of evaluating DMAs, we have observed what we call ‘surprise reactions’. These adverse reactions may be discounted in manual review of adverse drug reaction (ADR) lists because they are less clinically dramatic, less characteristic of drug effects in general, less serious than the classical type B hypersensitivity reactions or may have subtle pharmacological explanations. Thus these reactions may only become recognised when post hoc explanations are sought based on more refined pharmacological knowledge of the formulation. The objective of this study was to explore notions of ‘unexpectedness’ as relates to signal detection and data mining by introducing the concept of ‘surprise reactions’ and to determine if the latter associations, often first reported in the literature, represent a type of ADR amenable to detection with the assistance of adjunctive statistical calculations on SRS data. Methods: Using commonly cited thresholds, the multi-item gamma Poisson shrinker (MGPS) and proportional reporting ratios (PRRs) were applied to reports in the US FDA Adverse Event Reporting System (AERS) database of well documented ‘surprise reactions’ compiled by the authors. Results: There were 34 relevant surprise reactions involving 29 separate drugs in 17 different drug classes. Using PRRs (PRR >2, χ2 >4, N >2), 12 drug-event combinations were signalled before the first ADR citation appeared in MEDLINE, four occurred concurrently and 11 after. With empirical Bayes geometric mean (EBGM) analysis (EBGM >2, N >0), 12 signals occurred before, three concurrently and 11 after publication of the first literature citation. With EB05 (EB05 ≥2, N >0), six occurred before, two concurrently and 14 after MEDLINE citation. Discussion: Pharmacovigilance is rather unique in terms of the number and variety of events under surveillance. Some events may be more appropriate targets for statistical approaches than others. The experience of many organisations is that most statistical disproportionalities represent known associations but our findings suggest there could be events that may be discounted on manual review of adverse event lists, which may be usefully highlighted via DMAs. Conclusions: Identification of surprise reactions may serve as an important niche for DMAs.

Potential Utility of Data‐Mining Algorithms for Early Detection of Potentially Fatal/Disabling Adverse Drug Reactions: A Retrospective Evaluation

The objective of this study was to apply 2 data‐mining algorithms to a drug safety database to determine if these methods would have flagged potentially fatal/disabling adverse drug reactions that triggered black box warnings/drug withdrawals in advance of initial identification via “traditional” methods. Relevant drug‐event combinations were identified from a journal publication. Data‐mining algorithms using commonly cited disproportionality thresholds were then applied to the US Food and Drug Administration database. Seventy drug‐event combinations were considered sufficiently specific for retrospective data mining. In a minority of instances, potential signals of disproportionate reporting were provided clearly in advance of initial identification via traditional pharmacovigilance methods. Data‐mining algorithms have the potential to improve pharmacovigilance screening; however, for the majority of drug‐event combinations, there was no substantial benefit of either over traditional methods. They should be considered as potential supplements to, and not substitutes for, traditional pharmacovigilance strategies. More research and experience will be needed to optimize deployment of data‐mining algorithms in pharmacovigilance.

Communication of findings in pharmacovigilance : use of the term signal and the need for precision in its use

Dear Sirs, A principal concern of pharmacovigilance is the timely identification of adverse drug reactions (ADRs) to medicines that are novel in terms of their clinical nature, severity, and/or frequency. This can involve identifying ‘‘signals’’ of previously unreported drugevent combinations (DECs). There appears to be considerable semantic imprecision surrounding use of the term ‘‘signal.’’ A ‘‘signal’’ is defined by the World Health Organization (WHO) as ‘‘reported information on a possible causal relationship between an adverse event and drug, the relationship being unknown or incompletely documented previously. Usually more than a single report is required to generate a signal, depending on seriousness of event and quality of information’’ [7]. Meyboom et al. [6] provided a comprehensive definition that nicely illuminates the concept of a signal as well as the process of signal detection: ‘‘A set of data constituting a hypothesis that is relevant to the rational and safe use of a medicine. Such data are usually clinical, pharmacological, pathological or epidemiological in nature. A signal consists of a hypothesis together with data and arguments.’’ The aforementioned definitions provide a valuable framework for the execution and clear communication of signal detection activity, research, and findings. However, alternative uses for the term abound. An example appeared in a published prescription event monitoring study, noting that ‘‘signals’’ are simply ‘‘previously unrecognized adverse drug reactions’’ [5]. The crucial obligation of health authorities, pharmaceutical companies, transnational drug safety monitoring centers, and health care practitioners to cooperate in the provision and prescription of medicines that are safe, given their approved indications, has brought renewed scrutiny to the process of signal detection in light of recent high profile drug safety issues. As in any form of public health surveillance, clear communication of relevant findings to all cooperating persons and those needing to know the results of the surveillance activities (i.e., health care practitioners and patients) is of fundamental importance. These concerns are especially pertinent to pharmacovigilance for two reasons. First, pharmacovigilance typically uses multiple approaches, tools, and data sets, which may be associated with different concepts and terminology. In this context, the potential for confusion about the meaning of ‘‘signal’’ may be amplified with increasing use of statistical algorithms in signal detection, each of which has its own model assumptions, metrics, and ad hoc thresholds. Second, pharmacovigilance may be underemphasized in graduate and post-graduate medical education and training, possibly leaving key stakeholders in a suboptimal position from the perspective of receiving and understanding pharmacovigilance findings [2]. We believe that using clear and consistent terminology to describe signals and signal detection activities will facilitate the communication of surveillance information to all stakeholders and result in pharmacovigilance process improvements. Communicating to the medical community that a ‘‘signal’’ for a DEC has been identified often results from screening large drug safety databases using statistical and non-statistical procedures with various defining criteria. In one example, using non-statistical methodologies, a ‘‘signal’’ of an association of a so-called designated medical event (i.e., serious, rare, and high drugattributable risk) and drug may be said to exist based on presence of as few as 1–3 spontaneous ADRs [1]. In a second example using computational signal detection M. Hauben Æ L. Reich (&) Medical Director, Risk Management Strategy, Pfizer Inc., New York, NY, USA E-mail: lester.reich@pfizer.com

Data mining in pharmacovigilance : lessons from phantom ships

Pharmacovigilance experts devote considerable effort to post-marketing surveillance of adverse drug reactions (ADRs). Although the prepared mind of the pharmacovigilance expert remains the cornerstone of this process [1], statistical algorithms, also known as data mining algorithms (DMAs), are being promoted as supplementary tools for safety reviewers. Opinions vary on their utility and optimum deployment mainly because their use has not been completely validated for various reasons, including a lack of consensus on gold standards for causality. True positive associations may be inherently more interesting, but constructing reference sets for validation also require identification of “true negatives” for measuring performance of DMAs. Occasionally, drug-event associations (DEAs), originally considered credible based on traditional pharmacovigilance monitoring, are discounted with various levels of certitude after further investigation. We refer to these DEAs “phantom ships” [2]. Phantom associations may be discounted through epidemiological evidence, careful clinical analysis of the individual cases, and/or based on fundamental clinical pharmacological principles [3–9].

Illusions of objectivity and a recommendation for reporting data mining results

ObjectiveData mining algorithms (DMAs) are being applied to spontaneous reporting system (SRS) databases in the hope of obtaining timely insights into post-licensure safety data. Some DMAs have been characterized as “objective” screening tools. However, there are numerous available modifiable configuration parameters to choose from, including choice of vendor, that may affect results. Our objective is to compare the data mining results on pre-selected drug-event combinations (DECs) between two commonly used software programs using similar protocols.MethodsTwo DMAs, using three thresholds, were retrospectively applied to the USFDA safety database through Q2 2005 to a set of eight pre-selected DECs.ResultsDifferences between the two vendors were found for the number of cases associated with a signal of disproportionate reporting (SDR), first year of SDRs, and the magnitude of the SDR scores for the selected DECs. These were deemed to be potentially significant for 45.8% (11/24) of the data points.ConclusionThe observed differences between vendors could partially be explained by their differing methods of data cleaning and transformation as well as by the specific features of individual algorithms. The choices of vendors and available data mining configurations maximize the exploratory capacity of data mining, but they also raise questions about the claimed objectivity of data mining results and can make data mining exercises susceptible to confirmation bias given the exploratory nature of data mining in pharmacovigilance. When reporting results, the vendor and all data mining configuration details should be specified.

Multiple Cholesterol Emboli Syndrome—Six Cases Identified Through the Spontaneous Reporting System:

Six cases of suspected multiple cholesterol emboli syndrome were identified by a review of reports contained in the companys records of adverse event reports. Antecedent risk factors in these reports included cardiac catheterization, thrombolytic therapy, translumbar aortography, renal arteriography, subclavian arteriography, abdominal aortography, and heparinization. Unlike the commonly reported subacute presentation, onset occurred during or immediately after catheterization in 5 of the 6 patients reported. Acute renal failure; hypertension; back, leg, and/or abdominal pain; and livedo reticularis were the events most frequently reported. Angiographers should consider multiple choles terol embolization when multiple organ system dysfunction occurs during or immedi ately after intraarterial catheterization.

Putting the Cardiovascular Safety of Aromatase Inhibitors in Patients with Early Breast Cancer into Perspective

In the adjuvant setting, the third-generation aromatase inhibitors (AIs) anastrozole, letrozole and exemestane are recommended at some point during treatment, either in the upfront, switch after tamoxifen or extended treatment setting after tamoxifen in postmenopausal patients with hormone receptor-positive early breast cancer. AIs have demonstrated superior disease-free survival and overall benefit-to-risk profiles compared with tamoxifen. Potential adverse events, including cardiovascular (CV) side effects, should be considered in the long-term management of patients undergoing treatment with AIs. AIs reduce estrogen levels by inhibiting the aromatase enzyme, thus reducing the levels of circulating estrogen. This further reduction in estrogen levels may potentially increase the risk of developing CV disease.This systematic review evaluated published clinical data for changes in plasma lipoproteins and ischaemic CV events during adjuvant therapy with AIs in patients with hormone receptor-positive early breast cancer. The electronic databases MEDLINE, EMBASE, Derwent Drug File and BIOSIS were searched to identify English-language articles published from January 1998 to 15 April 2011 that reported data on AIs and plasma lipoproteins and/or ischaemic CV events. Overall, available data did not show any definitive patterns or suggest an unfavourable effect of AIs on plasma lipoproteins from baseline to follow-up assessment in patients with hormone receptor-positive early breast cancer. Changes that occurred in plasma lipoproteins were observed soon after initiation of AI therapy and generally remained stable throughout the studies. Available data do not support a substantial risk of ischaemic CV events associated with adjuvant AI therapy; however, studies with longer follow-up are required to better characterize the CV profile of AIs.

Detection of Spironolactone-Associated Hyperkalaemia Following the Randomized Aldactone Evaluation Study (RALES)

AbstractIntroduction: A population-based analysis has suggested that the publication of the RALES (Randomized Aldactone Evaluation Study) in late 1999 was associated with both the wider use of spironolactone to treat heart failure and a corresponding increase in hyperkalaemia-associated morbidity and mortality in patients also being treated with ACE inhibitors. Objectives: To gain further insight into the reporting of spironolactone-associated hyperkalaemia in an independent dataset by analysing the spontaneous reporting experience in relation to the publication of RALES, and to determine whether the implementation of a commonly used data mining algorithm (DMA) might have directed the attention of safety reviewers to the spironolactone/hyperkalaemia association in advance of epidemiological findings. Methods: We calculated the reporting rate of spironolactone-associated hyperkalaemia per 1000 reports per year from 1970 through to the end of 2005 by identifying relevant cases in the US FDA Adverse Event Reporting System. We did this for reports of spironolactone-associated hyperkalaemia (where spironolactone was listed as a suspect drug) and according to whether the reports listed an ACE inhibitor as a co-suspect or concomitant medication. A further statistical analysis of the overall reporting of spironolactone (suspect drug)-associated hyperkalaemia was also performed. We also performed 3-dimensional (3-D; drug-drug-event) disproportionality analyses using a DMA known as the multi-item gamma-Poisson shrinker, which allows the calculation and display of a 3-D disproportionality metric known as the ‘interaction signal score’ (INTSS). This 1144 metric is a measure of the strength of a higher order reporting relationship of a triplet (i.e. drug-drug-event) association above and beyond what would be expected from the largest disproportionalities associated with the individual 2-way associations. Results: Visual inspection of a graph of the reporting frequency of spironolactone (suspect drug)-associated hyperkalaemia per 1000 reports was highly suggestive of a change point. The t-test on the arcsine-transformed data showed a significant difference in reporting of spironolactone-hyperkalaemia combination through 1999 compared with 2000 onwards (p < 0.001). When examining the reporting time trends according to the presence or absence of an ACE inhibitor, the change point seemed to be mostly attributable to an increase in the number of spironolactone (suspect drug)-associated hyperkalaemia reports with ACE inhibitors listed as a co-suspect drug. No obvious change points in INTSSs for spironolactone-ACE inhibitor-hyperkalaemia reports were observed. Discussion: Although we could not pinpoint the relative contribution of many possible artifacts in the reporting process, as well as increased drug exposure, increased adverse event incidence and/or a change in patient monitoring practices, to our findings, we observed a notable change in reporting frequency of spironolactone-associated hyperkalaemia in temporal proximity to the publication of RALES. Evidence of this was provided by a trend analysis depicted in a simple graph that was supported by statistical analysis. The observed trend was in large part due to increased reporting of spironolactone-associated hyperkalaemia with reported co-medication with ACE inhibitors. Conclusion: These findings are consistent with those originally reported in an epidemiological analysis. In this retrospective exercise, a simple graph was more illuminating than more complex data mining analyses.