Kristina Juhlin

Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases

BackgroundMost pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.ObjectiveThe objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.Methods220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.ResultsDifferent algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.ConclusionsIn designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.

Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank.

BackgroundDetection of unknown risks with marketed medicines is key to securing the optimal care of individual patients and to reducing the societal burden from adverse drug reactions. Large collections of individual case reports remain the primary source of information and require effective analytics to guide clinical assessors towards likely drug safety signals. Disproportionality analysis is based solely on aggregate numbers of reports and naively disregards report quality and content. However, these latter features are the very fundament of the ensuing clinical assessment.ObjectiveOur objective was to develop and evaluate a data-driven screening algorithm for emerging drug safety signals that accounts for report quality and content.MethodsvigiRank is a predictive model for emerging safety signals, here implemented with shrinkage logistic regression to identify predictive variables and estimate their respective contributions. The variables considered for inclusion capture different aspects of strength of evidence, including quality and clinical content of individual reports, as well as trends in time and geographic spread. A reference set of 264 positive controls (historical safety signals from 2003 to 2007) and 5,280 negative controls (pairs of drugs and adverse events not listed in the Summary of Product Characteristics of that drug in 2012) was used for model fitting and evaluation; the latter used fivefold cross-validation to protect against over-fitting. All analyses were performed on a reconstructed version of VigiBase® as of 31 December 2004, at around which time most safety signals in our reference set were emerging.ResultsThe following aspects of strength of evidence were selected for inclusion into vigiRank: the numbers of informative and recent reports, respectively; disproportional reporting; the number of reports with free-text descriptions of the case; and the geographic spread of reporting. vigiRank offered a statistically significant improvement in area under the receiver operating characteristics curve (AUC) over screening based on the Information Component (IC) and raw numbers of reports, respectively (0.775 vs. 0.736 and 0.707, cross-validated).ConclusionsAccounting for multiple aspects of strength of evidence has clear conceptual and empirical advantages over disproportionality analysis. vigiRank is a first-of-its-kind predictive model to factor in report quality and content in first-pass screening to better meet tomorrow’s post-marketing drug safety surveillance needs.

Zoo or Savannah? Choice of Training Ground for Evidence-Based Pharmacovigilance

Pharmacovigilance seeks to detect and describe adverse drug reactions early. Ideally, we would like to see objective evidence that a chosen signal detection approach can be expected to be effective. The development and evaluation of evidence-based methods require benchmarks for signal detection performance, and recent years have seen unprecedented efforts to build such reference sets. Here, we argue that evaluation should be made against emerging and not established adverse drug reactions, and we present real-world examples that illustrate the relevance of this to pharmacovigilance methods development for both individual case reports and longitudinal health records. The establishment of broader reference sets of emerging safety signals must be made a top priority to achieve more effective pharmacovigilance methods development and evaluation.

Outlier removal to uncover patterns in adverse drug reaction surveillance – a simple unmasking strategy

This study aimed to develop an algorithm for uncovering associations masked by extreme reporting rates, characterize the occurrence of masking by influential outliers in two spontaneous reporting databases and evaluate the impact of outlier removal on disproportionality analysis.

Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases

IntroductionDisproportionality analyses are used in many organisations to identify adverse drug reactions (ADRs) from spontaneous report data. Reporting patterns vary over time, with patient demographics, and between different geographical regions, and therefore subgroup analyses or adjustment by stratification may be beneficial.ObjectiveThe objective of this study was to evaluate the performance of subgroup and stratified disproportionality analyses for a number of key covariates within spontaneous report databases of differing sizes and characteristics.MethodsUsing a reference set of established ADRs, signal detection performance (sensitivity and precision) was compared for stratified, subgroup and crude (unadjusted) analyses within five spontaneous report databases (two company, one national and two international databases). Analyses were repeated for a range of covariates: age, sex, country/region of origin, calendar time period, event seriousness, vaccine/non-vaccine, reporter qualification and report source.ResultsSubgroup analyses consistently performed better than stratified analyses in all databases. Subgroup analyses also showed benefits in both sensitivity and precision over crude analyses for the larger international databases, whilst for the smaller databases a gain in precision tended to result in some loss of sensitivity. Additionally, stratified analyses did not increase sensitivity or precision beyond that associated with analytical artefacts of the analysis. The most promising subgroup covariates were age and region/country of origin, although this varied between databases.ConclusionsSubgroup analyses perform better than stratified analyses and should be considered over the latter in routine first-pass signal detection. Subgroup analyses are also clearly beneficial over crude analyses for larger databases, but further validation is required for smaller databases.

Adverse drug reaction reports for cardiometabolic drugs from sub-Saharan Africa: a study in VigiBase.

Identifying key features in individual case safety reports (ICSR) of suspected adverse drug reactions (ADRs) with cardiometabolic drugs from sub‐Saharan Africa (SSA) compared with reports from the rest of the world (RoW).

Authors’ Reply to Harpaz et al. Comment on: “Zoo or Savannah? Choice of Training Ground for Evidence-Based Pharmacovigilance”

Harpaz et al. [1] have provided a thoughtful response to our recently published commentary on the choice of reference set in pharmacovigilance methods development and evaluation [2]. Their letter offers important complementary perspectives. We share their view that the development and maintenance of reference sets of emerging adverse drug reactions are challenging, and will benefit from a concerted effort. Harpaz et al. draw attention to the risk that backdated analyses may lack external validity for the context in which signal detection is performed today, and argue that efforts must be made to establish reference sets of more recently emerging adverse drug reactions. We agree: in some cases, changes in the properties of the underlying data could render conclusions based on historical data invalid, and this must be carefully considered on a case by case basis; in other cases, such changes will be inconsequential. Harpaz et al. note that more recently emerging adverse drug reactions may eventually be refuted. However, from a practical perspective this may not be a major limitation. Statistical signal detection is used to focus the attention of the pharmacovigilance assessors on those suspected adverse drug reactions that merit closer review, and this should include some that are eventually dismissed. Harpaz et al. emphasize that some post-approval adverse drug reactions emerge soon after marketing, before much evidence has gathered in a particular data set. Indeed, this reflects the reality of pharmacovigilance where no single source of information is capable of detecting all emerging adverse drug reactions early, and highly sensitive methods are sometimes required for timely discovery. Boosting difficult positive controls with post-detection data would be counterproductive and disguise the limitations of a data set or analysis method at hand. We agree that time-to-detection can be an important dimension for performance evaluation. Examples of its previous use include the comparison between regression and disproportionality analysis by Caster et al. [3] and the comparison between statistical and traditional signal detection by Alvarez et al. [4]. Time-to-detection is also one of the core metrics in the ongoing performance evaluations across multiple spontaneous reporting systems, within the European public-private partnership PROTECT (http://www.imi-protect.eu/). However, consideration of timeliness does not eliminate the need to distinguish between emerging and established adverse drug reactions; we would not recommend a comparison of statistical signal detection methods based on how early in the post-marketing phase they signalled adverse drug reactions that were known already from pre-marketing clinical trials. We commend Harpaz et al. for developing a reference set of recently emerging adverse drug reactions and making it openly available. This should enable more relevant pharmacovigilance performance evaluation, and provide a first building block of an open access test bed for statistical signal detection methods in pharmacovigilance.

A method for data-driven exploration to pinpoint key features in medical data and facilitate expert review

To develop a method for data‐driven exploration in pharmacovigilance and illustrate its use by identifying the key features of individual case safety reports related to medication errors.

Retrospective review of paediatric case reports of Stevens-Johnson syndrome and toxic epidermal necrolysis with lamotrigine from an international pharmacovigilance database

Objectives This study aims to characterise paediatric reports with lamotrigine (LTG) and Stevens-Johnson syndrome or toxic epidermal necrolysis (SJS/TEN), and to explore whether potential risk factors can be identified. Design This is a retrospective review of suspected adverse drug reaction (ADR) reports. Reported time from LTG start to SJS/TEN onset, indication for use and dose was explored. To identify potential risk groups, report features (eg, ages, patient sex, co-reported drugs) for LTG and SJS/TEN were contrasted with two reference groups in the same database, using shrinkage logOR. Setting Reports were retrieved from VigiBase, the WHO global database of individual case safety reports, in January 2015. Patients Data for patients aged ≤17 years old were extracted. Results There were 486 reports of SJS/TEN in LTG-treated paediatric patients. Ninety-seven per cent of the cases with complete information on time to onset of SJS/TEN occurred within 8 weeks of initiation of LTG therapy. The median time to onset was 15 days (IQR: 10–22 days). The proportion of SJS/TEN with LTG and valproic acid (VPA) co-reporting was significantly more than non-cutaneous ADRs (43% vs 19%, (logOR: 1.60 (99% CI: 1.33 to 1.84)). Conclusions The results suggest that VPA co-medication with LTG therapy is a risk factor for SJS/TEN in the paediatric population. Although this relationship has been identified from individual case reports, this is the first supportive study from a large compilation of cases. SJS/TEN risk is highest in first 8 weeks of treatment with LTG in children and clinicians should be aware of this risk during this period.

G96 Stevens-Johnson syndrome and toxic epidermal necrolysis associated with lamotrigine treatment in children: A review of 486 individual case safety reports

Aim SJS and TEN are life-threatening conditions, usually secondary to drug therapy. This study explored the characteristics of children with Stevens – Johnson syndrome (SJS) and Toxic epidermal necrolysis (TEN) following lamotrigine (LTG) treatment to identify potential risk factors. Methods A retrospective review of individual case safety reports (ICSRs) of SJS and TEN in LTG treated children. ICSRs were acquired from the WHO international database of suspected ADRs, which stores ADRs from national pharmacovigilance centres worldwide. Case reports of paediatric patients, ≤17 years old, were retrieved and compared with cases of LTG associated non-dermatological ADRs, cases of SJS/TEN not associated with LTG and reports of SJS/TEN associated with carbamazepine (CBZ) and phenobarbital (PBT). Results There were 486 reports of SJS/TEN in LTG treated children. The proportion of SJS/TEN reports in LTG and VPA co-medicated patients was significantly higher than the proportion reporting non-dermatological ADRs when taking both drugs (logOR 1.60; 99% CI: 1.33,1.84). A significantly higher proportion of cases of SJS/TEN, compared with all ADRs, were also reported with LTG and VPA co-medication (logOR 1.23; 99% CI: 0.96–1.47); while no significant difference was seen in the proportion of SJS/TEN reports with CBZ or PBT and VPA co-medication. Ninety six percent of the cases of SJS/TEN occurred within 6 weeks of initiation of LTG therapy. The median time to onset was 15 days [IQR: 10.8–22 days]. Conclusions LTG and VPA co-medication significantly increases the risk of SJS/TEN, which is likely to occur within 6 weeks of treatment in most children.