Seb Bacon

FDAAA TrialsTracker: A live informatics tool to monitor compliance with FDA requirements to report clinical trial results

Introduction Non-publication of clinical trials results is an ongoing issue. The US government recently updated the requirements on results reporting for trials registered at ClinicalTrials.gov. We set out to develop and deliver an online tool which publicly monitors compliance with these reporting requirements, facilitates open public audit, and promotes accountability. Methods We conducted a review of the relevant legislation to extract the requirements on reporting results. Specific areas of the statutes were operationalized in code based on the results of our policy review, and on the publicly available data from ClinicalTrials.gov. We developed methods to identify trials required to report results, using publicly accessible data; to download additional relevant information such as key dates and trial sponsors; and to determine when each trial became due. This data was then used to construct a live tracking website. Results There were a number of administrative and technical hurdles to successful operationalization in our tracker. Decisions and assumptions related to overcoming these issues are detailed along with clarifying details from outreach directly to ClinicalTrials.gov. The FDAAA TrialsTracker was successfully launched and provides users with an overview of results reporting compliance. Discussion Clinical trials continue to go unreported despite numerous guidelines, commitments and legal frameworks intended to address this issue. In the absence of formal sanctions from the FDA and others, we argue tools such as ours providing live data on trial reporting - can improve accountability and performance. In addition, our service helps sponsors identify their own individual trials that have not yet reported results: we therefore offer positive practical support for sponsors who wish to ensure that all their completed trials have reported.

Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource

The BMJ Video Playergolb038037 Abstract Objectives To ascertain compliance rates with the European Commission’s requirement that all trials on the EU Clinical Trials Register (EUCTR) post results to the registry within 12 months of completion (final compliance date 21 December 2016); to identify features associated with non-compliance; to rank sponsors by compliance; and to build a tool for live ongoing audit of compliance. Design Retrospective cohort study. Setting EUCTR. Participants 7274 of 11u2009531 trials listed as completed on EUCTR and where results could be established as due. Main outcome measure Publication of results on EUCTR. Results Of 7274 trials where results were due, 49.5% (95% confidence interval 48.4% to 50.7%) reported results. Trials with a commercial sponsor were substantially more likely to post results than those with a non-commercial sponsor (68.1% v 11.0%, adjusted odds ratio 23.2, 95% confidence interval 19.2 to 28.2); as were trials by a sponsor who conducted a large number of trials (77.9% v 18.4%, adjusted odds ratio 18.4, 15.3 to 22.1). More recent trials were more likely to report results (per year odds ratio 1.05, 95% confidence interval 1.03 to 1.07). Extensive evidence was found of errors, omissions, and contradictory entries in EUCTR data that prevented ascertainment of compliance for some trials. Conclusions Compliance with the European Commission requirement for all trials to post results on to the EUCTR within 12 months of completion has been poor, with half of all trials non-compliant. EU registry data commonly contain inconsistencies that might prevent even regulators assessing compliance. Accessible and timely information on the compliance status of each individual trial and sponsor may help to improve reporting rates.

Trends, geographical variation and factors associated with prescribing of gluten-free foods in English primary care: a cross-sectional study.

Objectives There is substantial disagreement about whether gluten-free foods should be prescribed on the National Health Service. We aim to describe time trends, variation and factors associated with prescribing gluten-free foods in England. Setting English primary care. Participants English general practices. Primary and secondary outcome measures We described long-term national trends in gluten-free prescribing, and practice and Clinical Commissioning Group (CCG) level monthly variation in the rate of gluten-free prescribing (per 1000 patients) over time. We used a mixed-effect Poisson regression model to determine factors associated with gluten-free prescribing rate. Results There were 1.3 million gluten-free prescriptions between July 2016 and June 2017, down from 1.8 million in 2012/2013, with a corresponding cost reduction from £25.4 million to £18.7 million. There was substantial variation in prescribing rates among practices (range 0 to 148 prescriptions per 1000 patients, IQR 7.3–31.8), driven in part by substantial variation at the CCG level, likely due to differences in prescribing policy. Practices in the most deprived quintile of deprivation score had a lower prescribing rate than those in the highest quintile (incidence rate ratio 0.89, 95% CI 0.87 to 0.91). This is potentially a reflection of the lower rate of diagnosed coeliac disease in more deprived populations. Conclusion Gluten-free prescribing is in a state of flux, with substantial clinically unwarranted variation between practices and CCGs.

Is use of homeopathy associated with poor prescribing in English primary care? A cross-sectional study

Objectives Prescribing of homeopathy still occurs in a small minority of English general practices. We hypothesised that practices that prescribe any homeopathic preparations might differ in their prescribing of other drugs. Design Cross-sectional analysis. Setting English primary care. Participants English general practices. Main outcome measures We identified practices that made any homeopathy prescriptions over six months of data. We measured associations with four prescribing and two practice quality indicators using multivariable logistic regression. Results Only 8.5% of practices (644) prescribed homeopathy between December 2016 and May 2017. Practices in the worst-scoring quartile for a composite measure of prescribing quality (>51.4 mean percentile) were 2.1 times more likely to prescribe homeopathy than those in the best category (<40.3) (95% confidence interval: 1.6–2.8). Aggregate savings from the subset of these measures where a cost saving could be calculated were also strongly associated (highest vs. lowest quartile multivariable odds ratio: 2.9, confidence interval: 2.1–4.1). Of practices spending the most on medicines identified as ‘low value’ by NHS England, 12.8% prescribed homeopathy, compared to 3.9% for lowest spenders (multivariable odds ratio: 2.6, confidence interval: 1.9–3.6). Of practices in the worst category for aggregated price-per-unit cost savings, 12.7% prescribed homeopathy, compared to 3.5% in the best category (multivariable odds ratio: 2.7, confidence interval: 1.9–3.9). Practice quality outcomes framework scores and patient recommendation rates were not associated with prescribing homeopathy (odds ratio range: 0.9–1.2). Conclusions Even infrequent homeopathy prescribing is strongly associated with poor performance on a range of prescribing quality measures, but not with overall patient recommendation or quality outcomes framework score. The association is unlikely to be a direct causal relationship, but may reflect underlying practice features, such as the extent of respect for evidence-based practice, or poorer stewardship of the prescribing budget.

Trends and variation in prescribing of low-priority treatments identified by NHS England: a cross-sectional study and interactive data tool in English primary care:

Objectives NHS England recently announced a consultation seeking to discourage the use of treatments it considers to be low-value. We set out to produce an interactive data resource to show savings in each NHS general practice and to assess the current use of these treatments, their change in use over time, and the extent and reasons for variation in such prescribing. Design Cross-sectional analysis. Setting English primary care. Participants English general practices. Main outcome measures We determined the cost per 1000 patients for prescribing of each of 18 treatments identified by NHS England for each month from July 2012 to June 2017, and also aggregated over the most recent year to assess total cost and variation among practices. We used mixed effects linear regression to determine factors associated with cost of prescribing. Results Spend on low-value treatments was £153.5u2009m in the last year, across 5.8u2009m prescriptions (mean, £26 per prescription). Among individual treatments, liothyronine had the highest prescribing cost at £29.6u2009m, followed by trimipramine (£20.2u2009m). Over time, the overall total number of low-value prescriptions decreased, but the cost increased, although this varied greatly between treatments. Three treatment areas increased in cost and two increased in volume, all others reduced in cost and volume. Annual practice level spending varied widely (median, £2262 per thousand patients; interquartile range £1439 to £3298). Proportion of patients over 65 was strongly associated with low-value prescribing, as was Clinical Commissioning Group. Our interactive data tool was deployed to OpenPrescribing.net where monthly updated figures and graphs can be viewed. Conclusions Prescribing of low-value treatments is extensive but varies widely by treatment, geographic area and individual practice. Despite a fall in prescription numbers, the overall cost of prescribing for low-value items has risen. Prescribing behaviour is clustered by Clinical Commissioning Group, which may represent variation in the optimisation efficiency of medicines, or in some cases access inequality.

Time trends and geographical variation in prescribing of drugs for diabetes in England from 1998 to 2017.

To measure the variation in prescribing of second‐line non‐insulin diabetes drugs.

New mechanism to identify cost savings in English NHS prescribing: minimising ‘price per unit’, a cross-sectional study

Background Minimising prescription costs while maintaining quality is a core element of delivering high-value healthcare. There are various strategies to achieve savings, but almost no research to date on determining the most effective approach. We describe a new method of identifying potential savings due to large national variations in drug cost, including variation in generic drug cost, and compare these with potential savings from an established method (generic prescribing). Methods We used English National Health Service (NHS) Digital prescribing data, from October 2015 to September 2016. Potential cost savings were calculated by determining the price per unit (eg, pill, millilitre) for each drug and dose within each general practice. This was compared against the same cost for the practice at the lowest cost decile to determine achievable savings. We compared these price-per-unit savings to the savings possible from generic switching and determined the chemicals with the highest savings nationally. A senior pharmacist manually assessed whether a random sample of savings were practically achievable. Results We identified a theoretical maximum of £410 million of savings over 12 months. £273 million of these savings were for individual prescribing changes worth over £50 per practice per month (mean annual saving £33u2009433 per practice); this compares favourably with generic switching, where only £35 million of achievable savings were identified. The biggest savings nationally were on glucose blood testing reagents (£12 million), fluticasone propionate (£9 million) and venlafaxine (£8 million). Approximately half of all savings were deemed practically achievable. Discussion We have developed a new method to identify and enable large potential cost savings within NHS community prescribing. Given the current pressures on the NHS, it is vital that these potential savings are realised. Our tool enabling doctors to achieve these savings is now launched in pilot form at OpenPrescribing.net. However, savings could potentially be achieved more simply through national policy change.

Measuring the Impact of an Open Online Prescribing Data Analysis Service on Clinical Practice: a Cohort Study in NHS England Data (Preprint)

Background OpenPrescribing is a freely accessible service that enables any user to view and analyze the National Health Service (NHS) primary care prescribing data at the level of individual practices. This tool is intended to improve the quality, safety, and cost-effectiveness of prescribing. Objective We aimed to measure the impact of OpenPrescribing being viewed on subsequent prescribing. Methods Having preregistered our protocol and code, we measured three different metrics of prescribing quality (mean percentile across 34 existing OpenPrescribing quality measures, available “price-per-unit” savings, and total “low-priority prescribing” spend) to see whether they changed after the viewing of Clinical Commissioning Group (CCG) and practice pages. We also measured whether practices whose data were viewed on OpenPrescribing differed in prescribing, prior to viewing, compared with those who were not. We used fixed-effects and between-effects linear panel regression to isolate change over time and differences between practices, respectively. We adjusted for the month of prescribing in the fixed-effects model to remove underlying trends in outcome measures. Results We found a reduction in available price-per-unit savings for both practices and CCGs after their pages were viewed. The saving was greater at practice level (−£40.42 per thousand patients per month; 95% CI −54.04 to −26.81) than at CCG level (−£14.70 per thousand patients per month; 95% CI −25.56 to −3.84). We estimate a total saving since launch of £243 thosand at practice level and £1.47 million at CCG level between the feature launch and end of follow-up (August to November 2017) among practices viewed. If the observed savings from practices viewed were extrapolated to all practices, this would generate £26.8 million in annual savings for the NHS, approximately 20% of the total possible savings from this method. The other two measures were not different after CCGs or practices were viewed. Practices that were viewed had worse prescribing quality scores overall prior to viewing. Conclusions We found a positive impact from the use of OpenPrescribing, specifically for the class of savings opportunities that can only be identified by using this tool. Furthermore, we show that it is possible to conduct a robust analysis of the impact of such a Web-based service on clinical practice.

7 Producing data-driven tools

Objectives EBM DataLab is a team that aims to produce tools from data, rather than just academic publications. We have a multidisciplinary approach that combines the skills of software engineers, clinicians and academics. In this workshop we will give an overview covering how to make effective interactive services driven by data, through: Providing an overview of the software and services which can be used to create your own data–driven tools. Demonstrating examples from the work of the EBM DataLab. Providing an opportunity for attendees to discuss their own ideas or experiences related to data–driven tools. Method In this session you will learn the very basics about a range of useful software and services including Python, pandas and iPython notebooks, GitHub, Google Analytics, Google Sheets, Google Forms, API’s (for Scopus, PubMed, and Web of Science), SQL databases such as BigQuery and Postgres, and graphical tools such as Tableau and d3. We will demonstrate the creative use of these tools with worked examples from our recent output including the Retractobot, various trials trackers, OpenPrescribing long term trends, the Drug Tariff explorer, and more. Results Attendees should leave this workshop with a better grasp of how to identify user needs, select the appropriate software or services for the job, design tools to be deliverable and impactful, manage the development cycle from prototyping to launch, and carry out the basic process of user testing. Conclusions Simple yet engaging presentation of data that allows people to act on it can be a critical step in disseminating evidence and improving quality of care. Live updating dynamic tools can keep information at the cutting-edge and provide a platform that provides real and enduring value to users. There are great resources available that can allow researchers to quickly and efficiently turn their findings into public-facing tools. We hope this workshop will empower attendees to begin presenting and sharing their data in new and effective ways in order to promote positive change in their respective fields.

Detecting change in comparison to peers in NHS prescribing data: a novel application of cumulative sum methodology

BackgroundThe widely used OpenPrescribing.net service provides standard measures which compare prescribing of Clinical Commissioning Groups (CCGs) and English General Practices against that of their peers. Detecting changes in prescribing behaviour compared with peers can help identify missed opportunities for medicines optimisation. Automating the process of detecting these changes is necessary due to the volume of data, but challenging due to variation in prescribing volume for different measures and locations. We set out to develop and implement a method of detecting change on all individual prescribing measures, in order to notify CCGs and practices of such changes in a timely manner.MethodsWe used the statistical process control method CUSUM to detect prescribing behaviour changes in relation to population trends for the individual standard measures on OpenPrescribing. Increases and decreases in percentile were detected separately, using a multiple of standard deviation as the threshold for detecting change. The algorithm was modified to continue re-triggering when trajectory persists. It was deployed, user-tested, and summary statistics generated on the number of alerts by CCG and practice.ResultsThe algorithm detected changes in prescribing for 32 prespecified measures, across a wide range of CCG and practice sizes. Across the 209 English CCGs, a mean of 2.5 increase and 2.4 decrease alerts were triggered per CCG, per month. For the 7578 practices, a mean of 1.3 increase and 1.4 decrease alerts were triggered per practice, per month.ConclusionsThe CUSUM method appears to effectively discriminate between random noise and sustained change in prescribing behaviour. This method aims to allow practices and CCGs to be informed of important changes quickly, with a view to improve their prescribing behaviour. The number of alerts triggered for CCGs and practices appears to be appropriate. Prescribing behaviour after users are alerted to changes will be monitored in order to assess the impact of these alerts.