Mark Kroese

How can the evaluation of genetic tests be enhanced? Lessons learned from the ACCE framework and evaluating genetic tests in the United Kingdom

Advances in genetic technology are increasing the availability of genetic tests, not only for rare single gene disorders, but also for common diseases such as breast and colo-rectal cancer. Before there can be widespread uptake of these tests, they must be evaluated to confirm the benefits of their use. But how should genetic tests be evaluated, given the speed at which new tests are emerging? One highly influential approach is the analytic validity, clinical validity, clinical utility and ethical, legal and social issues (ACCE) framework, which has provided a benchmark for the evaluation of genetic tests. The approach has been adopted and adapted by the United Kingdom Genetic Testing Network, with the help of the Public Health Genetics Unit in Cambridge, to evaluate new genetic tests for use in the National Health Service. We discuss a number of conceptual, methodological, and practical issues concerning the evaluation of genetic tests, based on lessons learned from applying the ACCE framework and from the UK experience, and make a number of recommendations to further strengthen the evaluation of genetic tests.

Genetic tests and their evaluation: Can we answer the key questions?

The rapid pace of research in the field of genetics has already yielded many benefits. The development of new genetic tests is one such example. Before there can be widespread uptake of these tests they need to be evaluated to confirm the benefits of their use. The authors review some of the key features of the evaluation of diagnostic tests focusing on analytical and clinical validity. Test properties such as sensitivity, specificity, likelihood ratios, positive and negative predictive values, and how they relate to molecular genetic testing are discussed. Associated issues such as the concepts of disease definition, imperfect reference standards, and false positives are also explored. The authors suggest possible approaches to addressing some of the problems identified.

Evaluation of non-invasive prenatal testing (NIPT) for aneuploidy in an NHS setting: a reliable accurate prenatal non-invasive diagnosis (RAPID) protocol

BackgroundNon-invasive prenatal testing (NIPT) for aneuploidies is now available through commercial companies in many countries, including through private practice in the United Kingdom (UK). Thorough evaluation of service delivery requirements are needed to facilitate NIPT being offered more widely within state funded healthcare systems such as the UK’s National Health Service (NHS). Successful implementation will require the development of laboratory standards, consideration of stakeholder views, an analysis of costs and development of patient and health professional educational materials.Methods/DesignNIPT will be offered in an NHS setting as a contingent screening test. Pregnant woman will be recruited through six maternity units in England and Scotland. Women eligible for Down’s syndrome screening (DSS) will be informed about the study at the time of booking. Women that choose routine DSS will be offered NIPT if they have a screening risk ≥1:1000. NIPT results for trisomy 21, 18, 13 will be reported within 7–10 working days. Data on DSS, NIPT and invasive testing uptake, pregnancy outcomes and test efficacy will be collected. Additional data will be gathered though questionnaires to a) determine acceptability to patients and health professionals, b) evaluate patient and health professional education, c) assess informed choice in women accepting or declining testing and d) gauge family expenses. Qualitative interviews will also be conducted with a sub-set of participating women and health professionals.DiscussionThe results of this study will make a significant contribution to policy decisions around the implementation of NIPT for aneuploidies within the UK NHS. The laboratory standards for testing and reporting, education materials and counselling strategies developed as part of the study are likely to underpin the introduction of NIPT into NHS practice.NIHR Portfolio Number13865

Uptake, outcomes, and costs of implementing non-invasive prenatal testing for Down's syndrome into NHS maternity care: prospective cohort study in eight diverse maternity units.

Objective To investigate the benefits and costs of implementing non-invasive prenatal testing (NIPT) for Down’s syndrome into the NHS maternity care pathway. Design Prospective cohort study. Setting Eight maternity units across the United Kingdom between 1 November 2013 and 28 February 2015. Participants All pregnant women with a current Down’s syndrome risk on screening of at least 1/1000. Main outcome measures Outcomes were uptake of NIPT, number of cases of Down’s syndrome detected, invasive tests performed, and miscarriages avoided. Pregnancy outcomes and costs associated with implementation of NIPT, compared with current screening, were determined using study data on NIPT uptake and invasive testing in combination with national datasets. Results NIPT was prospectively offered to 3175 pregnant women. In 934 women with a Down’s syndrome risk greater than 1/150, 695 (74.4%) chose NIPT, 166 (17.8%) chose invasive testing, and 73 (7.8%) declined further testing. Of 2241 women with risks between 1/151 and 1/1000, 1799 (80.3%) chose NIPT. Of 71 pregnancies with a confirmed diagnosis of Down’s syndrome, 13/42 (31%) with the diagnosis after NIPT and 2/29 (7%) after direct invasive testing continued, resulting in 12 live births. In an annual screening population of 698 500, offering NIPT as a contingent test to women with a Down’s syndrome screening risk of at least 1/150 would increase detection by 195 (95% uncertainty interval −34 to 480) cases with 3368 (2279 to 4027) fewer invasive tests and 17 (7 to 30) fewer procedure related miscarriages, for a non-significant difference in total costs (£−46 000, £−1 802 000 to £2 661 000). The marginal cost of NIPT testing strategies versus current screening is very sensitive to NIPT costs; at a screening threshold of 1/150, NIPT would be cheaper than current screening if it cost less than £256. Lowering the risk threshold increases the number of Down’s syndrome cases detected and overall costs, while maintaining the reduction in invasive tests and procedure related miscarriages. Conclusions Implementation of NIPT as a contingent test within a public sector Down’s syndrome screening programme can improve quality of care, choices for women, and overall performance within the current budget. As some women use NIPT for information only, the Down’s syndrome live birth rate may not change significantly. Future research should consider NIPT uptake and informed decision making outside of a research setting.

Evaluation of genetic tests for susceptibility to common complex diseases: why, when and how?

Recent research into the human genome has generated a wealth of scientific knowledge and increased both public and professional interest in the concept of personalised medicine. Somewhat unexpectedly, in addition to increasing our understanding about the genetic basis for numerous diseases, these new discoveries have also spawned a burgeoning new industry of ‘consumer genetic testing’. In this paper, we present the principles learnt though the evaluation of tests for single gene disorders and suggest a comparable framework for the evaluation of genetic tests for susceptibility to common complex diseases. Both physicians and the general public will need to be able to assess the claims made by providers of genetic testing services, and ultimately policy-makers will need to decide if and when such tests should be offered through state funded healthcare systems.

How can genetic tests be evaluated for clinical use? Experience of the UK Genetic Testing Network

The UK Department of Health supported the establishment of the UK Genetic Testing Network (UKGTN) in 2002. The UKGTN is a collaborative network of NHS molecular genetic laboratories that offer tests for human single gene germ-line disorders. Its objective is to provide high quality and equitable services for patients and their families who require genetic advice, diagnosis and management. The UKGTN has developed a ‘Gene Dossier’ process to evaluate genetic tests and recommend which tests will be provided by the National Health Service. This paper describes the UKGTN organisation and the ‘Gene Dossier’ process. A brief review of the UKGTN genetic test evaluation experience is presented.

Defining purpose: a key step in genetic test evaluation

The introduction of new genetic tests, like other medical innovations, can be conceptualized as a three-step process. Tests are proposed for use based on research findings and clinical reasoning; an evaluation occurs; and judgments are made about clinical use and reimbursement (Fig. 1). The evaluation may be informal, as when a clinician determines whether a new test will be helpful in a particular patient encounter, or formal, as when a practice guideline panel utilizes a defined methodology to assess a test or a health care funder utilizes a set of criteria to determine test coverage. Although genetic tests are often described in terms of technology, a full evaluation requires that the test be considered as a clinical process in which the laboratory assay, or other testing procedure, is done to acquire information about a particular health condition, in a defined population, for a specific clinical purpose.1 Genetic tests have a wide range of health care applications. They are used to confirm the presence of a genetic condition, identify reproductive risks, and select preventive therapy. Testing occurs in newborn screening programs and in primary, specialty, and prenatal care, and may be initiated on the basis of clinical symptoms, family history, or population demographics. Genetic tests also utilize a range of technologies, and vary considerably in their predictive value. This diversity poses a challenge for the evaluation process. Several groups have considered methods for genetic test evaluation,2–5 and model programs for systematic evaluation have been established in the United States (Evaluation of Genomic Applications in Practice and Prevention6) and the UK (the UK Genetic Testing Network7). To date, however, the evaluation process has not fully addressed the clinical diversity of genetic tests. The Evaluation of Genomic Applications in Practice and Prevention project has focused on genetic tests related to common disorders and drug therapy, whereas the United Kingdom Genetic Testing Network has primarily addressed tests for single gene disorders. An effort by the Secretary’s Advisory Committee on Genetic Testing to categorize genetic tests into those requiring higher versus lower levels of scrutiny was not successful.8 To address the challenge of genetic test diversity, we propose an outcome-orientated taxonomy for defining test purpose. A focus on health outcomes allows the definition of a small and informative set of purposes for genetic testing, despite the range of technologies and clinical settings in which testing occurs. Defining test purpose in turn clarifies the benefits to be expected from the testing process, and provides guidance to clinicians and policy makers concerning the evidence needed to support test use. Defining test purpose, therefore, is an important first step in genetic test evaluation.

Array-based comparative genomic hybridization for investigating chromosomal abnormalities in patients with learning disability: systematic review meta-analysis of diagnostic and false-positive yields.

Purpose: Array-based comparative genomic hybridization is increasingly being used in patients with learning disability, in addition to existing cytogenetic techniques. This paper reports the results of an evaluation of this emerging technology and discusses the challenges faced in conducting the evaluation.Methods: Systematic review and meta-analysis of studies investigating patients with learning disability and dysmorphic features in whom conventional cytogenetic analysis has proven negative. Conventional indices of clinical validity could not be calculated, and we use an alternative, based on the extent to which array-based comparative genomic hybridization met its clinical objectives.Results: Seven studies (462 patients) were included. The overall diagnostic yield of causal abnormalities was 13% (95% confidence interval: 10–17%; heterogeneity test statistic I2 = 0%), and the overall number needed to test was eight (95% confidence interval: 6–10). The false-positive yield of noncausal abnormalities ranged from 5% to 67%, although this range was only 5% to 10% in six of the studies.Conclusion: Although promising, there is insufficient evidence to recommend introduction of this test into routine clinical practice. A number of important technical questions need answering, such as optimal array resolution, which clones to include, and the most appropriate platforms. A thorough assessment of clinical utility and cost-effectiveness compared with existing tests is also required.

Points to consider for prioritizing clinical genetic testing services: a European consensus process oriented at accountability for reasonableness.

Given the cost constraints of the European health-care systems, criteria are needed to decide which genetic services to fund from the public budgets, if not all can be covered. To ensure that high-priority services are available equitably within and across the European countries, a shared set of prioritization criteria would be desirable. A decision process following the accountability for reasonableness framework was undertaken, including a multidisciplinary EuroGentest/PPPC-ESHG workshop to develop shared prioritization criteria. Resources are currently too limited to fund all the beneficial genetic testing services available in the next decade. Ethically and economically reflected prioritization criteria are needed. Prioritization should be based on considerations of medical benefit, health need and costs. Medical benefit includes evidence of benefit in terms of clinical benefit, benefit of information for important life decisions, benefit for other people apart from the person tested and the patient-specific likelihood of being affected by the condition tested for. It may be subject to a finite time window. Health need includes the severity of the condition tested for and its progression at the time of testing. Further discussion and better evidence is needed before clearly defined recommendations can be made or a prioritization algorithm proposed. To our knowledge, this is the first time a clinical society has initiated a decision process about health-care prioritization on a European level, following the principles of accountability for reasonableness. We provide points to consider to stimulate this debate across the EU and to serve as a reference for improving patient management.

A framework for the prioritization of investment in the provision of genetic tests.

Background: The UK Genetic Testing Network (UKGTN) established a process for the evaluation of genetic tests for entry onto the National Health Service (NHS) Directory of Molecular Genetic Testing. The Network requested the development and piloting of a prioritization framework that could be used for the commissioning of genetic tests by the NHS. Methods: A selected working group developed and piloted a multi-criteria prioritization process using 10 genetic tests evaluated by the UKGTN. Results: The framework was able to rank the 10 genetic tests used in the pilot. The rankings were also consistent with the commissioning recommendations for these genetic tests by the UKGTN. Conclusion:A set of criteria for the prioritization of genetic tests has been developed. The results from the pilot suggest that the methodology is valid and robust but requires considerable resources to implement. Further development of the process is needed before the framework could be used to influence commissioning decisions for clinical genetic services in the NHS.