Bethany Shinkins

Diagnostic accuracy studies: how to report and analyse inconclusive test results.

Failure to report inconclusive test results can lead to misleading conclusions regarding the accuracy and clinical usefulness of a diagnostic tool. We show that these results are often overlooked in research on test accuracy and provide guidance on suitable approaches to reporting and analysing these problematic results.

Symptoms, unmet needs, psychological well-being and health status in survivors of prostate cancer: implications for redesigning follow-up.

To explore ongoing symptoms, unmet needs, psychological wellbeing, self‐efficacy and overall health status in survivors of prostate cancer.

RIPOSTE: a framework for improving the design and analysis of laboratory-based research

Lack of reproducibility is an ongoing problem in some areas of the biomedical sciences. Poor experimental design and a failure to engage with experienced statisticians at key stages in the design and analysis of experiments are two factors that contribute to this problem. The RIPOSTE (Reducing IrreProducibility in labOratory STudiEs) framework has been developed to support early and regular discussions between scientists and statisticians in order to improve the design, conduct and analysis of laboratory studies and, therefore, to reduce irreproducibility. This framework is intended for use during the early stages of a research project, when specific questions or hypotheses are proposed. The essential points within the framework are explained and illustrated using three examples (a medical equipment test, a macrophage study and a gene expression study). Sound study design minimises the possibility of bias being introduced into experiments and leads to higher quality research with more reproducible results. DOI: http://dx.doi.org/10.7554/eLife.05519.001

Site and stage of colorectal cancer influence the likelihood and distribution of disease recurrence and postrecurrence survival: data from the FACS randomized controlled trial

Objectives:To describe patterns of recurrence and postrecurrence survival in a large cohort of accurately staged patients with Dukes’ A-C colorectal cancer. Background:Recurrence remains a frequent cause of mortality after the treatment of colorectal cancer with curative intent. Understanding the likelihood and site of recurrence informs adjuvant treatment and follow-up. Methods:Retrospective cohort analysis of data from the FACS (follow-up after colorectal cancer surgery) trial after a median 4.4 years of follow-up; postrecurrence survival was calculated using the Kaplan-Meier method. Results:Complete data were available for 94% of patients; 189 (17%) patients had experienced recurrence. Incidence of recurrence varied according to the site of the primary (right colon: 51/379, 14%; left colon: 68/421, 16%; rectum: 70/332, 21%; P = 0.023) and initial stage (Dukes’ A: 26/249, 10%; Dukes’ B: 81/537, 15%; Dukes’ C: 82/346, 24%; P < 0.0001). Pulmonary recurrence was most frequently associated with rectal tumors, and multisite/other recurrence with right-sided colonic tumors. Recurrences from lower-stage tumors were more likely to be treatable with curative intent (Dukes’ A: 13/26, 50%; Dukes’ B: 32/81, 40%; Dukes’ C: 20/82, 24%; P = 0.03). Those with rectal tumors benefited most from follow-up (proportion with treatable recurrence: rectum 30/332, 9%; left colon 23/421, 6%; right colon 12/379, 3%; P = 0.003). Both initial stage (log rank P = 0.005) and site of primary (log rank P = 0.01) influenced postrecurrence survival. Conclusions:The likelihood and site of recurrence, and survival, are influenced by the site and stage of the primary tumor. Those with rectal cancers benefited most from follow-up.ISRCTN 41458548

Diagnostic uncertainty: dichotomies are not the answer

Over two decades ago, Alvan Feinstein, a leading figure in the development and evaluation of diagnostic research methods, argued that the binary ‘positive–negative’ framework used in test accuracy research is not representative of diagnostic decision making in clinical practice.1 In particular, he called for a move away from the dichotomisation of quantitative test scales (tests that are on a continuous or ordinal scale) and advocated an approach that allows for the explicit recognition of an ‘uncertain’ diagnostic outcome. Twenty years on and little progress has been made on this issue; reporting the accuracy of quantitative tests based on a single ‘optimal’ threshold continues to be common practice. Here we present how improving the methods for evaluating quantitative tests would be of greatest benefit to GPs, providing them with a better evidence-based toolkit of strategies for recognising and handling diagnostic uncertainty head-on. Summerton reported that failures relating to diagnosis account for nearly one-third of GP complaints.2 Diagnostic uncertainty is particularly rife in general practice due to a number of obstacles intrinsic to the clinical setting.3 First, the prevalence of serious disease is typically low in a community population, weakening the overall predictive value of diagnostic tests. Secondly, the majority of disease presenting in general practice is commonly in its early stages, when many ‘red flag’ symptoms are yet to evolve and departures from physiological normality are slight. Furthermore, many of the tests used are not disease-specific; they are typically predictive of a symptom that …

Validating a decision tree for serious infection: diagnostic accuracy in acutely ill children in ambulatory care

Objective Acute infection is the most common presentation of children in primary care with only few having a serious infection (eg, sepsis, meningitis, pneumonia). To avoid complications or death, early recognition and adequate referral are essential. Clinical prediction rules have the potential to improve diagnostic decision-making for rare but serious conditions. In this study, we aimed to validate a recently developed decision tree in a new but similar population. Design Diagnostic accuracy study validating a clinical prediction rule. Setting and participants Acutely ill children presenting to ambulatory care in Flanders, Belgium, consisting of general practice and paediatric assessment in outpatient clinics or the emergency department. Intervention Physicians were asked to score the decision tree in every child. Primary outcome measures The outcome of interest was hospital admission for at least 24 h with a serious infection within 5 days after initial presentation. We report the diagnostic accuracy of the decision tree in sensitivity, specificity, likelihood ratios and predictive values. Results In total, 8962 acute illness episodes were included, of which 283 lead to admission to hospital with a serious infection. Sensitivity of the decision tree was 100% (95% CI 71.5% to 100%) at a specificity of 83.6% (95% CI 82.3% to 84.9%) in the general practitioner setting with 17% of children testing positive. In the paediatric outpatient and emergency department setting, sensitivities were below 92%, with specificities below 44.8%. Conclusions In an independent validation cohort, this clinical prediction rule has shown to be extremely sensitive to identify children at risk of hospital admission for a serious infection in general practice, making it suitable for ruling out. Trial registration number NCT02024282.

Common evidence gaps in point-of-care diagnostic test evaluation: a review of horizon scan reports

Objective Since 2008, the Oxford Diagnostic Horizon Scan Programme has been identifying and summarising evidence on new and emerging diagnostic technologies relevant to primary care. We used these reports to determine the sequence and timing of evidence for new point-of-care diagnostic tests and to identify common evidence gaps in this process. Design Systematic overview of diagnostic horizon scan reports. Primary outcome measures We obtained the primary studies referenced in each horizon scan report (n=40) and extracted details of the study size, clinical setting and design characteristics. In particular, we assessed whether each study evaluated test accuracy, test impact or cost-effectiveness. The evidence for each point-of-care test was mapped against the Horvath framework for diagnostic test evaluation. Results We extracted data from 500 primary studies. Most diagnostic technologies underwent clinical performance (ie, ability to detect a clinical condition) assessment (71.2%), with very few progressing to comparative clinical effectiveness (10.0%) and a cost-effectiveness evaluation (8.6%), even in the more established and frequently reported clinical domains, such as cardiovascular disease. The median time to complete an evaluation cycle was 9 years (IQR 5.5–12.5 years). The sequence of evidence generation was typically haphazard and some diagnostic tests appear to be implemented in routine care without completing essential evaluation stages such as clinical effectiveness. Conclusions Evidence generation for new point-of-care diagnostic tests is slow and tends to focus on accuracy, and overlooks other test attributes such as impact, implementation and cost-effectiveness. Evaluation of this dynamic cycle and feeding back data from clinical effectiveness to refine analytical and clinical performance are key to improve the efficiency of point-of-care diagnostic test development and impact on clinically relevant outcomes. While the ‘road map’ for the steps needed to generate evidence are reasonably well delineated, we provide evidence on the complexity, length and variability of the actual process that many diagnostic technologies undergo.

Carcinoembryonic Antigen Monitoring to Detect Recurrence of Colorectal Cancer: How Should We Interpret the Test Results?

To the Editor: It is routine clinical practice, supported by national guidelines in both North America and Europe, to measure blood carcinoembryonic antigen (CEA)1 to detect recurrence of colorectal cancer during follow-up after primary treatment. Blood CEA is usually measured every 3–6 months, and patients with a CEA concentration above an absolute threshold (5 μg/L according to American Society of Clinical Oncology guidelines) are investigated further by radiological imaging. However, the evidence underpinning both guidelines and routine practice is weak. We recently reported the interim results of the Follow-up After Colorectal Surgery (FACS) trial, a clinical trial comparing different types of posttreatment follow-up in 1200 patients with colorectal cancer (1). This trial confirmed that measuring CEA is an effective way of detecting recurrence at an early stage, thus increasing the number of recurrences that can be treated with curative intent. However, the threshold we applied to define an abnormal CEA concentration (7 μg/L above the patients postoperative concentration at trial entry) was more conservative than current guidelines, so we decided to reevaluate our data to assess retrospectively whether we could have done better by applying a different strategy to interpret the CEA …

The diagnostic accuracy of a single CEA blood test in detecting colorectal cancer recurrence: Results from the FACS trial

Objective To evaluate the diagnostic accuracy of a single CEA (carcinoembryonic antigen) blood test in detecting colorectal cancer recurrence. Background Patients who have undergone curative resection for primary colorectal cancer are typically followed up with scheduled CEA testing for 5 years. Decisions to investigate further (usually by CT imaging) are based on single test results, reflecting international guidelines. Methods A secondary analysis was undertaken of data from the FACS trial (two arms included CEA testing). The composite reference standard applied included CT-CAP imaging, clinical assessment and colonoscopy. Accuracy in detecting recurrence was evaluated in terms of sensitivity, specificity, likelihood ratios, predictive values, time-dependent area under the ROC curves, and operational performance when used prospectively in clinical practice are reported. Results Of 582 patients, 104 (17.9%) developed recurrence during the 5 year follow-up period. Applying the recommended threshold of 5μg/L achieves at best 50.0% sensitivity (95% CI: 40.1–59.9%); in prospective use in clinical practice it would lead to 56 missed recurrences (53.8%; 95% CI: 44.2–64.4%) and 89 false alarms (56.7% of 157 patients referred for investigation). Applying a lower threshold of 2.5μg/L would reduce the number of missed recurrences to 36.5% (95% CI: 26.5–46.5%) but would increase the false alarms to 84.2% (924/1097 referred). Some patients are more prone to false alarms than others—at the 5μg/L threshold, the 89 episodes of unnecessary investigation were clustered in 29 individuals. Conclusion Our results demonstrated very low sensitivity for CEA, bringing to question whether it could ever be used as an independent triage test. It is not feasible to improve the diagnostic performance of a single test result by reducing the recommended action threshold because of the workload and false alarms generated. Current national and international guidelines merit re-evaluation and options to improve performance, such as making clinical decisions on the basis of CEA trend, should be further assessed.

Point-of-care C reactive protein to identify serious infection in acutely ill children presenting to hospital: prospective cohort study.

Objective Acute infection is the most common presentation of children to hospital. A minority of these infections are serious, but early recognition and adequate management are essential. We aimed to develop improved tools to assess children attending ambulatory hospital care, integrating clinical features with point-of-care C reactive protein (CRP). Design Prospective observational diagnostic study. Setting and patients 5517 acutely ill children (1 month–16 years) presenting to 106 paediatricians at six outpatient clinics and six emergency departments in Belgium. Index test Point-of-care CRP alongside vital signs and objective symptoms measurements. Main outcome Hospital admission for >24 hours with a serious infection <5 days after presentation. Results An algorithm was developed consisting of clinical features and CRP. This achieved 97.1% (95% CI 94.3% to 98.7%) sensitivity and 99.6% (95% CI 99.2% to 99.8%) negative predictive value, excluding serious infections in 36.4% of children. It stratifies patients into three groups based on CRP level: high-risk group with CRP >75 mg/L (26.8% risk of infection), intermediate-risk group with CRP 20–75 mg/L and at least one of seven clinical features (8.1%), and lower risk group with CRP <20 mg/L with at least one of the 11 features (3.8%). Children in intermediate-risk or low-risk groups with normal clinical assessment have 0.6% and 0.4% risk of serious infections, respectively. Conclusions Conducting a CRP test may first enable children to be stratified into three risk groups, guiding assessment of clinical features that could be performed by junior doctors or nurses. In one-third of acutely ill children, the algorithm could exclude serious infection. Prospective validation of the algorithm is needed. Clinical trial registration NCT02024282 (post-results).