Lori Wozney

Judging the quality of evidence in reviews of prognostic factor research: adapting the GRADE framework

BackgroundPrognosis research aims to identify factors associated with the course of health conditions. It is often challenging to judge the overall quality of research evidence in systematic reviews about prognosis due to the nature of the primary studies. Standards aimed at improving the quality of primary studies on the prognosis of health conditions have been created, but these standards are often not adequately followed causing confusion about how to judge the evidence.MethodsThis article presents a proposed adaptation of Grading of Recommendations Assessment, Development and Evaluation (GRADE), which was developed to rate the quality of evidence in intervention research, to judge the quality of prognostic evidence.ResultsWe propose modifications to the GRADE framework for use in prognosis research along with illustrative examples from an ongoing systematic review in the pediatric pain literature. We propose six factors that can decrease the quality of evidence (phase of investigation, study limitations, inconsistency, indirectness, imprecision, publication bias) and two factors that can increase it (moderate or large effect size, exposure-response gradient).ConclusionsWe describe criteria for evaluating the potential impact of each of these factors on the quality of evidence when conducting a review including a narrative synthesis or a meta-analysis. These recommendations require further investigation and testing.

A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression

Depression is a common mental health condition for which many mobile apps aim to provide support. This review aims to identify self-help apps available exclusively for people with depression and evaluate those that offer cognitive behavioural therapy (CBT) or behavioural activation (BA). One hundred and seventeen apps have been identified after searching both the scientific literature and the commercial market. 10.26% (n = 12) of these apps identified through our search offer support that seems to be consistent with evidence-based principles of CBT or BA. Taking into account the non existence of effectiveness/efficacy studies, and the low level of adherence to the core ingredients of the CBT/BA models, the utility of these CBT/BA apps are questionable. The usability of reviewed apps is highly variable and they rarely are accompanied by explicit privacy or safety policies. Despite the growing public demand, there is a concerning lack of appropiate CBT or BA apps, especially from a clinical and legal point of view. The application of superior scientific, technological, and legal knowledge is needed to improve the development, testing, and accessibility of apps for people with depression.

Usability, learnability and performance evaluation of Intelligent Research and Intervention Software: A delivery platform for eHealth interventions.

Evaluation of an eHealth platform, Intelligent Research and Intervention Software was undertaken via cross-sectional survey of staff users and application performance monitoring. The platform is used to deliver psychosocial interventions across a range of clinical contexts, project scopes, and delivery modalities (e.g. hybrid telehealth, fully online self-managed, randomized control trials, and clinical service delivery). Intelligent Research and Intervention Software supports persuasive technology elements (e.g. tailoring, reminders, and personalization) as well as staff management tools. Results from the System Usability Scale involving 30 Staff and Administrative users across multiple projects were positive with overall mean score of 70 (“Acceptable”). The mean score for “Usability” sub-scale was 82 and for “Learnability” sub-scale 61. There were no significant differences in perceptions of usability across user groups or levels of experience. Application performance management analytics (e.g. Application Performance Index scores) across two test sites indicate the software platform is robust and reliable when compared to industry standards. Intelligent Research and Intervention Software is successfully operating as a flexible platform for creating, delivering, and evaluating eHealth interventions.

Distance-delivered interventions for PTSD: A systematic review and meta-analysis

This systematic review and meta-analysis evaluated the efficacy of distance-delivered, guided approaches to treatment (e.g., delivered via telephone, Internet, mail, videoconferencing) for clinical and subclinical posttraumatic stress disorder (PTSD). A comprehensive search yielded 19 randomized controlled trials (1491 participants) to be included. Meta-analyses revealed that distance-delivered interventions led to significant within-group improvements in PTSD symptoms at post-treatment (g=0.81, 95% CI 0.65 to 0.97) and 3-6 month follow-up (g=0.78, 95% CI 0.59 to 0.97). Within-group depression and quality of life outcomes showed similar results, with medium post-treatment and follow-up effects. Compared to a waiting list, distance delivery (specifically, Internet treatments) led to superior PTSD outcomes (g=0.68, 95% CI 0.51 to 0.86). Compared to face-to-face interventions, distance delivery (specifically, videoconferencing treatments) did not result in significantly different PTSD outcomes at post-treatment (g=-0.05, 95% CI -0.31 to 0.20) but led to inferior outcomes at 3-6 month follow-up (g=-0.25, 95% CI -0.44 to -0.07). Distance delivery of PTSD treatment is promising, but research is needed to determine its optimal use.

Online Tonsillectomy Resources: Are Parents Getting Consistent and Readable Recommendations?

Objective Parents frequently refer to information on the Internet to confirm or broaden their understanding of surgical procedures and to research postoperative care practices. Our study evaluated the readability, comprehensiveness, and consistency around online recommendations directed at parents of children undergoing tonsillectomy. Study Design A cross-sectional study design was employed. Setting Thirty English-language Internet websites. Subjects and Methods Three validated measures of readability were applied and content analysis was employed to evaluate the comprehensiveness of information in domains of perioperative education. Frequency effect sizes and percentile ranks were calculated to measure dispersion of recommendations across sites. Results The mean readability level of all sites was above a grade 10 level with fewer than half of the sites (n = 14, 47%) scoring at or below the eight-grade level. Provided information was often incomplete with a noted lack of psychosocial support and skills-training recommendations. Content analysis showed 67 unique recommendations spanning the full perioperative period. Most recommendations had low consensus, being reported in 5 or fewer sites (frequency effect size <16%). Conclusion Many online parent-focused resources do not meet readability recommendations, portray incomplete education about perioperative care and expectations, and provide recommendations with low levels of consensus. Up-to-date mapping of the research evidence around recommendations is needed as well as improved efforts to make online information easier to read.

Implementation of eMental Health care: viewpoints from key informants from organizations and agencies with eHealth mandates

BackgroundThe use of technology such as computers, tablets, and smartphones to improve access to and the delivery of mental health care (eMental Health care) is growing worldwide. However, despite the rapidly expanding evidence base demonstrating the efficacy of eMental Health care, its implementation in clinical practice and health care systems remains fragmented. To date, no peer-reviewed, key-informant studies have reported on the perspectives of decision-makers concerned with whether and how to implement eMental Health care.MethodsFrom September to November 2015, we conducted 31 interviews with key informants responsible for leadership, policy, research, and/ or information technology in organizations influential in the adoption of technology for eMental Health care. Deductive and inductive thematic analyses of transcripts were conducted using the Behavior Change Wheel as an organizing framework. Frequency and intensity effect sizes were calculated for emerging themes to further explore patterns within the data.ResultsKey informant responses (n = 31) representing 6 developed countries and multiple organizations showed consensus on common factors impacting implementation: individual and organizational capacities (e.g., computer literacy skills [patients and providers], knowledge gaps about cyber security, limited knowledge of available services); motivational drivers of technology-based care (e.g., extending care, data analytics); and opportunities for health systems to advance eMental Health care implementation (e.g., intersectoral research, rapid testing cycles, sustainable funding). Frequency effect sizes showed strong associations between implementation and credibility, knowledge, workflow, patient empowerment, electronic medical record (EMR) integration, sustained funding and intersectoral networks. Intensity effect sizes showed the highest concentration of statements (>10% of all comments) related to funding, credibility, knowledge gaps, and patient empowerment.ConclusionThis study provides previously unavailable information about key informant perspectives on eMental Health care implementation. The themes that emerged, namely the need to intensify intersectoral research, measure/monitor readiness to implement, define cost-utility benchmarks, raise awareness about available technologies, and test assumptions that ‘proven’ technologies will be easily integrated can inform the design and evaluation of eMental Health care implementation models.

Sociotechnical Human Factors Involved in Remote Online Usability Testing of Two eHealth Interventions

Background Research in the fields of human performance technology and human computer interaction are challenging the traditional macro focus of usability testing arguing for methods that help test moderators assess “use in context” (ie, cognitive skills, usability understood over time) and in authentic “real world” settings. Human factors in these complex test scenarios may impact on the quality of usability results being derived yet there is a lack of research detailing moderator experiences in these test environments. Most comparative research has focused on the impact of the physical environment on results, and rarely on how the sociotechnical elements of the test environment affect moderator and test user performance. Improving our understanding of moderator roles and experiences with conducting “real world” usability testing can lead to improved techniques and strategies Objective To understand moderator experiences of using Web-conferencing software to conduct remote usability testing of 2 eHealth interventions. Methods An exploratory case study approach was used to study 4 moderators’ experiences using Blackboard Collaborate for remote testing sessions of 2 different eHealth interventions. Data collection involved audio-recording iterative cycles of test sessions, collecting summary notes taken by moderators, and conducting 2 90-minute focus groups via teleconference. A direct content analysis with an inductive coding approach was used to explore personal accounts, assess the credibility of data interpretation, and generate consensus on the thematic structure of the results. Results Following the convergence of data from the various sources, 3 major themes were identified: (1) moderators experienced and adapted to unpredictable changes in cognitive load during testing; (2) moderators experienced challenges in creating and sustaining social presence and untangling dialogue; and (3) moderators experienced diverse technical demands, but were able to collaboratively troubleshoot with test users. Conclusions Results highlight important human-computer interactions and human factor qualities that impact usability testing processes. Moderators need an advanced skill and knowledge set to address the social interaction aspects of Web-based usability testing and technical aspects of conferencing software during test sessions. Findings from moderator-focused studies can inform the design of remote testing platforms and real-time usability evaluation processes that place less cognitive burden on moderators and test users.

Guided Internet-Based Parent Training for Challenging Behavior in Children With Fetal Alcohol Spectrum Disorder (Strongest Families FASD): Study Protocol for a Randomized Controlled Trial

Background Fetal alcohol spectrum disorder (FASD) is a term used to encompass the full range of neurobehavioral and cognitive dysfunction that may occur as a consequence of prenatal alcohol exposure. There is relatively little research on intervention strategies that specifically target the behavioral problems of children with FASD. Availability and access to services are barriers to timely and effective care for families. The Strongest Families FASD intervention was recently adapted from the Strongest Families “Parenting the Active Child” program to include FASD-specific content delivered via an Internet-based application in conjunction with 11 telephone coaching sessions. Objective Our objectives are to (1) evaluate the effectiveness of Strongest Families FASD in reducing externalizing problems (primary outcome), internalizing problems, and parent distress (secondary outcomes) in children aged between 4 and 12 years diagnosed with FASD when compared to a control group with access to a static resource Web page; (2) evaluate the effectiveness of Strongest Families FASD in improving social competence (secondary outcome) in school-aged children aged between 6 and 12 diagnosed with FASD when compared with an online psychoeducation control; and (3) explore parental satisfaction with the Strongest Families FASD online parenting program. Methods Parents and caregivers (N=200) of children diagnosed with FASD who have significant behavioral challenges, ages 4-12, are being recruited into a 2-arm randomized trial. The trial is designed to evaluate the effectiveness of the Web-based Strongest Families FASD parenting intervention on child behavior and caregiver distress, compared to a control group receiving access to a static resource Web page (ie, a list of FASD-specific websites, readings, videos, and organizations). Results The primary outcome will be externalizing problems measured by the Child Behavior Checklist (CBCL). Secondary outcomes include (1) internalizing problems and (2) social competence, both measured by the CBCL; and (3) parental distress measured by the Depression Anxiety Stress Scale-21. The Client Satisfaction Questionnaire-8 (CSQ-8) and the Satisfaction Survey are completed by the intervention group at the end of session 11. Results will be reported using the standards set out in the Consolidated Standards of Reporting Trials (CONSORT) Statement. Conclusions It is hypothesized that the Strongest Families FASD intervention group will improve child behavior and parental distress. Caregiver satisfaction is anticipated to be positive. Advancing evidence on the effectiveness and acceptance of distance services can inform policy and adoption of eHealth programs. ClinicalTrial ClinicalTrials.gov NCT02210455; https://clinicaltrials.gov/ct2/show/NCT02210455 (Archived by WebCite at http://www.webcitation.org/6bbW5BSsT)

eMental Healthcare Technologies for Anxiety and Depression in Childhood and Adolescence: Systematic Review of Studies Reporting Implementation Outcomes

Background Anxiety disorders and depression are frequent conditions in childhood and adolescence. eMental healthcare technologies may improve access to services, but their uptake within health systems is limited. Objective The objective of this review was to examine and describe how the implementation of eMental healthcare technologies for anxiety disorders and depression in children and adolescents has been studied. Methods We conducted a search of 5 electronic databases and gray literature. Eligible studies were those that assessed an eMental healthcare technology for treating or preventing anxiety or depression, included children or adolescents (<18 years), or their parents or healthcare providers and reported findings on technology implementation. The methodological quality of studies was evaluated using the Mixed Methods Appraisal Tool. Outcomes of interest were based on 8 implementation outcomes: acceptability (satisfaction with a technology), adoption (technology uptake and utilization), appropriateness (“fitness for purpose”), cost (financial impact of technology implementation), feasibility (extent to which a technology was successfully used), fidelity (implementation as intended), penetration (“spread” or “reach” of the technology), and sustainability (maintenance or integration of a technology within a healthcare service). For extracted implementation outcome data, we coded favorable ratings on measurement scales as “positive results” and unfavorable ratings on measurement scales as “negative results.” Those studies that reported both positive and negative findings were coded as having “mixed results.” Results A total of 46 studies met the inclusion criteria, the majority of which were rated as very good to excellent in methodological quality. These studies investigated eMental healthcare technologies for anxiety (n=23), depression (n=18), or both anxiety and depression (n=5). Studies of technologies for anxiety evaluated the following: (1) acceptability (78%) reported high levels of satisfaction, (2) adoption (43%) commonly reported positive results, and (3) feasibility (43%) reported mixed results. Studies of technologies for depression evaluated the following: (1) appropriateness (56%) reported moderate helpfulness and (2) acceptability (50%) described a mix of both positive and negative findings. Studies of technologies designed to aid anxiety and depression commonly reported mixed experiences with acceptability and adoption and positive findings for appropriateness of the technologies for treatment. Across all studies, cost, fidelity, and penetration and sustainability were the least measured implementation outcomes. Conclusions Acceptability of eMental healthcare technology is high among users and is the most commonly investigated implementation outcome. Perceptions of the appropriateness and adoption of eMental healthcare technology were varied. Implementation research that identifies, evaluates, and reports on costs, sustainability, and fidelity to clinical guidelines is crucial for making high-quality eMental healthcare available to children and adolescents.

Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people : A consensus statement with recommendations from the #iCBTLorentz Workshop Group

Initial internet-based cognitive behavioral therapy (iCBT) programs for anxiety disorders in children and young people (CYP) have been developed and evaluated, however these have not yet been widely adopted in routine practice. The lack of guidance and formalized approaches to the development and dissemination of iCBT has arguably contributed to the difficulty in developing iCBT that is scalable and sustainable beyond academic evaluation and that can ultimately be adopted by healthcare providers. This paper presents a consensus statement and recommendations from a workshop of international experts in CYP anxiety and iCBT (#iCBTLorentz Workshop Group) on the development, evaluation, engagement and dissemination of iCBT for anxiety in CYP.