Ellen M. Werner

Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services

Many of the Institutes, Agencies and Centers that make up the US Department of Health and Human Services (DHHS) have recognized the need for better instrumentation in health outcomes research, and provide support, both internally and externally, for research utilizing advances in measurement theory and computer technology (informatics). In this paper, representatives from several DHHS agencies and institutes will discuss their need for better instruments within their discipline and describe current or future initiatives for exploring the benefits of these technologies. Together, the perspectives underscore the importance of developing valid, precise, and efficient measures to capture the full burden of disease and treatment on patients. Initiatives, like the Patient-Reported Outcomes Measurement Information System (PROMIS) to create health-related quality of life item banks, represent a trans-DHHS effort to develop a standard set of measures for informing decision making in clinical research, practice, and health policy.

Patient reports of health outcome for adults living with sickle cell disease: development and testing of the ASCQ-Me item banks.

BackgroundProviders and patients have called for improved understanding of the health care requirements of adults with sickle cell disease (SCD) and have identified the need for a systematic, reliable and valid method to document the patient-reported outcomes (PRO) of adult SCD care. To address this need, the Adult Sickle Cell Quality of Life Measurement System (ASCQ-Me) was designed to complement the Patient Reported Outcome Measurement Information System (PROMIS®). Here we describe methods and results of the psychometric evaluation of ASCQ-Me item banks (IBs).MethodsAt seven geographically-disbursed clinics within the US, 556 patients responded to questions generated to assess cognitive, emotional, physical and social impacts of SCD. We evaluated the construct validity of the hypothesized domains using exploratory factor analysis (EFA), parallel analysis (PA), and bi-factor analysis (Item Response Theory Graded Response Model, IRT-GRM). We used IRT-GRM and the Wald method to identify bias in responses across gender and age. We used IRT and Cronbach’s alpha coefficient to evaluate the reliability of the IBs and then tested the ability of summary scores based on IRT calibrations to discriminate among tertiles of respondents defined by SCD severity.ResultsOf the original 140 questions tested, we eliminated 48 that either did not form clean factors or provided biased measurement across subgroups defined by age and gender. Via EFA and PA, we identified three subfactors within physical impact: sleep, pain and stiffness impacts. Analysis of the resulting six item sets (sleep, pain, stiffness, cognitive, emotional and social impacts of SCD) supported their essential unidimensionality. With the exception of the cognitive impact IB, these item sets also were highly reliable across a broad range of values and highly significantly related to SCD disease severity.ConclusionASCQ-Me pain, sleep, stiffness, emotional and social SCD impact IBs demonstrated exceptional measurement properties using modern and classical psychometric methods of evaluation. Further development of the cognitive impact IB is required to improve its sensitivity to differences in SCD disease severity. Future research will evaluate the sensitivity of the ASCQ-Me IBs to change in SCD disease severity over time due to health interventions.

Improving the value of clinical research through the use of Common Data Elements.

The use of Common Data Elements can facilitate cross-study comparisons, data aggregation, and meta-analyses; simplify training and operations; improve overall efficiency; promote interoperability between different systems; and improve the quality of data collection. A Common Data Element is a combination of a precisely defined question (variable) paired with a specified set of responses to the question that is common to multiple datasets or used across different studies. Common Data Elements, especially when they conform to accepted standards, are identified by research communities from variable sets currently in use or are newly developed to address a designated data need. There are no formal international specifications governing the construction or use of Common Data Elements. Consequently, Common Data Elements tend to be made available by research communities on an empiric basis. Some limitations of Common Data Elements are that there may still be differences across studies in the interpretation and implementation of the Common Data Elements, variable validity in different populations, and inhibition by some existing research practices and the use of legacy data systems. Current National Institutes of Health efforts to support Common Data Element use are linked to the strengthening of National Institutes of Health Data Sharing policies and the investments in data repositories. Initiatives include cross-domain and domain-specific resources, construction of a Common Data Element Portal, and establishment of trans-National Institutes of Health working groups to address technical and implementation topics. The National Institutes of Health is seeking to lower the barriers to Common Data Element use through greater awareness and encourage the culture change necessary for their uptake and use. As National Institutes of Health, other agencies, professional societies, patient registries, and advocacy groups continue efforts to develop and promote the responsible use of Common Data Elements, particularly if linked to accepted data standards and terminologies, continued engagement with and feedback from the research community will remain important.

State-based surveillance for selected hemoglobinopathies

Purpose:The lack of an ongoing surveillance system for hemoglobinopathies in the United States impedes the ability of public health organizations to identify individuals with these conditions, monitor their health-care utilization and clinical outcomes, and understand the effect these conditions have on the health-care system. This article describes the results of a pilot program that supported the development of the infrastructure and data collection methods for a state-based surveillance system for selected hemoglobinopathies.Methods:The system was designed to identify and gather information on all people living with a hemoglobinopathy diagnosis (sickle cell diseases or thalassemias) in the participating states during 2004–2008. Novel, three-level case definitions were developed, and multiple data sets were used to collect information.Results:In total, 31,144 individuals who had a hemoglobinopathy diagnosis during the study period were identified in California; 39,633 in Florida; 20,815 in Georgia; 12,680 in Michigan; 34,853 in New York, and 8,696 in North Carolina.Conclusion:This approach provides a possible model for the development of state-based hemoglobinopathy surveillance systems.Genet Med 17 2, 125–130.

Mortality of New York children with sickle cell disease identified through newborn screening

Purpose:Long-term follow-up of newborn screening for conditions such as sickle cell disease can be conducted using linkages to population-based data. We sought to estimate childhood sickle cell disease mortality and risk factors among a statewide birth cohort with sickle cell disease identified through newborn screening.Methods:Children with sickle cell disease identified by newborn screening and born to New York residents in 2000–2008 were matched to birth and death certificates. Mortality rates were calculated (using numbers of deaths and observed person-years at risk) and compared with mortality rates for all New York children by maternal race/ethnicity. Stratified analyses were conducted to examine associations between selected factors and mortality.Results:Among 1,911 infants with sickle cell disease matched to birth certificates, 21 deaths were identified. All-cause mortality following diagnosis was 3.8 per 1,000 person-years in the first 2 years of life and 1.0 per 1,000 person-years at ages 2–9 years. The mortality rate was significantly lower among children of foreign-born mothers and was significantly higher among preterm infants with low birth weight. The mortality rates were not significantly higher for infants after 28 days with sickle cell disease than for all New York births, but they were 2.7–8.4 times higher for children 1 through 9 years old with homozygous sickle cell disease than for those of all non-Hispanic black or Hispanic children born to New York residents.Conclusion:Estimated mortality risk in children with homozygous sickle cell disease remains elevated even after adjustment for maternal race/ethnicity. These results provide evidence regarding the current burden of child mortality among children with sickle cell disease despite newborn screening.Genet Med 17 6, 452–459.

Quality of care in sickle cell disease: Cross-sectional study and development of a measure for adults reporting on ambulatory and emergency department care.

AbstractDocumented deficiencies in adult sickle cell disease (SCD) care include poor access to knowledgeable providers and inadequate treatment in emergency departments (EDs).The aim of this study was to create patient-reported outcome measures of the quality of ambulatory and ED care for adults with SCD.We developed and pilot tested SCD quality of care questions consistent with Consumer Assessments of Healthcare Providers and Systems surveys. We applied psychometric methods to develop scores and evaluate reliability and validity.The participants of this study were adults with SCD (n = 556)—63% aged 18 to 34 years; 64% female; 64% SCD-SS—at 7 US sites.The measure used was Adult Sickle Cell Quality of Life Measurement information system Quality of Care survey.Most participants (90%) reported at least 1 severe pain episode (pain intensity 7.8 ± 2.3, 0–10 scale) in the past year. Most (81%) chose to manage pain at home rather than the ED, citing negative ED experiences (83%). Using factor analysis, we identified Access, Provider Interaction, and ED Care composites with reliable scores (Cronbach &agr; 0.70–0.83) and construct validity (r = 0.32–0.83 correlations with global care ratings). Compared to general adult Consumer Assessments of Healthcare Providers and Systems scores, adults with SCD had worse care, adjusted for age, education, and general health.Results were consistent with other research reflecting deficiencies in ED care for adults with SCD. The Adult Sickle Cell Quality of Life Measurement Quality of Care measure is a useful self-report measure for documenting and tracking disparities in quality of SCD care.

News from the NIH: Person-centered outcomes measurement: NIH-supported measurement systems to evaluate self-assessed health, functional performance, and symptomatic toxicity

There is rapidly growing interest in the capture of person-centered outcomes in clinical and population-based research and in healthcare delivery settings. Stakeholders (e.g., patients, clinicians, payers, regulators, researchers) increasingly agree that person-centered outcome measurement can accelerate the development of new knowledge, improve the efficiency and quality of care, and may also contribute to clinician or health system performance metrics and regulatory review of new therapies [1–3]. These outcomes may be incorporated into both observational studies and clinical trials, and provide salient endpoints in trials of preventive or disease-modifying treatments, as well as behavioral or psychosocial interventions. Over the past decade, the National Institutes of Health (NIH) has invested in the development and evaluation of several measurement systems that are now available for research and clinical use. These include the Patient Reported Outcomes Measurement Information System® (PROMIS®) [4], the NIH Toolbox for Assessment of Neurological and Behavioral Function (NIH Toolbox®) [5], the Quality of Life Outcomes in Neurological Disorders (Neuro-QoL) [6], Adult Sickle Cell Quality of Life Measurement Information System (ASCQ-Me) [7], and the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) [8]. In this paper, we (i) describe each system; (ii) highlight considerations in the design and interpretation of studies that employ one or more of these systems; and (iii) summarize future directions for continued implementation of these systems in clinical practice, population-based research, observational studies, and clinical trials.

Standard measures for sickle cell disease research: the PhenX Toolkit sickle cell disease collections

Standard measures and common data elements for sickle cell disease (SCD) will improve the data quality and comparability necessary for cross-study analyses and the development of guidelines that support effective treatments and interventions. In 2014, the National Institutes of Health, National Heart, Lung, and Blood Institute (NHLBI) funded an Administrative Supplement to the PhenX Toolkit (consensus measures for Phenotypes and eXposures; https://www.phenxtoolkit.org/) to identify common measures to promote data comparability across SCD research. An 11-member Sickle Cell Disease Research and Scientific Panel provided guidance to the project, establishing a core collection of SCD-related measures and defining the scope of 2 specialty collections: (1) cardiovascular, pulmonary, and renal complications, and (2) neurology, quality-of-life, and health services. For each specialty collection, a working group of SCD experts selected high-priority measures using a consensus process that included scientific community input. The SCD measures were released into the Toolkit in August 2015. The 25 measures included in the core collection are recommended for use by all NHLBI-funded investigators performing human-subject SCD research. The 10 neurology, quality-of-life, and health services measures and 14 cardiovascular, pulmonary, and renal measures are recommended for use within these specialized research areas. For SCD and other researchers, PhenX measures will promote collaborations with clinicians and patients, facilitate cross-study analysis, accelerate translational research, and lead to greater understanding of SCD phenotypes and epigenetics. For clinicians, using PhenX measures will help elucidate the etiology, progression, and treatment of SCD, leading to improved patient care and quality of life.

Enhancing diversity in the hematology biomedical research workforce: A mentoring program to improve the odds of career success for early stage investigators

The necessity for greater racial and ethnic diversity in the US biomedical research workforce is evident, however many challenges must be overcome to achieve this formidable goal. Historically, underrepresented minority (URM) groups are the most rapidly growing segment of the US population and there is an urgent need to ensure that scientific talent among these groups is recognized, mentored and actively supported. For example, in 2010, Hispanics/Latinos, Blacks/African Americans, and American Indians/Alaskan Natives represented 29.8% of the US population, yet only 4.8% of National Institutes of Health (NIH) research project grants (RPG) were awarded to URM principal investigators. A study by Ginther et al. revealed that PhD-trained African American applicants are 13.2% less likely than White applicants to be awarded RPG. While the NIH is the largest research funding agency in the world, it has not achieved proportional representation of URM investigators in the US biomedical research workforce. Likewise, the imperative to increase diversity is justified by inequities in access to health care and health outcomes. Improving these statistics will require interventions such as the introduction of innovative training models involving dedicated mentoring by established NIH-funded investigators, which are tested by rigorous evaluations. Analysis of the results from these training models will demonstrate the extent to which current interventions increase representation of URM groups in the biomedical research enterprise. Recently, the NIH established the National Research Mentoring Network (NRMN) to improve the success of URM investigators with the goal of diversifying our nation’s biomedical research workforce. There is a paucity of published data demonstrating that structured research mentoring programs promote grant funding, and professional development of early stage investigators (ESI). To provide expanded mentoring support for URM investigators, in 2006 the National Heart, Lung, and Blood Institute (NHLBI) established the Summer Institute Program to Increase Diversity (SIPID), and subsequently the PRIDE (Program to Increase Diversity Among Individuals Engaged in HealthRelated Research) Program. The scope of the PRIDE Program consists of seven academic sites, each focused on a specific research area. The objective of all programs is to provide intense research and career development mentorship coordinated through a central PRIDE Coordination Core (PCC) described recently. The PRIDE Program at Augusta University is focused on “Functional and Translational Genomics of Blood Disorders” (FTG-PRIDE), and has been funded by NHLBI since 2006. During each funding period, 3 cohorts of 6–10 mentees were recruited after the FTG-PRIDE Admissions Committee reviewed and ranked applications. Top candidates were interviewed to ensure the program requirements were fulfilled and a suitable mentor-mentee dyad could be established. While many approaches can be taken to address the inequity of URM representation in the US biomedical research workforce, the objectives of the PRIDE Program has principally focused on training ESI in grant writing skills to achieve extramural funding and expanded research-related technical skills. To evaluate program effectiveness, the PCC developed and administered a series of evaluation questionnaires during the 2-year training period and for 8 years after training completion. Mentee demographics and career development-related outcomes have been collected since 2006. To assess research self-efficacy, a 19item Clinical Research Appraisal Inventory (CRAI-19) previously validated in the PRIDE Program, is completed annually. To accomplish these objectives, the FTG-PRIDE Program leadership organizes two consecutive Summer Institutes at Augusta University, each lasting 2 to 3 weeks. In addition, a mid-year face-to-face meeting is attended by each mentee with their primary mentor to review research progress, and to update skills and adopt new technologies. Mentees are also required to attend the National PRIDE Meeting convened annually in Bethesda, MD. The purpose of this meeting is to provide opportunities for trainees to interact with NHLBI program staff, present their research to other trainees, mentors and teaching faculty from all PRIDE programs, and establish research collaborations. During the period 2006–2017, we trained 76 mentees in the FTGPRIDE Program (Supporting Information Table S1) under Institutional Review Board approval and informed consent for data collection by the PCC. Since the last cohort of participants in PRIDE 2 has not completed its second year of training, the data presented here are limited to the 48 mentees trained in SIPID and PRIDE 1. Of this group, 6 mentees were excluded from the analysis due to withdrawal from the program, matriculation into a second PRIDE Program, or noncompliance with program evaluations. As a result, the outcomes of 42 evaluable mentees are described in this report. The design of the 2-year FTG-PRIDE Program is summarized in Supporting Information Figure S1. The first Summer Institute commences with a Welcome Ceremony attended by mentees, mentors, teaching faculty, and program leadership along with high-level administrators from Augusta University. After orientation to review program requirements, one-on-one mentee/mentor meetings are held to initiate the