Karon F. Cook

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.

Background:The National Institutes of Health (NIH) Patient-Reported Outcomes Measurement Information System (PROMIS) Roadmap initiative (www.nihpromis.org) is a 5-year cooperative group program of research designed to develop, validate, and standardize item banks to measure patient-reported outcomes (PROs) relevant across common medical conditions. In this article, we will summarize the organization and scientific activity of the PROMIS network during its first 2 years. Design:The network consists of 6 primary research sites (PRSs), a statistical coordinating center (SCC), and NIH research scientists. Governed by a steering committee, the network is organized into functional subcommittees and working groups. In the first year, we created an item library and activated 3 interacting protocols: Domain Mapping, Archival Data Analysis, and Qualitative Item Review (QIR). In the second year, we developed and initiated testing of item banks covering 5 broad domains of self-reported health. Results:The domain mapping process is built on the World Health Organization (WHO) framework of physical, mental, and social health. From this framework, pain, fatigue, emotional distress, physical functioning, social role participation, and global health perceptions were selected for the first wave of testing. Item response theory (IRT)-based analysis of 11 large datasets supplemented and informed item-level qualitative review of nearly 7000 items from available PRO measures in the item library. Items were selected for rewriting or creation with further detailed review before the first round of testing in the general population and target patient populations. Conclusions:The NIH PROMIS network derived a consensus-based framework for self-reported health, systematically reviewed available instruments and datasets that address the initial PROMIS domains. Qualitative item research led to the first wave of network testing which began in the second year.

Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS)

Background:The construction and evaluation of item banks to measure unidimensional constructs of health-related quality of life (HRQOL) is a fundamental objective of the Patient-Reported Outcomes Measurement Information System (PROMIS) project. Objectives:Item banks will be used as the foundation for developing short-form instruments and enabling computerized adaptive testing. The PROMIS Steering Committee selected 5 HRQOL domains for initial focus: physical functioning, fatigue, pain, emotional distress, and social role participation. This report provides an overview of the methods used in the PROMIS item analyses and proposed calibration of item banks. Analyses:Analyses include evaluation of data quality (eg, logic and range checking, spread of response distribution within an item), descriptive statistics (eg, frequencies, means), item response theory model assumptions (unidimensionality, local independence, monotonicity), model fit, differential item functioning, and item calibration for banking. Recommendations:Summarized are key analytic issues; recommendations are provided for future evaluations of item banks in HRQOL assessment.

Development of a PROMIS item bank to measure pain interference

&NA; This paper describes the psychometric properties of the PROMIS‐pain interference (PROMIS‐PI) bank. An initial candidate item pool (n = 644) was developed and evaluated based on the review of existing instruments, interviews with patients, and consultation with pain experts. From this pool, a candidate item bank of 56 items was selected and responses to the items were collected from large community and clinical samples. A total of 14,848 participants responded to all or a subset of candidate items. The responses were calibrated using an item response theory (IRT) model. A final 41‐item bank was evaluated with respect to IRT assumptions, model fit, differential item function (DIF), precision, and construct and concurrent validity. Items of the revised bank had good fit to the IRT model (CFI and NNFI/TLI ranged from 0.974 to 0.997), and the data were strongly unidimensional (e.g., ratio of first and second eigenvalue = 35). Nine items exhibited statistically significant DIF. However, adjusting for DIF had little practical impact on score estimates and the items were retained without modifying scoring. Scores provided substantial information across levels of pain; for scores in the T‐score range 50–80, the reliability was equivalent to 0.96–0.99. Patterns of correlations with other health outcomes supported the construct validity of the item bank. The scores discriminated among persons with different numbers of chronic conditions, disabling conditions, levels of self‐reported health, and pain intensity (p < 0.0001). The results indicated that the PROMIS‐PI items constitute a psychometrically sound bank. Computerized adaptive testing and short forms are available.

What functional activities are important to patients with knee replacements

There is interest in quantifying the patient’s function and mobility after joint replacement. The current study identified activities important to patients having total knee replacement and the prevalence of limitations to participation in these activities. A Total Knee Function Questionnaire consisting of 55 questions addressing the patient’s participation in various activities was developed, validated, and mailed to 367 patients at least 1 year after knee replacement. Patients were asked the frequency with which they did each activity, the activity’s importance to them, and the extent to which their participation was limited by their knee replacement. The questionnaire was returned by 176 patients, 40% men and 60% women, with an average age of 70.5 years. The most prevalent activities were stretching exercises (73%), leg strengthening exercises (70%), kneeling (58%), and gardening (57%). The activities most important to the patients were stretching exercises (56%), kneeling (52%), and gardening (50%); those most difficult were squatting (75%), kneeling (72%), and gardening (54%). The current study showed a high correlation between the importance of activities and frequency of patient participation confirming that knee replacement successfully restores a significant degree of function. However, after knee replacement, improvements in knee function still are needed to allow patients to do all activities that they consider important.

Having a fit: impact of number of items and distribution of data on traditional criteria for assessing IRT's unidimensionality assumption.

PurposeConfirmatory factor analysis fit criteria typically are used to evaluate the unidimensionality of item banks. This study explored the degree to which the values of these statistics are affected by two characteristics of item banks developed to measure health outcomes: large numbers of items and nonnormal data.MethodsAnalyses were conducted on simulated and observed data. Observed data were responses to the Patient-Reported Outcome Measurement Information System (PROMIS) Pain Impact Item Bank. Simulated data fit the graded response model and conformed to a normal distribution or mirrored the distribution of the observed data. Confirmatory factor analyses (CFA), parallel analysis, and bifactor analysis were conducted.ResultsCFA fit values were found to be sensitive to data distribution and number of items. In some instances impact of distribution and item number was quite large.ConclusionsWe concluded that using traditional cutoffs and standards for CFA fit statistics is not recommended for establishing unidimensionality of item banks. An investigative approach is favored over reliance on published criteria. We found bifactor analysis to be appealing in this regard because it allows evaluation of the relative impact of secondary dimensions. In addition to these methodological conclusions, we judged the items of the PROMIS Pain Impact bank to be sufficiently unidimensional for item response theory (IRT) modeling.

NIH Toolbox for Assessment of Neurological and Behavioral Function

At present, there are many studies that collect information on aspects of neurologic and behavioral function (cognition, sensation, movement, emotion), but with little uniformity among the measures used to capture these constructs. Further, available measures are generally expensive, normed on homogenous nondiverse populations, not easily administered, do not cover the lifespan (or have easily linked pediatric and adult counterparts for the purposes of longitudinal comparison), and not based on the current thinking in the neuroscience community. There is also a paucity of measurement tools to gauge normal children in the motor and sensation domain areas, and many of these measures rely heavily on proxy reporting. Investigators have expressed the need for brief assessment tools that could address these issues and be used as a form of “common currency” across diverse study designs and populations. This ability to assess functionality along a common metric and “crosswalk” across measures is essential to the process of being able to pool data, which is often necessary when a large and diverse sample is needed. When individual studies employ unique assessment batteries, comparisons between studies and combining data from multiple studies can be problematic. The contract for the NIH Toolbox for the Assessment of Neurological and Behavioral Function (www.nihtoolbox.org) was initiated by the NIH Blueprint for Neuroscience Research (www.neuroscienceblueprint.nih.gov) to develop a set of state-of-the-art measurement tools to enhance collection of data in large cohort studies and to advance the biomedical research enterprise.

Development and Psychometric Analysis of the PROMIS Pain Behavior Item Bank

ABSTRACT The measurement of pain behavior is a key component of the assessment of persons with chronic pain; however, few self‐reported pain behavior instruments have been developed. We developed a pain behavior item bank as part of the Patient‐Reported Outcome Measurement Information System (PROMIS). For the Wave I testing, because of the large number of PROMIS items, a complex sampling approach was used where participants were randomly assigned to either respond to two full‐item banks or to multiple 7‐item blocks of items. A web‐based survey was designed and completed by 15,528 members of the general population and 967 individuals with different types of chronic pain. Item response theory (IRT) analysis models were used to evaluate item characteristics and to scale both items and individuals on the pain behavior domain. The pain behavior item bank demonstrated good fit to a unidimensional model (Comparative Fit Index = 0.94). Several iterations of IRT analyses resulted in a final 39‐item pain behavior bank, and different IRT models were fit to the total sample and to those participants who experienced some pain. The results indicated that these items demonstrated good coverage of the pain behavior construct. Pain behavior scores were strongly related to pain intensity and moderately related to self‐reported general health status. Mean pain behavior scores varied significantly by groups based on pain severity and general health status. The PROMIS pain behavior item bank can be used to develop static short‐form and dynamic measures of pain behavior for clinical studies.

Comparing CESD-10, PHQ-9, and PROMIS depression instruments in individuals with multiple sclerosis.

PURPOSE This study evaluated psychometric properties of the Patient Health Questionnaire-9 (PHQ-9), the Center for Epidemiological Studies Depression Scale-10 (CESD-10), and the 8-item PROMIS Depression Short Form (PROMIS-D-8; 8b short form) in a sample of individuals living with multiple sclerosis (MS). RESEARCH METHOD Data were collected by a self-reported mailed survey of a community sample of people living with MS (n = 455). Factor structure, interitem reliability, convergent/discriminant validity and assignment to categories of depression severity were examined. RESULTS A 1-factor, confirmatory factor analytic model had adequate fit for all instruments. Scores on the depression scales were more highly correlated with one another than with scores on measures of pain, sleep disturbance, and fatigue. The CESD-10 categorized about 37% of participants as having significant depressive symptoms. At least moderate depression was indicated for 24% of participants by PHQ-9. PROMIS-D-8 identified 19% of participants as having at least moderate depressive symptoms and about 7% having at least moderately severe depression. None of the examined scales had ceiling effects, but the PROMIS-D-8 had a floor effect. CONCLUSIONS Overall, scores on all 3 scales demonstrated essential unidimensionality and had acceptable interitem reliability and convergent/discriminant validity. Researchers and clinicians can choose any of these scales to measure depressive symptoms in individuals living with MS. The PHQ-9 offers validated cutoff scores for diagnosing clinical depression. The PROMIS-D-8 measure minimizes the impact of somatic features on the assessment of depression and allows for flexible administration, including Computerize Adaptive Testing (CAT). The CESD-10 measures 2 aspects of depression, depressed mood and lack of positive affect, while still providing an interpretable total score.

A comparison of three sets of criteria for determining the presence of differential item functioning using ordinal logistic regression

BackgroundSeveral techniques have been developed to detect differential item functioning (DIF), including ordinal logistic regression (OLR). This study compared different criteria for determining whether items have DIF using OLR.ObjectivesTo compare and contrast findings from three different sets of criteria for detecting DIF using OLR. General distress and physical functioning items were evaluated for DIF related to five covariates: age, marital status, gender, race, and Hispanic origin.Research designCross-sectional study.Subjects1,714 patients with cancer or HIV/AIDS.MeasuresA total of 23 items addressing physical functioning and 15 items addressing general distress were selected from a pool of 154 items from four different health-related quality of life questionnaires.ResultsThe three sets of criteria produced qualitatively and quantitatively different results. Criteria based on statistical significance alone detected DIF in almost all the items, while alternative criteria based on magnitude detected DIF in far fewer items. Accounting for DIF by using demographic-group specific item parameters had negligible effects on individual scores, except for race.ConclusionsSpecific criteria chosen to determine whether items have DIF have an impact on the findings. Criteria based entirely on statistical significance may detect small differences that are clinically negligible.

Patient-reported outcomes measurement information system (PROMIS) domain names and definitions revisions: further evaluation of content validity in IRT-derived item banks.

PurposeContent validity of patient-reported outcomes (PROs) is evaluated primarily during item development, but subsequent psychometric analyses, particularly for item response theory (IRT)-derived scales, often result in considerable item pruning and potential loss of content. After selecting items for the PROMIS banks based on psychometric and content considerations, we invited external content expert reviews of the degree to which the initial domain names and definitions represented the calibrated item bank content.MethodsA minimum of four content experts reviewed each item bank and recommended a domain name and definition based on item content. Domain names and definitions then were revealed to the experts who rated how well these names and definitions fit the bank content and provided recommendations for definition revisions.ResultsThese reviews indicated that the PROMIS domain names and definitions remained generally representative of bank content following item pruning, but modifications to two domain names and minor to moderate revisions of all domain definitions were needed to optimize fit with the item bank content.ConclusionsThis reevaluation of domain names and definitions following psychometric item pruning, although not previously documented in the literature, appears to be an important procedure for refining conceptual frameworks and further supporting content validity.