Chih Hung Chang

Developing tailored instruments: item banking and computerized adaptive assessment

Item banks and Computerized Adaptive Testing (CAT) have the potential to greatly improve the assessment of health outcomes. This review describes the unique features of item banks and CAT and discusses how to develop item banks. In CAT, a computer selects the items from an item bank that are most relevant for and informative about the particular respondent; thus optimizing test relevance and precision. Item response theory (IRT) provides the foundation for selecting the items that are most informative for the particular respondent and for scoring responses on a common metric. The development of an item bank is a multi-stage process that requires a clear definition of the construct to be measured, good items, a careful psychometric analysis of the items, and a clear specification of the final CAT. The psychometric analysis needs to evaluate the assumptions of the IRT model such as unidimensionality and local independence; that the items function the same way in different subgroups of the population; and that there is an adequate fit between the data and the chosen item response models. Also, interpretation guidelines need to be established to help the clinical application of the assessment. Although medical research can draw upon expertise from educational testing in the development of item banks and CAT, the medical field also encounters unique opportunities and challenges.

Item Response Theory and its Applications to Patient-Reported Outcomes Measurement

This article provides an overview of item response theory (IRT) models and how they can be appropriately applied to patient-reported outcomes (PROs) measurement. Specifically, the following topics are discussed: (a) basics of IRT, (b) types of IRT models, (c) how IRT models have been applied to date, and (d) new directions in applying IRT to PRO measurements.

What are the most important symptom targets when treating advanced cancer? A survey of providers in the National Comprehensive Cancer Network (NCCN).

We derived a set of brief, clinically relevant symptom indices for assessing symptomatic response to chemotherapy for advanced bladder, brain, breast, colorectal, head and neck, hepatobiliary/pancreas, lung, ovarian, and prostate cancers. Questions were extracted from a multidimensional cancer quality of life (QOL) measurement system, the Functional Assessment of Cancer Therapy (FACT). Surveys of disease-related symptoms were presented to expert physicians and nurses at 17 National Comprehensive Cancer Network (NCCN) member institutions. In a two-step procedure, each expert narrowed the list to no more than five of the very most important to attend to when assessing the value of drug treatment for advanced disease. Symptoms endorsed at a frequency greater than chance probability were retained for the nine symptom indices. The resulting NCCN/FACT symptom indices are comprised of 6–15 items, depending on disease. Fatigue, pain, nausea, weight loss, worry about worsening condition, and contentment with current QOL were consistently selected by experts as priority symptoms across tumor sites. These nine tumor-specific symptom indices indicate the most important clinician-rated targets of chemotherapy for many advanced cancers. These results await validation in patient populations and examination of the extent to which changes in symptomatology translate into meaningful improvement to the patient.

Response to Hays et al and McHorney and Cohen: A Discussion of Item Response Theory and Its Applications in Health Status Assessment

We read with great interest the 2 articles on the use of item response theory (IRT) measurement models in the arena of health status assessment. For reasons that are more accidental than logical, classic approaches have dominated health status assessment until very recently. Now, IRT is entering the field. This is accompanied by enthusiasm for the prospect of deriving better definitions of underlying constructs, new hope for the prospect of individual diagnosis, and an opportunity to turn our attention away from static tests and scales to items and the incremental information they provide. The careful use of IRT-based assessment, either by preselecting an item list that applies to the population being studied, or by computerized adaptive testing (CAT) can make assessment briefer, more flexible, more efficient, and if desired, more precise than conventional approaches.

Assessment of Patient-Reported Clinical Outcome in Pancreatic and Other Hepatobiliary Cancers: The FACT Hepatobiliary Symptom Index

This studys aim was to develop and validate a symptom index derived from the Functional Assessment of Cancer Therapy-Hepatobiliary, a questionnaire measuring general and hepatobiliary disease specific aspects of quality of life. The item pool was narrowed to 26 questions that assess symptoms and function. Each of 95 hepatobiliary cancer experts narrowed the list to 5 of the most important to attend to when treating advanced hepatobiliary disease. Eight symptoms were endorsed by more than 20% of the experts (3 pain, 2 fatigue, nausea, weight loss, jaundice) and were named the FACT-Hepatobiliary Symptom Index-8 (FHSI-8). Among 51 hepatobiliary cancer patients, the FHSI-8 showed good internal consistency (0.79), test-retest reliability (r = 0.86), strong association with mood (r = -0.56), and patient differentiation by ECOG Performance Status Rating ( P < 0.0001) and treatment status ( P = 0.057). Symptom scaling in diseases such as hepatobiliary cancer is feasible and may provide an efficient, clinically-relevant endpoint for following groups over time.

Applying item response theory to enhance health outcomes assessment

In June 2004, the National Cancer Institute and Drug Information Association co-sponsored a conference focused on developing patient-reported outcome (PRO) questionnaires, analyzing data collected from patients, and utilizing findings to enhance decision making for treatment and health policy, ‘‘Advances in Health Outcomes Measurement: Exploring the Current State and the Future of Item Response Theory (IRT), Item Banks, and ComputerAdaptive Testing.’’ Invited speakers were internationally recognized experts in psychometrics and health-outcomes measurement. Their rich experiences are reflected in the articles in this supplement that focus on describing the methods of modern measurement theory and its applications, as well as challenges for improving PRO measures such as pain, fatigue, physical function, and emotional distress. Many health outcomes instruments used in clinical research and practice were developed based on input from experienced researchers, clinicians, and patients. However, concerns have been raised that these instruments are cumbersome for respondents, burdensome for clinical care sites, not applicable over the continuum of care or in a variety of research settings, suffering from floor and ceiling effects, and/or lacking a standardized scoring metric to allow comparisons across different health conditions [1, 2]. The methods from modern measurement theory, which includes IRT, provide a powerful framework to address these limitations and to build instruments that are more efficient, reliable, and valid measures of health-related quality of life (HRQOL).

The prevalence of premenstrual dysphoric disorder in a randomly selected group of urban and rural women

BACKGROUND Premenstrual dysphoric disorder (PMDD) was included as a provisional diagnostic category in the appendices of Diagnostic and Statistical Manual of Mental Disorders (DSM)-III-R (then called late luteal phase dysphoric disorder) and remained as an appendix in DSM-IV. Our study aimed to determine the prevalence of PMDD using all four DSM-IV research diagnostic criteria in a representative sample of women of reproductive age in the United States. METHOD Data were collected in the homes of women between the ages of 13 and 55 years in two urban and two rural sites using a random sampling procedure developed by the National Opinion Research Center. Women completed daily symptom questionnaires and provided urine specimens each day for two consecutive ovulatory menstrual cycles (ovulation was estimated for women taking oral contraceptives) and were screened for psychiatric disorders by trained interviewers. Symptoms were counted toward a diagnosis of PMDD if they worsened significantly during the late luteal week during two consecutive ovulatory menstrual cycles, occurred on days in which women reported marked interference with functioning, and were not due to another mental disorder. RESULTS In the final analysis, 1246 women who had had at least one menstrual cycle and were neither naturally nor surgically menopausal nor pregnant were selected. Of the women in the study, 1.3% met criteria for the diagnosis as defined in DSM-IV. CONCLUSIONS The prevalence of PMDD is considerably lower than DSM-IV estimates and all but one of the estimates obtained from previous studies when all DSM-IV diagnostic criteria are considered. We suggest a new process for diagnosing PMDD based on our findings.

IRT health outcomes data analysis project: an overview and summary

BackgroundIn June 2004, the National Cancer Institute and the Drug Information Association co-sponsored the conference, “Improving the Measurement of Health Outcomes through the Applications of Item Response Theory (IRT) Modeling: Exploration of Item Banks and Computer-Adaptive Assessment.” A component of the conference was presentation of a psychometric and content analysis of a secondary dataset.ObjectivesA thorough psychometric and content analysis was conducted of two primary domains within a cancer health-related quality of life (HRQOL) dataset.Research designHRQOL scales were evaluated using factor analysis for categorical data, IRT modeling, and differential item functioning analyses. In addition, computerized adaptive administration of HRQOL item banks was simulated, and various IRT models were applied and compared.SubjectsThe original data were collected as part of the NCI-funded Quality of Life Evaluation in Oncology (Q-Score) Project. A total of 1,714 patients with cancer or HIV/AIDS were recruited from 5 clinical sites.MeasuresItems from 4 HRQOL instruments were evaluated: Cancer Rehabilitation Evaluation System–Short Form, European Organization for Research and Treatment of Cancer Quality of Life Questionnaire, Functional Assessment of Cancer Therapy and Medical Outcomes Study Short-Form Health Survey.Results and conclusionsFour lessons learned from the project are discussed: the importance of good developmental item banks, the ambiguity of model fit results, the limits of our knowledge regarding the practical implications of model misfit, and the importance in the measurement of HRQOL of construct definition. With respect to these lessons, areas for future research are suggested. The feasibility of developing item banks for broad definitions of health is discussed.

Methodological issues for building item banks and computerized adaptive scales

This paper reviews important methodological considerations for developing item banks and computerized adaptive scales (commonly called computerized adaptive tests in the educational measurement literature, yielding the acronym CAT), including issues of the reference population, dimensionality, dichotomous versus polytomous response scales, differential item functioning (DIF) and conditional scoring, mode effects, the impact of local dependence, and innovative approaches to assessment using CATs in health outcomes research.

Knowledge About Breast Cancer and Participation in a Faith-Based Breast Cancer Program and Other Predictors of Mammography Screening Among African American Women and Latinas

This article assessed the impact of knowledge of breast cancer and type and intensity of participation in a church-based breast cancer education program and other factors on mammography screening among African Americans and Latinas. Logistic regression was used to assess the impact of these factors on self-reported mammography utilization. Passive participation in church-sponsored activities, measured by breast cancer information that was heard, seen, or read, was found to be significantly associated with the likelihood of mammography use among African Americans. Moreover, African Americans who reported hearing, seeing, or reading about mammograms at their churches four or more times were 15 times more likely to report mammography use within the past year than were those who encountered information only once. Messages from pastors and church bulletin announcements were the most significant predictors. An increase in knowledge was not associated with higher mammography use. For Latinas, none of the hypothesized knowledge or participation variables was found to be significant. The results suggest that faith-based breast cancer programs can be effective by adopting tailored strategies to raise awareness about the importance of early detection.