Lori McLeod

On the relationship between the higher-order factor model and the hierarchical factor model

The relationship between the higher-order factor model and the hierarchical factor model is explored formally. We show that the Schmid-Leiman transformation produces constrained hierarchical factor solutions. Using a generalized Schmid-Leiman transformation and its inverse, we show that for any unconstrained hierarchical factor model there is an equivalent higher-order factor model with direct effects (loadings) on the manifest variables from the higher-order factors. Therefore, the class of higher-order factor models (without direct effects of higher-order factors) is nested within the class of unconstrained hierarchical factor models. In light of these formal results, we discuss some implications for testing the higher-order factor model and the issue of general factor. An interesting aspect concerning the efficient fitting of the higher-order factor model with direct effects is noted.

Interpreting patient-reported outcome results: US FDA guidance and emerging methods.

In recent years, the US FDA has become more critical of instruments used to measure patient-reported outcomes (PROs) in clinical trials. To facilitate decisions related to the approval of drugs, labels and promotional claims based on PROs, the FDA created the Study Endpoints and Label Development (SEALD) group. SEALD has developed a PRO guidance related to the use of PRO measures used to support drug approvals and label claims, including recommendations for establishing thresholds for meaningful change at the individual level (i.e., defining a responder). This article examines in detail the FDA-recommended methodology for defining a responder and analyzing responder-based PRO measure results. We also present other responder analysis approaches for consideration in furthering the precision and interpretation of this methodology.

Using Classical Test Theory, Item Response Theory, and Rasch Measurement Theory to Evaluate Patient-Reported Outcome Measures: A Comparison of Worked Examples

OBJECTIVE To provide comparisons and a worked example of item- and scale-level evaluations based on three psychometric methods used in patient-reported outcome development-classical test theory (CTT), item response theory (IRT), and Rasch measurement theory (RMT)-in an analysis of the National Eye Institute Visual Functioning Questionnaire (VFQ-25). METHODS Baseline VFQ-25 data from 240 participants with diabetic macular edema from a randomized, double-masked, multicenter clinical trial were used to evaluate the VFQ at the total score level. CTT, RMT, and IRT evaluations were conducted, and results were assessed in a head-to-head comparison. RESULTS Results were similar across the three methods, with IRT and RMT providing more detailed diagnostic information on how to improve the scale. CTT led to the identification of two problematic items that threaten the validity of the overall scale score, sets of redundant items, and skewed response categories. IRT and RMT additionally identified poor fit for one item, many locally dependent items, poor targeting, and disordering of over half the response categories. CONCLUSIONS Selection of a psychometric approach depends on many factors. Researchers should justify their evaluation method and consider the intended audience. If the instrument is being developed for descriptive purposes and on a restricted budget, a cursory examination of the CTT-based psychometric properties may be all that is possible. In a high-stakes situation, such as the development of a patient-reported outcome instrument for consideration in pharmaceutical labeling, however, a thorough psychometric evaluation including IRT or RMT should be considered, with final item-level decisions made on the basis of both quantitative and qualitative results.

Validation of the Knee Injury and Osteoarthritis Outcome Score Subscales for Patients With Articular Cartilage Lesions of the Knee

Background: The Knee Injury and Osteoarthritis Outcome Score (KOOS) assesses acute and chronic knee injuries or early-onset osteoarthritis in young, active patients. The United States Food and Drug Administration guidelines recommend that patient-reported outcome instruments used to support clinical trial label claims should demonstrate content validity using patient input and have acceptable psychometric properties in the target population. To use the KOOS subscales in safety and efficacy trials assessing new treatments for patients with articular cartilage lesions, additional validation work, using input from patients with articular cartilage lesions, was necessary. Purpose: Qualitative and quantitative evaluations of the KOOS subscales’ validity among patients with articular cartilage lesions were conducted to support their use as clinically meaningful end points in clinical trials. Study Design: Cohort study (diagnosis); Level of evidence, 2. Methods: For qualitative analysis, cognitive interviews involving concept elicitation and cognitive debriefing with the KOOS items were conducted with 15 participants aged 25 to 52 years. Participants either were candidates for cartilage repair or had undergone cartilage repair 6 months or more before the study. For the quantitative analysis, a psychometric evaluation of the KOOS was conducted with clinical trial data from 54 patients, aged 18 to 55 years, evaluating the Cartilage Autograft Implantation System in the United States (n = 29) and the European Union (n = 25). Data were collected before surgery and at 7 postsurgical visits up to 12 months. Internal consistency and test-retest reliability, construct validity, responsiveness, and estimates of the minimal detectable change (MDC) were assessed. Test-retest reliability was assessed using data from months 2 and 3 on a subset of stable patients. Results: Qualitative research confirmed that concepts measured on the KOOS are important to patients with articular cartilage lesions. Most participants reported the KOOS was comprehensive and appropriate. In the quantitative research, KOOS subscales showed excellent internal consistency reliability (range, .74-.97 at baseline) and test-retest reliability (range, .78-.82). Construct validity results supported hypothesized relationships, with significant correlations (r ≥ .50) in the expected directions. Responsiveness analyses demonstrated excellent sensitivity to change; standardized response means ranged from 0.8 to 1.2, and MDC estimates ranged from 7.4 to 12.1. Conclusion: The study results support the use of the KOOS subscales among patients with articular cartilage lesions.

A Bayesian Method for the Detection of Item Preknowledge in Computerized Adaptive Testing

With the increased use of continuous testing in computerized adaptive testing, new concerns about test security have evolved, such as how to ensure that items in an item pool are safeguarded from theft. In this article, procedures to detect test takers using item preknowledge are explored. When test takers use item preknowledge, their item responses deviate from the underlying item response theory (IRT) model, and estimated abilities may be inflated. This deviation may be detected through the use of person-fit indices. A Bayesian posterior log odds ratio index is proposed for detecting the use of item preknowledge. In this approach to person fit, the estimated probability that each test taker has preknowledge of items is updated after each item response. These probabilities are based on the IRT parameters, a model specifying the probability that each item has been memorized, and the test taker’s item responses. Simulations based on an operational computerized adaptive test (CAT) pool are used to demonstrate the use of the odds ratio index.

Responsiveness of the Acne-Specific Quality of Life Questionnaire (Acne-QoL) to treatment for acne vulgaris in placebo-controlled clinical trials

The Acne-Specific Quality of Life Questionnaire (Acne-QoL) was developed to measure the impact of facial acne across four dimensions of patient quality of life. The main objective of the current study was to evaluate the responsiveness of this instrument. Secondarily, this study provided an opportunity to extend the developers psychometric validation. The Acne-QoL was utilized in two randomized, double-blind, placebo-controlled studies of the efficacy of Estrostep® (norethindrone acetate/ethinyl estradiol) in the treatment of facial acne; a total of 296 Estrostep® and 295 placebo patients were evaluated. The Acne-QoL was completed at the beginning, middle (cycle 3), and end (cycle 6) of the 6-month treatment period. The responsiveness of the Acne-QoL was demonstrated through its ability to detect both small (baseline to mid-study) and moderate (baseline to study end) treatment advantages for Estrostep® patients. Confirmatory factor analysis supported the subscale structure, and internal consistency estimates were excellent. Convergent and discriminant validity were supported by correlations between Acne-QoL scores and clinical measures that were both in the direction and relative magnitude hypothesized. Finally, item response theory analyses confirmed that each item is highly related to its subscales latent construct and that each subscale is sensitive across a broad range of the underlying continuum. The results of this evaluation confirm that the Acne-QoL is responsive, internally consistent, and valid.

US FDA patient-reported outcome guidance: great expectations and unintended consequences

Release of the US FDA patient-reported outcome (PRO) guidance raised expectations within the pharmaceutical industry for the use of PRO measures in support of labeling claims. The FDA developed the guidance with admirable intent, and the recommendations within this document are based on sound scientific principles. However, implementation of the guidance has been somewhat inconsistent within the Study Endpoints and Label Development (SEALD) and across the various FDA-reviewing divisions. Industry and regulatory bodies need to work toward gaining common ground to best support registration of treatments that could extend patients’ lives, reduce symptoms, and/or improve health-related quality of life. PROs are valuable tools in communicating these messages, and realistic implementation of the FDA PRO Guidance may truly facilitate this process.

The king’s foot of patient-reported outcomes: current practices and new developments for the measurement of change

PurposeAssessing change remains a challenge in patient-reported outcomes. In June 2009, a group of psychometricians, biostatisticians, and behavioral researchers from other disciplines convened as a Longitudinal Analysis of Patient-Reported Outcomes Working group as part of the Statistical and Applied Mathematical Sciences Institute Summer Psychometric program to discuss the complex issues that arise when conceptualizing and operationalizing “change” in patient-reported outcome (PRO) measures and related constructs. This white paper summarizes these issues and provides recommendations and possible paths for dealing with the complexities of measuring change.Methods/ResultsThis article presents and discusses issues associated with: (1) conceptualizing and operationalizing change in PRO measures; (2) modeling change using state-of-the-art statistical methods; (3) impediments to detecting true change; (4) new developments to deal with these challenges; and (5) important gaps that are fertile ground for future research.ConclusionsThere was a consensus that important research still needs to be performed in order develop and refine high-quality PRO measures and statistical methods to analyze and model change in PRO constructs.

Development and preliminary validation of the Menopause Symptoms Treatment Satisfaction Questionnaire (MS-TSQ)

Objective: The Menopause Symptoms Treatment Satisfaction Questionnaire, an eight-item questionnaire with a 4-week recall period, was developed to assess womens satisfaction with treatment for symptoms associated with menopause. We describe the development and initial testing of the scale. Design: Following standard instrument-development procedures, focus groups were conducted with menopausal women experiencing hot flushes to generate potential constructs. Multiple items were drafted to address each construct. An iterative process of cognitive testing, item revision, and item reduction was followed to identify the most appropriate items and optimal response scales. The psychometric validation of the questionnaire used data collected through a multicenter, randomized, double-blind, placebo-controlled study including 543 postmenopausal women. Psychometric analyses were conducted to explore potential item reduction and to address questionnaire scaling and scoring. Internal consistency reliability, construct validity, and discriminant validity of the new scale were also examined. Results: The questionnaire includes items addressing the control of daytime and nighttime hot flushes; effects of treatment on sleep, mood, libido, and cognition; medication tolerability; and overall satisfaction. Correlation analyses indicated that the items are related to each other without being overly redundant and that the item set is best described using a one-factor model. The subsequent scale score demonstrated sound internal consistency reliability, strong construct validity, and good discriminant validity. Conclusions: The results of the development and initial validation are favorable. It is expected that the questionnaire will prove to be a worthwhile tool for assessing womens satisfaction with treatment for menopausal symptoms.

Evaluating Minimal Clinically Important Differences for the Acne-Specific Quality of Life Questionnaire

AbstractBackground: The Acne-Specific Quality of Life Questionnaire (Acne-QoL) is a responsive, reliable and valid instrument developed to measure the impact of facial acne across four dimensions of patient QOL. Score changes on this instrument have been used to report statistically significant treatment advantages for a low-dose oral contraceptive (Estrostep®), containing norethisterone (norethindrone) acetate (NA) 1mg and ethinylestradiol (EE) [20, 30, 35mg] as compared with placebo in women with moderate acne vulgaris. However, the question remained if these statistically significant results were also clinically meaningful. Objectives: To evaluate the statistically significant Acne-QoL benefits observed with NA/EE in terms of their clinical significance, and to compare the three different approaches for defining a minimal clinically important difference (MCID) for the Acne-QoL instrument. Methods: Since the optimum method for estimating MCIDs has yet to be established, three different published approaches for determining MCIDs were applied and compared using data from two randomised, double-blind, placebocontrolled studies of the efficacy of NA/EE in the treatment of facial acne. Results: Although the approaches differed substantially, the resulting MCID estimates were comparable. Specifically, the MCID estimates ranged from 0.50–10.3 mean change per item, depending on the domain. The results showed that the statistically significant treatment advantages for NA/EE were also clinically significant. Conclusion: When applied to the change scores present, the results showed that the statistically significant treatment advantages for NA/EE were also clinically significant.