Anastasia E. Raczek

The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs

Cross-sectional data from the Medical Outcomes Study (MOS) were analyzed to test the validity of the MOS 36-Item Short-Form Health Survey (SF-36) scales as measures of physical and mental health constructs. Results from traditional psychometric and clinical tests of validity were compared. Principal components analysis was used to test for hypothesized physical and mental health dimensions. For purposes of clinical tests of validity, clinical criteria defined mutually exclusive adult patient groups differing in severity of medical and psychiatric conditions. Scales shown in the components analysis to primarily measure physical health (physical functioning and role limitations-physical) best distinguished groups differing in severity of chronic medical condition and had the most pure physical health interpretation. Scales shown to primarily measure mental health (mental health and role limitations-emotional) best distinguished groups differing in the presence and severity of psychiatric disorders and had the most pure mental health interpretation. The social functioning, vitality, and general health perceptions scales measured both physical and mental health components and, thus, had the most complex interpretation. These results are useful in establishing guidelines for the interpretation of each scale and in documenting the size of differences between clinical groups that should be considered very large.

The validity and relative precision of MOS short- and long-form health status scales and Dartmouth COOP charts. Results from the Medical Outcomes Study

This study estimated the validity and relative precision (RP) of four methods (MOS long- and short-form scales, global items, and COOP Poster Charts) in measuring six general health concepts. The authors also tested whether and how precisely each method discriminated relatively well adult patients (N = 638) from those with only severe chronic medical (N = 168) and only psychiatric conditions (N = 163), as clinically defined. For comparisons between the well group and both medical and psychiatric groups, RP estimates favored long-form over short-form, multi-item scales, and favored multi-item scales over single-item global measures and poster charts. In relation to long forms, short-form multi-item scales achieved a median RP of .93; RP estimates for global items and poster charts were .81 and .67, respectively. Variations in RP across methods and concepts were linked to differences in the coarseness of measurement scales, reliability, and content (including the effects of chart illustrations). These variations in RP have implications for the interpretation of scores, the statistical power of comparisons between clinical groups, and the size of confidence intervals around individual patient scores.

Comparison of Rasch and summated rating scales constructed from SF-36 physical functioning items in seven countries: results from the IQOLA Project. International Quality of Life Assessment

Rasch models for polytomous items were used to assess the scaling assumptions and compare item response patterns in the 10-item SF-36 physical functioning scale (PF-10) for general population respondents in Denmark, Germany, Italy, the Netherlands, Sweden, the United Kingdom, and the United States. The Rasch model of physical functioning developed in the United States was compared to models for other countries, and each country was compared to a multinational composite. Strong scale congruence across the seven countries was demonstrated; items that varied between countries and from the composite may reflect unique cultural response patterns or differences in translation. Scoring algorithms based on the Rasch model for each country were superior to the current Likert scoring in tests of relative validity (RV) in discriminating among age groups in all countries. In relation to the Likert PF-10 scoring (RV = 1.00), scores estimated using the Rasch rating scale model achieve a median RV of 1.31 (range: 1.01-1.59), while the Rasch partial credit model attained a median RV of 1.44 (range: 1.01-2.23). Rasch models hold good potential for improving health status measures, estimating individual scores when responses to scale items are missing, and equating scores across countries.

Comparison of Rasch and Summated Rating Scales Constructed from SF-36 Physical Functioning Items in Seven Countries: Results from the IQOLA Project

Rasch models for polytomous items were used to assess the scaling assumptions and compare item response patterns in the 10-item SF-36 physical functioning scale (PF-10) for general population respondents in Denmark, Germany, Italy, the Netherlands, Sweden, the United Kingdom, and the United States. The Rasch model of physical functioning developed in the United States was compared to models for other countries, and each country was compared to a multinational composite. Strong scale congruence across the seven countries was demonstrated; items that varied between countries and from the composite may reflect unique cultural response patterns or differences in translation. Scoring algorithms based on the Rasch model for each country were superior to the current Likert scoring in tests of relative validity (RV) in discriminating among age groups in all countries. In relation to the Likert PF-10 scoring (RV = 1.00), scores estimated using the Rasch rating scale model achieve a median RV of 1.31 (range: 1.01-1.59), while the Rasch partial credit model attained a median RV of 1.44 (range: 1.01-2.23). Rasch models hold good potential for improving health status measures, estimating individual scores when responses to scale items are missing, and equating scores across countries.

IRT health outcomes data analysis project: an overview and summary

BackgroundIn June 2004, the National Cancer Institute and the Drug Information Association co-sponsored the conference, “Improving the Measurement of Health Outcomes through the Applications of Item Response Theory (IRT) Modeling: Exploration of Item Banks and Computer-Adaptive Assessment.” A component of the conference was presentation of a psychometric and content analysis of a secondary dataset.ObjectivesA thorough psychometric and content analysis was conducted of two primary domains within a cancer health-related quality of life (HRQOL) dataset.Research designHRQOL scales were evaluated using factor analysis for categorical data, IRT modeling, and differential item functioning analyses. In addition, computerized adaptive administration of HRQOL item banks was simulated, and various IRT models were applied and compared.SubjectsThe original data were collected as part of the NCI-funded Quality of Life Evaluation in Oncology (Q-Score) Project. A total of 1,714 patients with cancer or HIV/AIDS were recruited from 5 clinical sites.MeasuresItems from 4 HRQOL instruments were evaluated: Cancer Rehabilitation Evaluation System–Short Form, European Organization for Research and Treatment of Cancer Quality of Life Questionnaire, Functional Assessment of Cancer Therapy and Medical Outcomes Study Short-Form Health Survey.Results and conclusionsFour lessons learned from the project are discussed: the importance of good developmental item banks, the ambiguity of model fit results, the limits of our knowledge regarding the practical implications of model misfit, and the importance in the measurement of HRQOL of construct definition. With respect to these lessons, areas for future research are suggested. The feasibility of developing item banks for broad definitions of health is discussed.

A New Model for Student Support in High-Poverty Urban Elementary Schools Effects on Elementary and Middle School Academic Outcomes

Efforts to support children in schools require addressing not only academic issues, but also out-of-school factors that can affect students’ ability to succeed. This study examined academic achievement of students participating in City Connects, a student support intervention operating in high-poverty elementary schools. The sample included 7,948 kindergarten to fifth-grade students in a large urban district during 1999–2009. School- and student-level treatment effects on report card grades and standardized test scores in elementary through middle school were estimated. Propensity score methods accounted for pre-intervention group differences. City Connects students demonstrated higher report card scores than comparisons and scored higher on middle school English language arts and mathematics tests. This study provides evidence for the value of addressing out-of-school factors that impact student learning.

Comparing Teacher Assessment and Standard Task Results in England: The relationship between pupil characteristics and attainment

Abstract This study examines the 1992 National Curriculum assessment data from one large LEA in England in order to address the issue of equity. For comparison purposes we also present additional data obtained front the same sample of pupils on an NFER standardised word recognition test. The report focuses on the relative performance of gender, low income, linguistic, and special needs groups on a standardised reading test and the teacher (TA) and standard task (ST) performance assessments administered in 1992 to 7‐year‐olds as part of the national curriculum (NC) in England and Wales. The impact of schools and teacher effectiveness on student attainments scores is also examined and discussed. Briefly, the findings show that irrespective of the method of assessment, differences in attainment were found between most pupil groups investigated. However, importantly, only very modest evidence was found that particular methods of assessment appeared either to reduce or increase the differences in attainment an...

Comparison of Rasch and Summated Rating Scales Constructed from SF-36 Physical Functioning Items in Seven Countries

Rasch models for polytomous items were used to assess the scaling assumptions and compare item response patterns in the 10-item SF-36 physical functioning scale (PF-10) for general population respondents in Denmark, Germany, Italy, the Netherlands, Sweden, the United Kingdom, and the United States. The Rasch model of physical functioning developed in the United States was compared to models for other countries, and each country was compared to a multinational composite. Strong scale congruence across the seven countries was demonstrated; items that varied between countries and from the composite may reflect unique cultural response patterns or differences in translation. Scoring algorithms based on the Rasch model for each country were superior to the current Likert scoring in tests of relative validity (RV) in discriminating among age groups in all countries. In relation to the Likert PF-10 scoring (RV = 1.00), scores estimated using the Rasch rating scale model achieve a median RV of 1.31 (range: 1.01-1.59), while the Rasch partial credit model attained a median RV of 1.44 (range: 1.01-2.23). Rasch models hold good potential for improving health status measures, estimating individual scores when responses to scale items are missing, and equating scores across countries.

The Long-Term Impact of Systematic Student Support in Elementary School: Reducing High School Dropout.

Dropping out of high school has adverse consequences, including negative effects on employment, lifetime earnings, and physical health. Students often fail to complete high school for complex reasons that often manifest long before they reach high school. This study examines the link between participation in a comprehensive elementary school student support intervention and high school dropout. In this study, students who attended intervention elementary schools in a large, urban, high-poverty district during 2001–2014 (N=894) were compared to students who did not attend intervention schools (N=10,200). Likelihood of dropping out in grades 9–12 was estimated using propensity score-weighted Discrete Event History Analysis. Intervention students had approximately half the odds of dropout (p<.001); the probability of dropout for intervention was 9.2%, compared to 16.6% for non-intervention students. Individually tailored student support interventions during elementary school can lead to lasting and meaningful effects.

Performance Assessment & Issues of Differential Impact: The British Experience- Lessons for America

In the late 1980s, a powerful movement, alternatively labeled authentic, alternative, performance or, incorrectly, new assessment, emerged to contest the hegemony of standardized multiple-choice testing (e.g., Mitchell, 1992, Wiggins, 1989). (We use the simple term “assessment” in referring to this movement). Proponents of assessment believe that student learning and progress are best appraised, not by having students select answers, but by having them supply an extended response, generate material for portfolios, perform exhibitions, carry out experiments, or produce a tangible product that can be evaluated on its merits.