Everett V. Smith

Optimizing rating scales for self-efficacy (and other) research

This article (a) discusses the assumptions underlying the use of rating scales, (b) describes the use of information available within the context of Rasch measurement that may be useful for optimizing rating scales, and (c) demonstrates the process in two studies. Participants in the first study were 330 fourth- and fifth-grade students. Participants provided responses to the Index of Self-Efficacy for Writing. Based on category counts, average measures, thresholds and category fit statistics, the responses on the original 10-point scale were better represented by a 4-point scale. The modified 4-point scale was given to a replication sample of 668 fourth- and fifth-grade students. The rating scale structure was found to be congruent with the results from the first study. In addition, the item fit statistics and item hierarchy indicated the writing self-efficacy construct to be stable across the two samples. Combined, these results provide evidence for the generalizability of the findings and hence utility of this scale for use with samples of respondents from the same population.

A Rasch Perspective

Measurement error is a common theme in classical measurement models used in testing and assessment. In classical measurement models, the definition of measurement error and the subsequent reliability coefficients differ on the basis of the test administration design. Internal consistency reliability specifies error due primarily to poor item sampling. Rater reliability indicates error due to inconsistency among raters. For estimates of test-retest reliability, error is attributed mainly to changes over time. In alternate-forms reliability, error is assumed to be due largely to variation between samples of items on test forms. Rasch models can also compute reliability estimates of scores under different test situations. The authors therefore present the Rasch perspective on calculating reliability (measurement error) and present Rasch measurement model programs to compute the various reliability estimates.

An Application of Generalizability Theory and Many-Facet Rasch Measurement Using a Complex Problem-Solving Skills Assessment:

This study describes the use of generalizability theory (GT) and many-facet Rasch measurement (MFRM) to evaluate psychometric properties of responses obtained from an assessment designed to measure complex problem-solving skills. The assessment revolved around the school activity of kickball. The task required of each student was to decide on a team T-shirt based on visual characteristics of the T-shirts and written descriptions of other T-shirt characteristics. Forty-four fourth-grade students, comprising the control group of a longitudinal research project to foster complex problem-solving skills of disadvantaged urban youth, participated in this study. Results indicate both measurement techniques agree on the relative magnitudes of variation among the facets but differ on how to handle the sources of variation.

A study strategies self-efficacy instrument for use with community college students

Theories of self-efficacy and self-regulation were used to examine scores from an instrument that measures self-efficacy for using self-regulatory study strategies. The authors investigated the dimensionality of responses to the Study Skills Self-Efficacy Scale using exploratory factor analysis and Rasch measurement. They also investigated the utility of the Rasch measures in differentiating between groups of students who report being academically successful or at risk. The participants were 550 social science students at a midsized northeastern community-technical college. Results indicated that responses define three related dimensions and that measures were able to differentiate between students reporting to be academically successful or at risk. Additional items need to be developed to increase measurement precision along various portions of the self-efficacy dimensions.

VALIDITY OF THE TIMPSI FOR ESTIMATING CONCURRENT PERFORMANCE ON THE TEST OF INFANT MOTOR PERFORMANCE

Purpose: The TIMPSI, a short version of the Test of Infant Motor Performance (TIMP), is used for screening. The project was designed to compare concurrent TIMPSI scores with those on the TIMP and recommend cutscores for clinical decision making. Methods: Infants (n = 990) were recruited reflecting the race/ethnicity of US infants with low birth weight. From 67 to 97 infants were tested in 2-week age groups ranging from 34 to 35 weeks postmenstrual age through 16 to 17 weeks post-term. Rasch analysis of raw scores was used. TIMPSI cutscores ranging from the mean to −1.00 standard deviation (SD) were compared with performance above/below −0.5 SD on the TIMP to assess accuracy of classification. Results: The TIMPSI was a valid screening instrument when compared with concurrent performance on the TIMP. Conclusion: A cutscore of −0.25 SD appears useful in predicting the best combination of false negatives (5.8%) and false positives (12.5%) with overall accuracy of classification of 81.7%.

Measuring Sexual Risk for HIV: A Rasch Scaling Approach

In this study, we developed an HIV transmission risk scale and examined its psychometric properties using data on sexual behavior obtained from a probability sample of adult men who have sex with men living in Chicago. We used Messick’s (Am Psychol 50:741–749, 1995) conceptualization of unified validity theory to organize the psychometric properties of data. Evidence related to scale content was investigated via Rasch item fit statistics, point-measure correlations, and expert evaluation. The substantive aspect of validity was addressed by interpreting the meaningfulness of the item difficulty hierarchy (continuum of risky behaviors) and assessment of person fit. The structural aspect of validity was assessed using Rasch item fit statistics, principal component analysis of standardized residuals, and other residual analyses. The generalizability aspect of validity was investigated via internal consistency reliability estimates for both items and persons, and aspects of external validity were addressed by examining between-group differences with respect to levels of risky behavior. Applications and suggested future studies are discussed.

Psychometric Properties of the Pediatric Balance Scale Using Rasch Analysis

Purpose: The Pediatric Balance Scale (PBS) is a 14-item measure of functional balance for children. This study examined PBS dimensionality, rating scale function, and hierarchical properties using Rasch analysis. Methods: The PBS data were analyzed retrospectively for 823 children, aged 2 to 13 years, with uni- and multidimensional Rasch partial credit models. Results: The PBS best fits a unidimensional model based on the Bayesian information criterion analysis (12 400.73 vs 12 404.48), strong correlations between 3 proposed dimensions (r = 0.946-0.979), and high internal consistency (Cronbach &agr; = 0.94). Analysis of rating scale functioning is limited by small numbers of children achieving low scores on easy items. Item maps indicated a ceiling effect but no substantive gaps between item difficulty estimates. Conclusion: The PBS items are best targeted to preschool-age children; additional children with known balance dysfunction are required to fully assess functioning of the easiest PBS items. Revisions may improve PBS utility in older children.

Assessing student perceptions of affective outcomes of special education programs: Instrument development, validation, and comparisons to regular education students

The development of the 45-item likert-type response scale Student Survey for assessing the following five affective goals of Connecticuts Common Core of Learning is described: Academic Competence, Social Competence, Social Integration, Involvement in Educational Decision Making, and Consumer Satisfaction. Content validity literature-based and judgmental procedures are described; supportive evidence of construct validity is provided using a sample of 436 special education students (Grades 4, 6, and 8) from eight school districts who were eligible to take the statewide Connecticut Mastery Test. These data were analyzed using confirmatory factor analysis and item response theory techniques. Alpha reliabilities for the special education sample ranged from .70 to .91 for the five dimensions. Differences between special education students and a sample of 1,229 regular education students from the same districts are described.

MultRasch: SAS code for the estimation of the multiplicative Rasch model parameters

MultRasch was created to estimate parameters using marginal maximum likelihood estimation for the multiplicative Rasch model originally proposed by Jim Ramsay in the 1970s (but not published; personal communication, January 28, 2010) and brought back to the psychometric community’s attention by Howard Wainer (Wainer, 2010). The probability of a correct response in the multiplicative Rasch model is given by P(x 1⁄4 1|y) 1⁄4 exp(by)/[k + exp(by)], where b is item difficulty, k is the number of distractors, and y is person ability. Person ability is represented on the positive real half-line with a natural zero point, which, as Wainer notes, may provide measurements of person ability on a ratio scale. A ratio scale would allow the use of parametric statistics methodologies that assume a continuous scale of measurement without having to worry about the potential issues associated with the use of ordinal data (Embretson, 1996, 2006; Romanoski & Douglas, 2002). As with a traditional dichotomous Rasch model, the probability of a correct answer asymptotes at one as y increases. Guessing is accommodated by the value of k, with, for example, a four-option item having a lower asymptote of .25 as y approaches zero. MultRasch was designed to run on Windows-based computers with SAS 9.1.3 or above installed. MultRasch consists of two SAS files: the executable code (MultRasch.sas), which contains the macros for the Multiplicative model and data preparation, and the call program (Call MultRasch.sas), which calls the SAS macros. The call program contains detailed comments indicating what aspects of the code can be modified (e.g., number of items, number of distractors, and file locations). The Excel output file contains three sheets: (a) ModelFitStats, which includes global model fit indices (i.e., –2LL, AIC, AICC, and BIC); (b) Item Difficulty, which contains the estimated item difficulties and standard errors along with the mean of the y estimates; and (c) PersonEBest, which provides the empirical Bayes estimates of person ability and standard errors. A small simulation study (30 replications) was conducted to investigate the correlations between generating (20 items, U[.1, 1.1]); 1,000, persons N[0, 1] with the code adjusting for negative values) and estimated values of b and y using MultRasch. The average correlations between generating and estimated item difficulties and person abilities were .99 (SD 1⁄4 .001) and .75 (SD 1⁄4 .026), respectively. We are currently extending the range of simulated conditions (e.g., number of items and persons; varying distributions) and evaluating the recovery of the multiplicative Rasch model compared to other item response theory models. MultRasch can be downloaded for free at http://www.uic.edu/~evsmith/MultRasch, along with a sample input and corresponding output files. Computer Program Exchange

Psychometric Evaluation of the Family Effectiveness Measure Among Low-Income African American Single-Parent Families

Aims: Family functioning is influenced by culture, socioeconomic status, family structure, and the family life cycle stage, and an instrument that is developed to assess family functioning in one population may not validly assess families in another. Measurement of family functioning in diverse populations requires instruments tailored to the population. The purpose of this study was to conduct psychometric testing and refinement of the Family Effectiveness Measure (FEM) to produce a reliable, valid measure of family functioning for use by clinicians and researchers with low-income African American mother-headed single-parent families with children. Theoretical framework: Psychometric theory combining approaches from classical test theory and item response theory (Rasch rating scale analysis) guided this research. Participants: A convenience sample of 415 low-income African American single mothers took part. Method: The 42-item FEM was administered once with instruments measuring constructs theoretically associated with family functioning: depression, satisfaction with psychological support from family, and parent–child conflict tactics. After 2 weeks, a random sample of 83 participants completed the survey again. Utilizing classical test theory analyses, this study assessed factor structure, internal consistency and stability reliability, and construct validity. Construct validity was assessed by correlations between the FEM and the three other instruments. Rasch analyses were used to assess dimensionality of items, person and item reliabilities, Western Journal of Nursing Research Volume 30 Number 8 December 2008 1021-1022