Ken Fujimoto

An assessment of the validity of the ECERS-R with implications for measures of child care quality and relations to child development.

The Early Childhood Environment Rating Scale-Revised (ECERS-R) is widely used to associate child care quality with child development, but its validity for this purpose is not well established. We examined the validity of the ECERS-R using the multidimensional Rasch partial credit model (PCM), factor analyses, and regression analyses with data from the Early Childhood Longitudinal Study Birth Cohort. The PCM identified rating category disordering, indicating previously unrecognized problems with the scales response process validity. Factor analyses identified neither a single factor nor the ECERS-R six subscales, replicating prior research regarding the scales structural validity. Criterion validity results were mixed, with small effect sizes for regressions predicting child outcomes and moderate effect sizes for regressions predicting teacher-reported quality. Our results lend empirical support to recent critiques of the ECERS-R, and we discuss implications for its future use and for the development of future measures.

Acculturative Family Distancing (AFD) and Depression in Chinese American Families

OBJECTIVE Knowledge of acculturative processes and their impact on immigrant families remains quite limited. Acculturative family distancing (AFD) is the distancing that occurs between immigrant parents and their children and is caused by breakdowns in communication and cultural value differences. It is a more proximal and problem-focused formulation of the acculturation gap and is hypothesized to increase depression via family conflict. METHOD Data were collected from 105 Chinese American high school students and their mothers. Rasch modeling was used to refine the AFD measure, and structural equation modeling was used to determine the effects of AFD on youth and maternal depression. RESULTS Findings indicate that greater AFD was associated with higher depressive symptoms and risk for clinical depression. Family conflict partially mediated this relation for youths, whereas for mothers, AFD directly increased risk for depression. Greater mother-child heritage enculturation discrepancies were associated with greater mother and child AFD. Mainstream acculturation discrepancies and language gaps between mothers and youths were not significantly associated with any of the primary outcome variables. CONCLUSIONS Results highlight the need for better understanding of how AFD and other acculturation-gap phenomena affect immigrant mental health. They also underscore the need for prevention and intervention programs that target communication difficulties and intergenerational cultural value differences.

Culturally Adapted Cognitive-Behavioral Therapy for Chinese Americans With Depression: A Randomized Controlled Trial

OBJECTIVE No randomized controlled trials (RCTs) for adults have compared the effectiveness of a well-specified psychotherapy and a culturally adapted version of the same treatment. This study evaluated the effectiveness of cognitive-behavioral therapy (CBT) and culturally adapted CBT (CA-CBT) in treating depressed Chinese-American adults. METHODS This RCT treated 50 Chinese Americans who met criteria for major depression and sought treatment at community mental health clinics. Screening of participants began in September 2008, and the last assessment was conducted in March 2011. Participants were stratified by whether they were already taking antidepressants when they first came to the clinic and randomly assigned to 12 sessions of CBT or CA-CBT. The study did not influence regular prescription practices. The primary outcomes were dropout rates and Hamilton Depression Rating Scale scores at baseline, session 4, session 8, and session 12. RESULTS Participants in CA-CBT demonstrated a greater overall decrease in depressive symptoms compared with participants in CBT, but the groups had similarly high depression rates at week 12. Differences in dropout rates for the two groups approached, but did not meet, statistical significance (7%, CA-CBT; 26%, CBT). CONCLUSIONS Chinese Americans entered this study with very severe depression. Participants in both CBT and CA-CBT demonstrated significant decreases in depressive symptoms, but the majority did not reach remission. Results suggest that these short-term treatments were not sufficient to address such severe depression and that more intensive and longer treatments may be needed. Results also indicate that cultural adaptations may confer additional treatment benefits.

Rasch Validation of the Falls Prevention Strategies Survey

UNLABELLED Finlayson ML, Peterson EW, Fujimoto KA, Plow MA. Rasch validation of the Falls Prevention Strategies Survey. OBJECTIVE To validate the Falls Prevention Strategies Survey. DESIGN Cross-sectional descriptive survey design. With the use of Rasch analysis, the following aspects of the Falls Prevention Strategies Survey were evaluated: rating scale structure, item quality, participant fit and participant ability, invariance structure, and the potential to change in response to intervention. SETTING Community-based, national sample. PARTICIPANTS Adult registrants of the North America Research Committee on Multiple Sclerosis volunteer patient registry (N=457). INTERVENTIONS Not applicable. MAIN OUTCOME MEASURES The Falls Prevention Strategies Survey is a self-report instrument addressing protective behaviors related to fall risk among adults with multiple sclerosis (MS) (eg, monitoring MS symptoms, wearing proper footwear, modifying activities). Response options reflect the frequency with which the respondent engages in the behavior (ie, never, sometimes, regularly). RESULTS Analysis indicated that the rating scale structure (ie, response options) was valid. Of the original 19 items, 8 of them misfit and needed to be dropped to obtain a valid instrument under the Rasch model. With the final 11 items, the instrument was able to distinguish participants of different ability levels across a range of 11.58 logits. Invariance structure analysis demonstrated that the instrument functioned equally for men and women, for mobility device users and nonusers, and for participants with diagnosed MS for less than or greater than 10 years. Findings indicated that approximately 50% of respondents would have room to improve on their Falls Prevention Strategies Survey scores over time. CONCLUSIONS Rasch analysis supports the use of the Falls Prevention Strategies Survey to examine the frequency of engaging in protective behaviors related to fall risk among adults with MS. The instrument shows potential to track outcomes of behaviorally oriented fall reduction interventions in this population.

Identifying High-Quality Preschool Programs: New Evidence on the Validity of the Early Childhood Environment Rating Scale–Revised (ECERS-R) in Relation to School Readiness Goals

Research Findings: The Early Childhood Environment Rating Scale–Revised (ECERS-R) is widely used, often to evaluate whether preschool programs are of sufficient quality to improve children’s school readiness. We examined the validity of the measure for this purpose. Item response theory (IRT) analyses revealed that many items did not fit together to measure single dimensions, particularly when rated by consultants as indicating aspects of quality relevant for multiple domains of child development. IRT results also conflicted with the scale developers’ expectations in terms of whether markers that they attached to higher response categories represented higher quality empirically. When reanalyzed based on experts’ ratings, IRT results also showed that relatively few indicators captured the moderate to high range of quality. Practice or Policy: Our results suggest that policymakers should carefully consider whether measures designed for specific purposes are appropriate for other high-stakes uses. We encourage continued refinement of existing quality measures, development of new measures, and the accumulation of evidence for their various uses.

Acculturative family distancing: psychometric analysis with the extended two-tier item response theory.

The acculturative family distancing (AFD) rating questionnaire measures the distancing that occurs between immigrant parents and their children, which is caused by breakdowns in communication and cultural value differences. The items included in the AFD questionnaire are indicators for more proximal and problem-focused aspects of the acculturation gap. In this study, a psychometric analysis is performed on the data gathered with the AFD instrument. One hundred twenty-three Chinese American families (112 mothers, 85 fathers, and 120 children for a total of 317) completed the questionnaire. In addition, the specifications for an extended 2-tier item factor analysis model within a Bayesian multidimensional item response theory framework are provided to psychometrically analyze the AFD data. Results suggest that the original 2 dimensions of Communication Difficulties and Cultural Values Incongruency can be further refined into 4 clinically relevant dimensions (i.e., Verbal Communication Difficulties, Nonverbal Communication Difficulties, General Cultural Values Incongruency, and Family Values Incongruency). Moreover, reducing the number of items can improve AFD estimates and reduce assessment burden.

Dependent Dirichlet Process Rating Model

Typical item response theory (IRT) rating scale models assume that the rating category threshold parameters are the same over examinees. Unfortunately, such models are inappropriate for rating data that exhibit differential item functioning (DIF). The authors introduce a new Bayesian nonparametric IRT model for rating scale items, which is more appropriate for rating data that contain DIF. The model is an infinite mixture of Rasch partial credit models, with mixture distribution modeled by the local (Dependent) Dirichlet process. The model treats the rating category thresholds as the random parameters that are subject to the mixture, with (stick-breaking) mixture weights that are covariate-dependent. Thus, the model allows the rating category thresholds to differ across items and examinees, while allowing the form of the distribution for the category thresholds to vary flexibly as a function of covariates. The authors illustrate the new model through the analysis of simulated data and real data. The model demonstrated the ability to correctly identify DIF items. Moreover, the model attained better predictive-fit performance than did other commonly used IRT rating models.

MultRasch: SAS code for the estimation of the multiplicative Rasch model parameters

MultRasch was created to estimate parameters using marginal maximum likelihood estimation for the multiplicative Rasch model originally proposed by Jim Ramsay in the 1970s (but not published; personal communication, January 28, 2010) and brought back to the psychometric community’s attention by Howard Wainer (Wainer, 2010). The probability of a correct response in the multiplicative Rasch model is given by P(x 1⁄4 1|y) 1⁄4 exp(by)/[k + exp(by)], where b is item difficulty, k is the number of distractors, and y is person ability. Person ability is represented on the positive real half-line with a natural zero point, which, as Wainer notes, may provide measurements of person ability on a ratio scale. A ratio scale would allow the use of parametric statistics methodologies that assume a continuous scale of measurement without having to worry about the potential issues associated with the use of ordinal data (Embretson, 1996, 2006; Romanoski & Douglas, 2002). As with a traditional dichotomous Rasch model, the probability of a correct answer asymptotes at one as y increases. Guessing is accommodated by the value of k, with, for example, a four-option item having a lower asymptote of .25 as y approaches zero. MultRasch was designed to run on Windows-based computers with SAS 9.1.3 or above installed. MultRasch consists of two SAS files: the executable code (MultRasch.sas), which contains the macros for the Multiplicative model and data preparation, and the call program (Call MultRasch.sas), which calls the SAS macros. The call program contains detailed comments indicating what aspects of the code can be modified (e.g., number of items, number of distractors, and file locations). The Excel output file contains three sheets: (a) ModelFitStats, which includes global model fit indices (i.e., –2LL, AIC, AICC, and BIC); (b) Item Difficulty, which contains the estimated item difficulties and standard errors along with the mean of the y estimates; and (c) PersonEBest, which provides the empirical Bayes estimates of person ability and standard errors. A small simulation study (30 replications) was conducted to investigate the correlations between generating (20 items, U[.1, 1.1]); 1,000, persons N[0, 1] with the code adjusting for negative values) and estimated values of b and y using MultRasch. The average correlations between generating and estimated item difficulties and person abilities were .99 (SD 1⁄4 .001) and .75 (SD 1⁄4 .026), respectively. We are currently extending the range of simulated conditions (e.g., number of items and persons; varying distributions) and evaluating the recovery of the multiplicative Rasch model compared to other item response theory models. MultRasch can be downloaded for free at http://www.uic.edu/~evsmith/MultRasch, along with a sample input and corresponding output files. Computer Program Exchange

A general Bayesian multilevel multidimensional IRT model for locally dependent data

Many item response theory (IRT) models take a multidimensional perspective to deal with sources that induce local item dependence (LID), with these models often making an orthogonal assumption about the dimensional structure of the data. One reason for this assumption is because of the indeterminacy issue in estimating the correlations among the dimensions in structures often specified to deal with sources of LID (e.g., bifactor and two-tier structures), and the assumption usually goes untested. Unfortunately, the mere fact that assessing these correlations is a challenge for some estimation methods does not mean that data seen in practice support such orthogonal structure. In this paper, a Bayesian multilevel multidimensional IRT model for locally dependent data is presented. This model can test whether item response data violate the orthogonal assumption that many IRT models make about the dimensional structure of the data when addressing sources of LID, and this test is carried out at the dimensional level while accounting for sampling clusters. Simulations show that the model presented is effective at carrying out this task. The utility of the model is also illustrated on an empirical data set.

Examining the Category Functioning of the ECERS-R Across Eight Data Sets

Classroom quality measures, such as the Early Childhood Environment Rating Scale, Revised (ECERS-R), are widely used in research, practice, and policy. Increasingly, these uses have been for purposes not originally intended, such as contributing to consequential policy decisions. The current study adds to the recent evidence of problems with the ECERS-R standard stop-scoring by analyzing eight studies offering 14 waves of data collection in approximately 4,000 classrooms. Our analysis, which featured the nominal response model, generalized partial credit model, partial credit model, within-category averages of total scores, and point-biserial correlations, revealed that all 36 items had categories that did not follow an ordinal progression with respect to quality. Additionally, our results showed that the category problems accumulated to the scale score. The results caution against the use of the standard raw scoring and encourage development of alternative scoring methods for the ECERS-R.