Pui-Wa Lei

Development and Initial Validation of the Counseling Center Assessment of Psychological Symptoms-34

A short version of the Counseling Center Assessment of Psychological Symptoms–62 (CCAPS-62) was created via three studies. The final short version (CCAPS-34), which contains 34 items and 7 subscales, demonstrated good discrimination power, support for the proposed factor structure, strong initial convergent validity, and adequate test–retest stability over 1-week and 2-week intervals.

Controlling Item Exposure and Test Overlap in Computerized Adaptive Testing

This article proposes an item exposure control method, which is the extension of the Sympson and Hetter procedure and can provide item exposure control at both the item and test levels. Item exposure rate and test overlap rate are two indices commonly used to track item exposure in computerized adaptive tests. By considering both indices, item exposure can be monitored at both the item and test levels. To control the item exposure rate and test overlap rate simultaneously, the modified procedure attempted to control not only the maximum value but also the variance of item exposure rates. Results indicated that the item exposure rate and test overlap rate could be controlled simultaneously by implementing the modified procedure. Item exposure control was improved and precision of trait estimation decreased when a prespecified maximum test overlap rate was stringent.

Effects of Vertical Scaling Methods on Linear Growth Estimation.

Vertical scaling is necessary to facilitate comparison of scores from test forms of different difficulty levels. It is widely used to enable the tracking of student growth in academic performance over time. Most previous studies on vertical scaling methods assume relatively long tests and large samples. Little is known about their performance when the sample is small or the test is short, challenges that small testing programs often face. This study examined effects of sample size, test length, and choice of item response theory (IRT) models on the performance of IRT-based scaling methods (concurrent calibration, separate calibration with Stocking–Lord, Haebara, Mean/Mean, and Mean/Sigma transformation) in linear growth estimation when the 2-parameter IRT model was appropriate. Results showed that IRT vertical scales could be used for growth estimation without grossly biasing growth parameter estimates when sample size was not large, as long as the test was not too short (≥20 items), although larger sample sizes would generally increase the stability of the growth parameter estimates. The optimal rate of return in total estimation error reduction as a result of increasing sample size appeared to be around 250. Concurrent calibration produced slightly lower total estimation error than separate calibration in the worst combination of short test length (≤20 items) and small sample size (n ≤ 100), whereas separate calibration, except in the case of the Mean/Sigma method, produced similar or somewhat lower amounts of total error in other conditions.

CTTITEM: SAS macro and SPSS syntax for classical item analysis

This article describes the functions of a SAS macro and an SPSS syntax that produce common statistics for conventional item analysis including Cronbach’s alpha, item difficulty index (p-value or item mean), and item discrimination indices (D-index, point biserial and biserial correlations for dichotomous items and item-total correlation for polytomous items). These programs represent an improvement over the existing SAS and SPSS item analysis routines in terms of completeness and user-friendliness. To promote routine evaluations of item qualities in instrument development of any scale, the programs are available at no charge for interested users. The program codes along with a brief user’s manual that contains instructions and examples are downloadable from suen.ed.psu.edu/~pwlei/plei.htm.

Performance of Fit Indices in Choosing Correct Cognitive Diagnostic Models and Q-Matrices:

In applications of cognitive diagnostic models (CDMs), practitioners usually face the difficulty of choosing appropriate CDMs and building accurate Q-matrices. However, functions of model-fit indices that are supposed to inform model and Q-matrix choices are not well understood. This study examines the performance of several promising model-fit indices in selecting model and Q-matrix under different sample size conditions. Relative performance between Akaike information criterion and Bayesian information criterion in model and Q-matrix selection appears to depend on the complexity of data generating models, Q-matrices, and sample sizes. Among the absolute fit indices, MX2 is least sensitive to sample size under correct model and Q-matrix specifications, and performs the best in power. Sample size is found to be the most influential factor on model-fit index values. Consequences of selecting inaccurate model and Q-matrix in classification accuracy of attribute mastery are also evaluated.

Small-Sample DIF Estimation Using SIBTEST, Cochran’s Z, and Log-Linear Smoothing

Minimum sample sizes of about 200 to 250 per group are often recommended for differential item functioning (DIF) analyses. However, there are times when sample sizes for one or both groups of interest are smaller than 200 due to practical constraints. This study attempts to examine the performance of Simultaneous Item Bias Test (SIBTEST), Cochran’s Z test, and log-linear smoothing with these methods in DIF detection accuracy at a number of small-sample and ability distribution combinations. Effects of item parameters and DIF magnitudes are also investigated. Results show that when ability distributions between groups are identical, Type I error for these DIF methods can be adequately controlled at all sample sizes, and their power to detect a large amount of unidirectional DIF can be tolerably high (power > .6) when sample size is not too small (at least 100 per group). When ability distributions are different, Type I inflation is higher for easier items and larger sample sizes, and power depends on DIF direction. Log-linear smoothing with SIBTEST tends to lower both Type I error rate and power. The effect of smoothing with Cochran’s Z test is not as consistent. Implications of the findings are discussed.

The moderating effect of supervisor and coworker support for error management on service recovery performance and helping behaviors

Purpose The purpose of this paper is to amend and extend the emerging research that has utilized an employee-focused approach to examining the service recovery process. In doing so, the authors examine the influences of supervisor and coworker support for error management on two measures of employee service performance: service recovery performance and helping behaviors during service failure and recoveries. Specifically, this study examines the linear and non-linear interaction effects of supervisor and coworker support for error management on the outcome variables. Design/methodology/approach To examine the proposed relationships, the authors conducted a field study that utilized survey data from a sample of 243 restaurant employees and their immediate supervisors. Employee ratings of supervisor and coworker support for error management were matched with the data gathered for the two dependent variables (i.e. supervisory ratings of service recovery performance and helping behaviors). Structural equation modeling was used to examine the linear interaction effects on the outcome variables. To examine the non-linear interaction effects on the outcome variables the authors utilized polynomial regression and response surface modeling. Findings The results showed that the interaction effects of supervisor and coworker support for error management was significantly positively related to both service recovery performance and helping behaviors. In addition, an alternative analysis of the shape of the interaction effects using polynomial regression and response surface modeling showed that the moderating effects may be better conceptualized as non-linear. Originality/value These findings offer new insights about the roles and impact of various forms of support in the service recovery process. First, the current study focuses specifically on supervisor and coworker support for error management and the impact on employees’ service recovery performance and helping behaviors. Second, this research investigates the interaction effects of these two forms of support on service recovery performance and helping behaviors. Third, along with linear interaction effects, the current work examines non-linear interaction effects. These relationships examined in this study have not been tested before. Thus, the findings of this research make a unique contribution to research in service management. The findings of this study provide more prescriptive insights about the means to prevent and respond effectively to service errors.

Making Teamwork Work: Team Knowledge for Team Effectiveness

ABSTRACT This study examined the impact of two types of team knowledge on team effectiveness. The study assessed the impact of taskwork knowledge and teamwork knowledge on team satisfaction and performance. A longitudinal study was conducted with 27 service-management teams involving 178 students in a real-life restaurant setting. Teamwork knowledge was found to impact both team outcomes. Furthermore, team learning behavior was found to mediate the relationships between teamwork knowledge and team outcomes. Educators and managers should therefore ensure these types of knowledge are developed in teams along with learning behavior for maximum effectiveness.

General Test Overlap Control Improved Algorithm for CAT and CCT

This article proposed a new online test overlap control algorithm that is an improvement of Chen’s algorithm in controlling general test overlap rate for item pooling among a group of examinees. Chen’s algorithm is not very efficient in that not only item pooling between current examinee and prior examinees is controlled for but also item pooling between previous examinees, which would have been controlled for when they were current examinees. The proposed improvement increases efficiency by only considering item pooling between current and previous examinees, and its improved performance over Chen is demonstrated in a simulated computerized adaptive testing (CAT) environment. Moreover, the proposed algorithm is adapted for computerized classification testing (CCT) using the sequential probability ratio test procedure and is evaluated against some existing exposure control procedures. The proposed algorithm appears to work best in controlling general test overlap rate among the exposure control procedures examined without sacrificing much classification precision, though longer tests might be required for more stringent control of item pooling among larger groups. Given the capability of the proposed algorithm in controlling item pooling among a group of examinees of any size and its ease of implementation, it appears to be a good test overlap control method.