Yanyan Sheng

Is Coefficient Alpha Robust to Non-Normal Data?

Coefficient alpha has been a widely used measure by which internal consistency reliability is assessed. In addition to essential tau-equivalence and uncorrelated errors, normality has been noted as another important assumption for alpha. Earlier work on evaluating this assumption considered either exclusively non-normal error score distributions, or limited conditions. In view of this and the availability of advanced methods for generating univariate non-normal data, Monte Carlo simulations were conducted to show that non-normal distributions for true or error scores do create problems for using alpha to estimate the internal consistency reliability. The sample coefficient alpha is affected by leptokurtic true score distributions, or skewed and/or kurtotic error score distributions. Increased sample sizes, not test lengths, help improve the accuracy, bias, or precision of using it with non-normal data.

Bayesian Multidimensional IRT Models With a Hierarchical Structure

As item response models gain increased popularity in large-scale educational and measurement testing situations, many studies have been conducted on the development and applications of unidimensional and multidimensional models. Recently, attention has been paid to IRT-based models with an overall ability dimension underlying several ability dimensions specific for individual test items, where the focus is mainly on models with dichotomous latent traits. The purpose of this study is to propose such models with continuous latent traits under the Bayesian framework. The proposed models are further compared with the conventional IRT models using Bayesian model choice techniques. The results from simulation studies as well as actual data suggest that (a) such models can be developed; (b) compared with the unidimensional IRT model, the proposed models better describe the actual data; and (c) the use of the proposed IRT models and the multiunidimensional model should be based on different beliefs about the underlying dimensional structure of a test.

Comparing Multiunidimensional and Unidimensional Item Response Theory Models

For tests consisting of multiple subtests, unidimensional item response theory (IRT) models apply when the subtests are known to measure a common underlying ability. However, in many instances, due to the lack of a satisfactory index for assessing the dimensionality assumption, the test structure is not clear. A more general IRT model, the multiunidimensional model, is more flexible and efficient in various test situations. This article compares these two classes of normal ogive two-parameter models and shows that the multiunidimensional model offers a better way to represent test situations not realized in unidimensional models.

Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics

The behavioral differences between expert and novice performance is a well-studied area in training literature. Advances in technology have made it possible to trace players’ actions and behaviors within an online gaming environment as user-generated data for performance assessment. In this study, we introduce the use of string similarity to differentiate likely-experts from a group of unknown performers (mixture of novices and experts) based on how similar their in-game actions are to that of experts. Our findings indicate that string similarity is viable as an empirical assessment method to differentiate likely-experts from novices and potentially useful as the first performance metric for Serious Games Analytics (SEGA).

Serious Games Analytics: Theoretical Framework

“Serious Games” is a unique industry that is concerned with the training/learning performance assessment of its clients. It is one of three digital technology industries (along with digital games, and online learning) that are rapidly advancing into the arena of analytics. The analytics from these industries all came from the tracing of user-generated data as they interacted with the systems, but differed from one another in the primary purposes for such analytics. For example, the purpose of game analytics is to support the growth of digital (entertainment) games, while that of learning analytics is to support the online learning industries. Although some game and learning analytics can indeed be used in serious games, they lack specific metrics and methods that outline the effectiveness of serious games—an important feature to stakeholders. Serious Games Analytics need to provide (actionable) insights that are of values to the stakeholders—specific strategies/policies to improve the serious games, and to (re)train or remediate play-learners for performance improvement. Since the performance metrics from one industry are unlikely to transfer well into another industry, those that are optimal for use in the Serious Games industry must be properly identified as Serious Games Analytics—to properly measure, assess, and improve performance with serious games.

A Gibbs Sampler for the Multidimensional Item Response Model

Current procedures for estimating compensatory multidimensional item response theory (MIRT) models using Markov chain Monte Carlo (MCMC) techniques are inadequate in that they do not directly model the interrelationship between latent traits. This limits the implementation of the model in various applications and further prevents the development of other types of IRT models that offer advantages not realized in existing models. In view of this, an MCMC algorithm is proposed for MIRT models so that the actual latent structure is directly modeled. It is demonstrated that the algorithm performs well in modeling parameters as well as intertrait correlations and that the MIRT model can be used to explore the relative importance of a latent trait in answering each test item.

Measuring Expert Performance for Serious Games Analytics: From Data to Insights

Advances in technology have made it possible to trace players’ actions and behaviors (as user-generated data) within online serious gaming environments for performance measurement and improvement purposes. Instead of a Black box approach (such as pretest/posttest), we can approach serious games as a White box, assessing performance of play-learners by manipulating the performance variables directly. In this chapter, we describe the processes to obtain user-generated gameplay data in situ using serious games for training—i.e., data tracing, cleaning, mining, and visualization. We also examine ways to differentiate expert-novice performances in serious games, including behavior profiling. We introduce a new Expertise Performance Index, based on string similarities that take into account the “course of actions” chosen by experts and compare that to those of the novices. The Expertise Performance Index can be useful as a metric for serious games analytics because it can rank play-learners according to their competency levels in the serious games.

A Comparison of Estimation Methods for a Multi-unidimensional Graded Response IRT Model

This study compared several parameter estimation methods for multi-unidimensional graded response models using their corresponding statistical software programs and packages. Specifically, we compared two marginal maximum likelihood (MML) approaches (Bock-Aitkin expectation-maximum algorithm, adaptive quadrature approach), four fully Bayesian algorithms (Gibbs sampling, Metropolis-Hastings, Hastings-within-Gibbs, blocked Metropolis), and the Metropolis-Hastings Robbins-Monro (MHRM) algorithm via the use of IRTPRO, BMIRT, and MATLAB. Simulation results suggested that, when the intertrait correlation was low, these estimation methods provided similar results. However, if the dimensions were moderately or highly correlated, Hastings-within-Gibbs had an overall better parameter recovery of item discrimination and intertrait correlation parameters. The performances of these estimation methods with different sample sizes and test lengths are also discussed.

A GPU-Based Gibbs Sampler for a Unidimensional IRT Model

Item response theory (IRT) is a popular approach used for addressing large-scale statistical problems in psychometrics as well as in other fields. The fully Bayesian approach for estimating IRT models is usually memory and computationally expensive due to the large number of iterations. This limits the use of the procedure in many applications. In an effort to overcome such restraint, previous studies focused on utilizing the message passing interface (MPI) in a distributed memory-based Linux cluster to achieve certain speedups. However, given the high data dependencies in a single Markov chain for IRT models, the communication overhead rapidly grows as the number of cluster nodes increases. This makes it difficult to further improve the performance under such a parallel framework. This study aims to tackle the problem using massive core-based graphic processing units (GPU), which is practical, cost-effective, and convenient in actual applications. The performance comparisons among serial CPU, MPI, and compute unified device architecture (CUDA) programs demonstrate that the CUDA GPU approach has many advantages over the CPU-based approach and therefore is preferred.

High Performance Gibbs Sampling for IRT Models Using Row-Wise Decomposition

Item response theory (IRT) is a popular approach used for addressing statistical problems in psychometrics as well as in other fields. The fully Bayesian approach for estimating IRT models is computationally expensive. This limits the use of the procedure in real applications. In an effort to reduce the execution time, a previous study shows that high performance computing provides a solution by achieving a considerable speedup via the use of multiple processors. Given the high data dependencies in a single Markov chain for IRT models, it is not possible to avoid communication overhead among processors. This study is to reduce communication overhead via the use of a row-wise decomposition scheme. The results suggest that the proposed approach increased the speedup and the efficiency for each implementation while minimizing the cost and the total overhead. This further sheds light on developing high performance Gibbs samplers for more complicated IRT models.