S. Klinkenberg

Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation

In this paper we present a model for computerized adaptive practice and monitoring. This model is used in the Maths Garden, a web-based monitoring system, which includes a challenging web environment for children to practice arithmetic. Using a new item response model based on the Elo (1978) rating system and an explicit scoring rule, estimates of the ability of persons and the difficulty of items are updated with every answered item, allowing for on the fly item calibration. In the scoring rule both accuracy and response time are accounted for. Items are sampled with a mean success probability of .75, making the tasks challenging yet not too difficult. In a period of ten months our sample of 3648 children completed over 3.5 million arithmetic problems. The children completed about 33% of these problems outside school hours. Results show better measurement precision, high validity and reliability, high pupil satisfaction, and many interesting options for monitoring progress, diagnosing errors and analyzing development.

A Practical Procedure for the Construction and Reliability Analysis of Fixed Length Tests with Random Drawn Test Items

A procedure to construct valid and fair fixed-length tests with randomly drawn items from an item bank is described. The procedure provides guidelines for the set-up of a typical achievement test with regard to the number of items in the bank and the number of items for each position in a test. Further, a procedure is proposed to calculate the relative difficulty for individual tests and to correct the obtained score for each student based on the mean difficulty for all students and the particular test of a student. Also, two procedures are proposed for the problem to calculate the reliability of tests with randomly drawn items. The procedures use specific interpretations of regularly used methods to calculate Cronbach’s alpha and KR20 and the Spearman-Brown prediction formula. A simulation with R is presented to illustrate the accuracy of the calculation procedures and the effects on pass-fail decisions.

High Speed High Stakes Scoring Rule

In this paper we will present the results of a three year subsidized research project investigating the performance of a new scoring rule for digital assessment. The scoring rule incorporates response time and accuracy in an adaptive environment. The project aimed to assess the validity and reliability of the ability estimations generated with the new scoring rule. It was also assessed whether the scoring rule was vulnerable for individual differences. Results show a strong validity and reliability in several studies within different domains: e.g. math, statistics and chess. We found no individual differences in the performance of the HSHS scoring rule for risk taking behavior and performance anxiety, nor did we find any performance differences for gender.

The Role of Formative Assessment in a Blended Learning Course

Assessing the effectiveness of a course design in higher education is an almost unfeasible task. The practical inability to conduct randomised controlled trials in a natural setting limits teachers who want to interpret their design choices to non causal evaluation. With at least a moderate amount of common sense these evaluations could be useful in gathering insight into what works and what does not work in education. In our blended learning course we wanted to assess the role of formative assessment while also taking lecture attendance into account. There are certainly many confounds for which we cannot control. We found no effect of lecture attendance but formative assessments did predict a substantial amount of course outcome. This was probably due to not including the formative performance in the final grading and our emphasis in the course that to err is to be student. The validity of the formative assessment paves the way for diagnostic use and remedial teaching.