S. Draaijer

The Emergence of Large-Scale Computer Assisted Summative Examination Facilities in Higher Education

A case study is presented of VU University Amsterdam where a dedicated large-scale CAA examination facility was established. In the facility, 385 students can take an exam concurrently. The case study describes the change factors and processes leading up to the decision by the institution to establish the facility, the start-up of the facility, the foreseen optimization of the use of the facility, threats to the sustainability of the facility and possible future developments. Comparisons are made with large-scale CAA practice at the University of Southampton in the UK. The conclusions are that some specific coincidental situations may be needed to support the decision by senior management to establish such a facility. Long-term sustainability of the dedicated facility is expected to be dependent on the payment structure, the scheduling possibilities and on the educational and assessment benefits that can be achieved. Hybrid models of dedicated facilities and regular computer rooms for CAA seem likely to be adopted, thus balancing cost and benefits. The case shows that sustained effort in building up expertise and momentum are needed to result in viable and sustainable CAA exam facilities.

Online Proctoring for Remote Examination: A State of Play in Higher Education in the EU

We present some preliminary findings of the Erasmus+ KA2 Strategic Partnership project “Online Proctoring for Remote Examination” (OP4RE). OP4RE aims to develop, implement and disseminate up to par practices for remote examination procedures. More specifically, OP4RE strives to develop guidelines and minimum standards for the secure, legal, fair and trustworthy administration of exams in a remote location away from physical exam rooms in a European context. We present findings and issues regarding security, cheating prevention and deterrence, privacy and data protections as well as practical implementation.

Formative Quizzing and Learning Performance in Dutch First-Year Higher Education Students

In this research paper, a cross-sectional study into the effects of formative quizzing in higher education and its relation to learning performance is presented. For the current study, six online Formative Quizzing modules, consisting of texts, graphics and video clips followed by two or more test questions to reiterate the material, were provided to students. Students could not earn marks and were free to use the material, but were informed that in the final examination, questions relating to the material would be asked. Data analysis showed that students who completed all six modules had a statistical significant higher chance to score better on the final examination. This was true for high achieving students, but also, and even stronger, for low achieving students. The results therefore show in this particular set-up a potential causal relationship of online formative quizzing on learning performance in higher education.

A Practical Procedure for the Construction and Reliability Analysis of Fixed Length Tests with Random Drawn Test Items

A procedure to construct valid and fair fixed-length tests with randomly drawn items from an item bank is described. The procedure provides guidelines for the set-up of a typical achievement test with regard to the number of items in the bank and the number of items for each position in a test. Further, a procedure is proposed to calculate the relative difficulty for individual tests and to correct the obtained score for each student based on the mean difficulty for all students and the particular test of a student. Also, two procedures are proposed for the problem to calculate the reliability of tests with randomly drawn items. The procedures use specific interpretations of regularly used methods to calculate Cronbach’s alpha and KR20 and the Spearman-Brown prediction formula. A simulation with R is presented to illustrate the accuracy of the calculation procedures and the effects on pass-fail decisions.

Calculating the Random Guess Score of Multiple-Response and Matching Test Items

For achievement tests, the guess score is often used as a baseline for the lowest possible grade for score to grade transformations and setting the cut scores. For test item types such as multiple-response, matching and drag-and-drop, determining the guess score requires more elaborate calculations than the more straightforward calculation of the guess score for True-False and multiple-choice test item formats. For various variants of multiple-response and matching types with respect to dichotomous and polytomous scoring, methods for determining the guess score are presented and illustrated with practical applications. The implications for theory and practice are discussed.

Tentamens met de computer: een vergelijking van meerkeuzevragen en alternatieve vraagvormen

In het medisch onderwijs wordt steeds meer gebruik gemaakt van beeldschermtoetsing en alternatieve vraagvormen. In een speciaal geprepareerd beeldschermtentamen werden traditionele meerkeuzevragen en alternatieve vraagvormen aan studenten voorgelegd om deze voor wat betreft scores en slaagpercentages met elkaar te kunnen vergelijken. De meerkeuzevragen dienden tevens als referentie. Voor het tentamen werden drie alternatieve vraag - vormen gebruikt: Drag-and-dropvragen, Multiple Responsevragen en Matchingvragen. De slaaggrens werd bepaald volgens het model waarbij de helft van het aantal punten gescoord moet worden nadat een correctie voor het scoren op basis van de raadkans is toegepast. Deze methode wordt in het hoger onderwijs op grote schaal gebruikt. De resultaten laten zien dat de alternatieve vraagvormen vele mogelijkheden bieden en scores opleveren in dezelfde ordegrootte als meerkeuzevragen. De alternatieve vraagvormen resulteren echter wel in verschillende slaagpercentages. (Draaijer S, Bos GC van den. Tentamens met de computer: een vergelijking van meerkeuzevragen en alternatieve vraagvormen. Tijdschrift voor Medisch Onderwijs 2009;28(1):13-21.)