Sally Jordan

e‐Assessment for learning? The potential of short‐answer free‐text questions with tailored feedback

A natural language based system has been used to author and mark short-answer free-text assessment tasks. Students attempt the questions online and are given tailored and relatively detailed feedback on incorrect and incomplete responses, and have the opportunity to repeat the task immediately so as to learn from the feedback provided. The answer matching has been developed in the light of student responses to the questions. A small number of the questions are now in low-stakes summative use, alongside other e-assessment tasks and tutor-marked assignments, to give students instantaneous feedback on constructed response items, to help them to monitor their progress and to encourage dialogue with their tutor. The answer matching has been demonstrated to be of similar or greater accuracy than specialist human markers. Students have been observed attempting the questions and have been seen to respond in differing ways to both the questions themselves and the feedback provided. We discuss features of appropriate items for assessment of this type.

A comparison of human and computer marking of short free-text student responses

The computer marking of short-answer free-text responses of around a sentence in length has been found to be at least as good as that of six human markers. The marking accuracy of three separate computerised systems has been compared, one system (Intelligent Assessment Technologies FreeText Author) is based on computational linguistics whilst two (Regular Expressions and OpenMark) are based on the algorithmic manipulation of keywords. In all three cases, the development of high-quality response matching has been achieved by the use of real student responses to developmental versions of the questions and FreeText Author and OpenMark have been found to produce marking of broadly similar accuracy. Reasons for lack of accuracy in human marking and in each of the computer systems are discussed.

Interactive screen experiments: innovative virtual laboratories for distance learners

The desirability and value of laboratory work for physics students is a well-established principle and issues arise where students are inherently remote from their host institution, as is the case for the UKs Open University. In this paper, we present developments from the Physics Innovations Centre for Excellence in Teaching and Learning (piCETL) in the production and technology of the virtual laboratory resources, interactive screen experiments, and the benefits and drawbacks of such resources. We also explore the motivations behind current implementation of interactive screen experiments and examine evaluation strategies and outcomes through a series of case studies.

Using Interactive Computer-Based Assessment to Support Beginning Distance Learners of Science.

Interactive computer‐marked assignments (iCMAs) are in use on a range of Science Faculty level 1 modules at the Open University UK, alongside tutor‐marked assignments, for summative and formative purposes. They are also used diagnostically to help prospective students to make an appropriate choice of starting point. iCMAs have been very well received by students, but minor problems can easily lead to a loss of confidence. Students engage more with the questions when they carry some weighting and most students feel that their mark for iCMA questions should count towards their overall course score. Two case studies are presented, showing how evaluation has led to further improvements at both the question and whole assignment levels.

Using concept inventories to measure understanding

Abstract Measuring understanding is notoriously difficult. Indeed, in formulating learning outcomes the word ‘understanding’ is usually avoided, but in the sciences, developing understanding is one of the main aims of instruction. Scientific knowledge is factual, having been tested against empirical observation and experimentation, but knowledge of facts alone is not enough. There are also models and theories containing complex ideas and interrelationships that must be understood, and considerable attention has been devoted across a range of scientific disciplines to measuring understanding. This case study will focus on one of the main tools employed: the concept inventory and in particular the Force Concept Inventory (FCI). The success of concept inventories in physics has spawned concept inventories in chemistry, biology, astronomy, materials science and maths, to name a few. We focus here on the FCI and ask how useful concept inventories are for evaluating learning gains. Finally, we report on recent work by the authors to extend conceptual testing beyond the multiple-choice format.

Developing academics' assessment practices in open, distance and e-learning: an institutional change agenda

AbstractUniversities across the globe are attempting to change assessment practice to address challenges in student engagement and achievement. Integral to this objective are strategies to develop academics’ assessment practice. These frequently focus on attendance at formal Continuous Professional Development events and/or implementation of institutional blueprints. This editorial article uses a case study from the Open University (UK) to explore an alternative ‘communities of practice’ approach to the improvement of assessment arguing that academics’ professional expertise is best deepened through participation in authentic activities of teaching and scholarship. The discussion identifies what is involved in such an approach including the role of an enabling principles-based framework, the constraints on implementation and the implications for HE leaders.

Impact of scaffolding and question structure on the gender gap

We address previous hypotheses about possible factors influencing the gender gap in attainment in physics. Specifically, previous studies claim that male advantage may arise from multiple-choice style questions, and that scaffolding may preferentially benefit female students. We claim that female students are not disadvantaged by multiple-choice style questions, and also present some alternative conclusions surrounding the scaffolding hypothesis. By taking both student attainment level and the degree of question scaffolding into account, we identify questions which exhibit real bias in favour of male students. We find that both multi-dimensional context and use of diagrams are common elements of such questions.

Calculating the Random Guess Score of Multiple-Response and Matching Test Items

For achievement tests, the guess score is often used as a baseline for the lowest possible grade for score to grade transformations and setting the cut scores. For test item types such as multiple-response, matching and drag-and-drop, determining the guess score requires more elaborate calculations than the more straightforward calculation of the guess score for True-False and multiple-choice test item formats. For various variants of multiple-response and matching types with respect to dichotomous and polytomous scoring, methods for determining the guess score are presented and illustrated with practical applications. The implications for theory and practice are discussed.

Nightclubs, beetles, spiders and maps

Suppose that you are at a nightclub and that the women will only dance with men whom they have met before. Assuming that there are equal numbers of men and women, what are the chances of pairing everyone off? It turns out that all the women will get to dance if, and only if, the women in any group have met at least as many of the men as there are women in that group.