Suki Goodman

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.

The need for theory evaluation in global citizenship programmes: The case of the GCSA programme

Many education programmes lack a documented programme theory. This is a problem for programme planners and evaluators as the ability to measure programme success is grounded in the plausibility of the programmes underlying causal logic. Where the programme theory has not been documented, conducting a theory evaluation offers a foundational evaluation step as it gives an indication of whether the theory behind a programme is sound. This paper presents a case of a theory evaluation of a Global Citizenship programme at a top-ranking university in South Africa, subsequently called the GCSA Programme. This evaluation highlights the need for documented programme theory in global citizenship-type programmes for future programme development. An articulated programme theory produced for the GCSA Programme, analysed against the available social science literature, indicated it is comparable to other such programmes in terms of its overarching framework. What the research found is that most other global citizenship programmes do not have an articulated programme theory. These programmes also do not explicitly link their specific activities to their intended outcomes, making demonstrating impact impossible. In conclusion, we argue that taking a theory-based approach can strengthen and enable outcome evaluations in global citizenship programmes.