Liesbeth Baartman

A model for programmatic assessment fit for purpose

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

Facilitating Evaluations of Innovative, Competence-Based Assessments: Creating Understanding and Involving Multiple Stakeholders.

Schools are held more responsible for evaluating, quality assuring and improving their student assessments. Teachers lack of understanding of new, competence-based assessments as well as the lack of key stakeholders involvement, hamper effective and efficient self-evaluations by teachers of innovative, competence-based assessments (CBAs). While evaluating two CBAs in Agricultural Vocational Education and Training institutions, two interventions in the evaluation process aimed to tackle these problems were examined: (1) starting with explicating the CBA in the teacher team using a concrete explication format and (2) qualitatively involving key stakeholders (i.e., teachers, students and employers) in the evaluation of the CBA through mixed-group interviews. Quantitative and qualitative analysis, as well as stakeholders perceptions are used to find indications for the added value of these interventions for evaluation and further improvement of the CBAs. Results show that external facilitation is needed to make both interventions work. However, under this condition, explicating the CBA led to more complete, concrete and shared understandings of the actual CBA among teachers and mixed-groups interviews resulted in more concrete and elaborate evaluations of the CBAs quality and more ideas for improvement. Both interventions can facilitate building up elaborate, more valid and concrete arguments for CBA quality in self-evaluations, certainly in the case of evaluating innovative assessments. Lessons learned will provide guidelines for incorporating the interventions into other evaluations of innovative programs.

Feedback Dialogues That Stimulate Students' Reflective Thinking

How can feedback dialogues stimulate students reflective thinking? This study aims to investigate: (1) the effects of feedback dialogues between teachers and students on students perceptions of teacher feedback and (2) the relation between features of feedback dialogues and students thinking activities as part of reflective thinking. A quasi-experiment was carried out in which 72 secondary education students received written teacher feedback. Half of them had an additional feedback dialogue with their teacher. The latter group perceived teachers feedback as more useful than the group without a dialogue. The feedback dialogues could be clustered in 2 groups that varied in the number of segments containing teacher and student interaction. The number of segments containing interaction was positively correlated with students use of thinking activities.

Self-evaluation of assessment programs : a cross-case analysis

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and assessment, and an innovative school with extensive experience. The self-evaluation was based on 12 quality criteria for CAPs, including both validity and reliability, and criteria stressing the importance of the formative function of assessment, such as meaningfulness and educational consequences. In each school, teachers, management and examination board participated. Results show that the two schools use different approaches to assure assessment quality. The innovative school seems to be more aware of its own strengths and weaknesses, to have a more positive attitude towards teachers, students, and educational innovations, and to explicitly involve stakeholders (i.e., teachers, students, and the work field) in their assessments. This school also had a more explicit vision of the goal of competence-based education and could design its assessments in accordance with these goals.

Exploring the role of assessment criteria during teachers’ collaborative judgement processes of students’ portfolios

Student portfolios are increasingly used for assessing student competences in higher education, but results about the construct validity of portfolio assessment are mixed. A prerequisite for construct validity is that the portfolio assessment is based on relevant portfolio content. Assessment criteria, are often used to enhance this condition. This study aims to identify whether assessment criteria can improve content, argumentation and communication during teacher moderation while judging student portfolios. Six teachers scored 32 student portfolios in dyads with and without assessment criteria. Their judgement processes were qualitatively analysed. Results indicated that the quality of their judgement processes was low, since teachers based their judgements mainly on their own personal opinion and less on evidence found in the portfolio. Teachers barely paid attention to quality checks and easily agreed with each other. When teachers used assessment criteria, the quality of their judgements slightly improved. They based their judgements more on relevant evidence, used less personal experiences and more often checked the quality of their judgement processes. It is concluded that the quality of teacher portfolio judgement is low, and that the use of assessment criteria can enhance its quality.

Transfer, transitions and transformations of learning

This book explores one of the enduring issues in educational research and one of the challenges for formal education. That is, understanding the relationship between learning in one context, setting or time and a subsequent related learning experience or activity. The chapters in the book examine the issue drawing on existing theory as starting points but using each authors own research to push existing boundaries of what we know in terms of the ideas captured in the title of the book: transfer, transitions and transformations of learning. The chapters explore the issue through a range of approaches and settings including: possibilities for a concept-context approach to transfer, transfer between knowledge domains, transfer as an iterative process between contexts, transfer as boundary crossing between vocations, transfer as integration of theory and practice, transferring standards in assessment, representation in the transition from novice to expert, transformation of self through sustainability education, transforming identities of first year design and technology teachers and the role of implicit knowledge in understanding the relationship between declarative and procedural knowledge in the transition to expertise. This book should be of interest to teachers in schools and the adult education sector, research students, teacher educators, researchers and policy-makers who are involved in learning in, through or with technology.

Assessment in Dutch vocational education : Overview and tensions of the past 15 years

This chapter describes the developments in assessment practices in Dutch Vocational Education and Training (VET) in the past 15 years. Two developments have been particularly influential: (i) continuous changes in the national qualification structure describing the end goals of VET trajectories, and (ii) changes in the system of quality assurance of VET assessments. The goal of this chapter is to describe how VET institutions (re)developed their assessment practices to address these changes, through a combination of addressing changing policy and scientific research. Five tensions were identified that characterise the struggle around VET assessments: (1) new goals require new assessment methods, (2) assessment as a one-shot measurement versus a coherent programme appraisal, (3) the increased involvement of the labour market in VET, (4) securing a balance between governmental control and VET institutions’ responsibility in developing and quality assuring assessments, and (5) realising the balance between the formative and summative functions of assessment. The chapter concludes with the presentation of the Process Architecture Assessment, which is advanced as representing the state-of-the art in VET assessments. In the Process Architecture, the entire assessment process is described including the responsibilities of the different stakeholders.

The Power of Assessment in Teacher Education

is to increase student teachers’ involvement in assessmentby handing over the responsibilities of assessor from teacher educator to studentteacher. After all, one of the main responsibilities of teachers is to assess their pupilsin schools. For teacher educators, this means that they need to equip student teach-ers with assessor skills, like developing appropriate assessment criteria and givingadequate feedback on student performance.Because assessment is a delicate issue and many important educational decisionsare based on assessment outcomes, it is important to assure the quality of assess-ment. A

Enhancing learning through formative assessment and feedback – By Alastair Irons

The article reviews the book Enhancing Learning Through Formative Assessment and Feedback, by Alastair Irons.

Knowing everything from soup to dessert: an exploratory study to describe what characterises students’ vocational knowledge

Abstract Students’ vocational knowledge can be defined as all knowledge students require performing within occupational practice. In the context of vocational education and training, students’ vocational knowledge is often discussed from a perspective of either what should be taught and learned in schools or different kinds of knowledge students should gain in occupational practice. Much less focus is on students’ vocational knowledge itself. This exploratory in-depth study aims to describe what characterises students’ vocational knowledge. To explore students’ vocational knowledge, an analytic framework is used to describe vocational knowledge characteristics specifying: (1) occupation-specific knowledge components, and (2) qualities. Results show the framework provides a structure to gain insight into the nature and meaning of vocational knowledge, and is valuable to describe characteristics in terms of knowledge components such as technical procedures or the social and occupational environment, and qualities such as richness, complexity and specificity. Additionally, to use an existing framework – originally developed in the contexts of ICT and Social Work – its usefulness is explored in a new context, namely, the hospitality industry. Recommendations about the framework serving as a potential tool to support students’ learning processes are provided.