Katherine Ely

Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure?

We conducted a meta-analysis to clarify the construct validity of self-assessments of knowledge in education and workplace training. Self-assessment’s strongest correlations were with motivation and satisfaction, two affective evaluation outcomes. The relationship between self-assessment and cognitive learning was moderate. Even under conditions that optimized the self-assessment–cognitive learning relationship (e.g., when learners practiced self-assessing and received feedback on their self-assessments), the relationship was still weaker than the self-assessment–motivation relationship. We also examined how researchers interpreted self-assessed knowledge, and discovered that nearly a third of evaluation studies interpreted self-assessed knowledge data as evidence of cognitive learning. Based on these findings, we offer recommendations for evaluation practice that involve a more limited role for self-assessment. ........................................................................................................................................................................

Sometimes You Need a Reminder: The Effects of Prompting Self-Regulation on Regulatory Processes, Learning, and Attrition

Prompting self-regulation involves asking trainees reflective questions to stimulate self-regulatory engagement. Research has found positive effects for prompting self-regulation on learning, but a scarcity of evidence exists regarding whether self-regulatory processes mediate the effect of prompting self-regulation, whether the intervention reduces attrition, and the optimal timing of implementing the intervention. Using a longitudinal design, we found that prompting self-regulation throughout training increased learning and reduced attrition, relative to the control condition. Moreover, the effect on learning was fully mediated by time on task. The intervention also moderated the effect of learning on subsequent self-regulatory activity and attrition. Learning performance had less of a positive effect on subsequent self-regulatory activity and less of a negative effect on subsequent attrition when trainees were prompted to self-regulate. These results highlight the importance of adopting a longitudinal design to examine how self-regulatory interventions affect the cyclical relationships among self-regulatory processes, learning, and attrition.

The effects of technical difficulties on learning and attrition during online training.

Although online instruction has many potential benefits, technical difficulties are one drawback to the increased use of this medium. A repeated measures design was used to examine the effect that technical difficulties have on learning and attrition from voluntary online training. Adult learners (N = 530) were recruited online and volunteered to participate in a 4-hr training program on using computer spreadsheets. Technical difficulties were inserted in some of the training modules in the form of error messages. Using multilevel modeling, the results indicated that the presence of these technical difficulties impaired learning, such that test scores were lower in modules where trainees encountered technical difficulties than in modules where they did not encounter technical difficulties. Furthermore, the effect on learning was greater among trainees who eventually withdrew from the course than among trainees who completed the course. With regards to attrition, pretraining motivation provided a buffer against dropping out, especially when trainees encountered technical difficulties. Learning also predicted attrition from the subsequent module, such that attrition was higher among trainees with low test scores in the previous module. The current study disentangles some of the implications of technical difficulties and suggests that organizations should provide trainees with the technical support required to overcome technical difficulties in training. Furthermore, the findings contribute to our theoretical understanding of the implications of interruptions on performance in online training.