Molly H. Goldstein

Assessing idea fluency through the student design process

Engineering design is a complex activity for students to undertake and for instructors to assess. This research uses large learner data sets collected through automatic, unobtrusive logging of student actions in a CAD platform to address this difficulty in observing design behavior. We used a computer-aided design software that captured student design activities to investigate patterns of student design behaviors that are associated with idea fluency. We show how micro-level process data can be used to validate observations made from viewing the student design process through design replays. Students who engaged in high idea fluency showed evidence of fluency in both process data and design replays. Similar patterns were observed for low idea fluency students. There is great potential to investigate student design learning through system-collected data. Yet, how to justify the inferences made about students based on their process data is largely unexplored. Our results demonstrate how traditional forms of assessment data can be used to validate inferences made by process data. Implications of this work would be highly relevant to engineering educators as well as researchers who are interested in understanding the relationship between learner analytics and student learning.

The relationship between design reflectivity and conceptions of informed design among high school students

ABSTRACT Although reflection is a key behaviour of expert designers, it is often a challenging task for new designers. In addition, research on the reflectivity of student designers is limited. The purpose of this study is twofold: (1) to identify the levels of reflectivity while designing, and (2) to study the relationship between reflectivity and conceptions of informed design. We collected data from high school students engaged in an engineering design project. We developed a coding protocol to score levels of reflectivity in student reflections at three levels (low, medium, and high), and used the conceptions of design test to assess changes in student understanding of design activities. Using Wilcoxon signed-rank tests, we determined if students tended to select more ‘key’ design activities and fewer ‘distractors’ within each reflection group. We also performed McNemar’s tests to determine which specific design activities were important within each reflection group after the design project. The results show moderately reflective students had higher gains in understanding of informed design activities compared to those with high or low reflectivity. Results also indicate that different design activities became important for students within each of the three reflective groups. Implications from this research indicate that groups of students experience changing conceptions of design in different ways. An understanding of what students deem important while designing would better allow teachers to encourage behaviours that are like those of informed designers.

Traversing the Barriers to Using Big Data in Understating How High School Students Design

The context of this paper is a “large learner data” project that seeks to respond to existing challenges by introducing educational data mining and learning analytics into K-12 engineering design research. To probe deeply into student learning, we are developing and refining computational techniques to analyze large process analytics datasets generated through a CAD-based software, Energy3D, that logs design process data as students complete an assigned design challenge, such as a net-zero energy efficient building. We are combining these process analytics with demographic data and pre/post-tests of science and design knowledge. In this paper, we revisit three illustrative research cases to reflect on our experiences and lessons learned with navigating big data, generating useful data visualizations, and integrating process analytics with traditional performance assessment methods to relate design actions to knowledge and learning outcomes.