Melissa Dyehouse

A Comparison of Linear and Systems Thinking Approaches for Program Evaluation Illustrated Using the Indiana Interdisciplinary GK-12.

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.

The role of authenticity in design-based learning environments: The case of engineering education

The term authenticity is pervasive in the education literature in general and specifically in the design education and engineering education literature; yet, the construct is often used un-reflected and ill defined. The purpose of this paper is (1) to critically examine current conceptualizations of authenticity as principles to design learning modules and environments within engineering education and (2) to propose a systematically derived model of authenticity. The context of the project is toward pre-college engineering education yet findings are applicable across the lifespan of engineering education. A systematic literature review guided by procedures set forth by the Centre for Reviews and Dissemination was conducted in the engineering education literature to synthesize the findings. Based on an initial sample of papers (n = 36) a rubric was developed to identify authenticity and authentic experiences in engineering education. Using the developed rubric, a total pool of 1058 references was evaluated using the rubric with 88%-100% inter-rater reliability for each category of authenticity. A frequency analysis of references revealed that the majority of work is seen in undergraduate education, and only 14 instances of authenticity in engineering education appeared at the K-12 level. The proposed model of authenticity includes two additions to existing models introducing impact authenticity and value authenticity. The findings and the model are described. Implications include the use of different types of authenticity to provide more appropriate and promising principles for better design of engineering curricula and standards for curriculum developers and professional development providers, including more use of authenticity in the K-12 classroom.

The Development of a Systematic Coding System for Elementary Students' Drawings of Engineers

The Draw an Engineer Test (DAET) is a common measure of students’ perceptions of engineers. The coding systems currently used for K- 12 research are general rubrics or checklists to capture the images presented in the drawing, which leave out some of the richness of students’ perceptions, currently only captured with an accompanying student interview. The purpose of this study is to build a reliable coding system, which first establishes an inventory of pictorial elements irrespective of their potential relationship with engineering and second captures aspects of students’ engineering perceptions inductively (from the ground up) while at the same time incorporating categories from previous research. The coding system will be used to help researchers understand how young students’ perceptions of engineering, engineers, and the work of engineers evolve and are impacted by interventions. The longterm goal of this project is to create a standalone measure that can be broadly applied to diverse populations, and to create a large multiinstitution student database, with both K- 12 and university populations represented. This database would provide a rich dataset for better understanding common misconceptions about engineering and thus enabling the development of methods to address them.

Examining the relationship between resistance to change and undergraduate engineering students’ environmental knowledge and attitudes

Engineering professional associations identified environmental sustainability as a key responsibility of the educated engineer. Data from national surveys of the general public demonstrate low environmental knowledge levels and a high level of resistance when it comes to environmental behavior. The purpose of this study was to examine the relationship between first-year engineering students’ environmental knowledge and attitudes and resistance to change (RtC). The authors administered instruments measuring RtC and environmental knowledge and attitudes to three groups (n = 3169) of first-year engineering students in the fall semesters of 2008–2010. Students showed the highest mean scores on the Cognitive Rigidity subscale of RtC. Overall, weak, negative correlations were found between most RtC subscales and environmental knowledge/attitudes, meaning that students with higher RtC score lower overall on environmental knowledge and consider sustainable development less important. Findings can aid researchers and curriculum designers in understanding students’ knowledge levels and the relationship between RtC factors and knowledge/attitudes.

Validity Evidence for a Computer-Based Alternate Assessment Instrument

This study investigated the validity of a statewide alternate assessment program, IASEP (Indiana Assessment System of Educational Proficiencies) by examining supporting profile patterns on the 100 core IASEP items of individuals with significant disabilities. Participants were 5,192 students ranging in age from 7-21 years with special education diagnoses of autism, mild cognitive disabilities, moderate cognitive disabilities, multiple disabilities, and severe cognitive disabilities. A profile analysis using a repeated-measures multivariate analysis of variance (MANOVA) was performed to examine profile patterns across the five disability groups and items in each domain. Results support existing profile patterns among disability groups, indicating that IASEP can be used as a model for an alternate assessment instrument.