Deborah Bennett

A Comparison of Linear and Systems Thinking Approaches for Program Evaluation Illustrated Using the Indiana Interdisciplinary GK-12.

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.

The Effects of Computer-Administered Choice on Students with and without Characteristics of Attention-Deficit/Hyperactivity Disorder.

Providing choice to individual students with attention-deficit/hyperactivity disorder (ADHD) has previously been demonstrated (in single-subject research) to reduce undesirable behavior but has failed to demonstrate gains that are independent of task difficulty or interests/preferences. We examined choice of feedback type on a computerized math task for 9 students with symptoms of attention-deficit/hyperactivity disorder (ADHD) and 17 typical peers. A brief choice was sufficient to modulate behavior of students with ADHD, especially over time. In addition, spoken problems with positive visual feedback “normalized“ their performance. The educational implications of these findings were that even minimal curriculum options in general education could differentially improve the behavioral responses of students with ADHD.

An initial study of visuohaptic simulation of point-charge interactions

We report an initial study on the use of visuohaptic simulation in teaching STEM (Science, Technology, Engineering and Mathematics) topics, with particular emphasis on physics concepts related to the learning of nanotechnology at the undergraduate level. Visuohaptic simulations of point charges and their interactions were developed. Thirty-eight undergraduate students from a physics lab course were recruited to participate in an educational study. Half of the students were assigned to a visuohaptic (VH) group where they could see as well as feel interaction forces rendered with a Falcon force-feedback device. The other half of the students were assigned to a visual (V) group where they could interact with the same simulation via a computer mouse but did not feel any forces. Results from a 10-question content test showed that both groups benefited from the computer simulation. There were no statistical differences between the VH and V groups, presumably due to the relatively small number of participants who completed quantitative assessments. However, qualitative results from observations and interviews indicated that students in the VH groups were more motivated and engaged in the lab activities, reported more positive attitude towards learning, and felt more confident about their understanding and retention of knowledge. Our findings have been used to design a larger-scale study that will further investigate the use of visuohaptic technology in nanotechnology education.

Short-term memory of children with and without characteristics of attention deficit hyperactivity disorder

Abstract The effects of verbal interference and strategy instruction on recall of visual and auditory material were assessed in children with and without attention deficit/hyperactivity disorder (AD/HD). Children were randomly assigned to a strategy instruction or to a control group and (a) completed two sets (four administrations) of the Learning Efficiency Test (LET)-II, (b) a self-report measure of ‘task-unrelated thoughts (TUTs),’ and (c) an incidental sentence recall task. Recall of both visual and auditory material was greater for those children who received strategy instruction and when there was no added interference. Poor memory and failure to filter incoming stimulation were not supported as explanations of AD/HD under the conditions of this study.

The Development of a Computer-based Alternate Assessment System

This article describes the development of an alternate assessment system for students with moderate to severe disabilities. The system integrates what is known about effective feedback in classroom assessment with broader requirements for system accountability under the 1997 Amendments to the Individuals with Disabilities Education Act (Public Law 105-17). The alternate assessment provides teachers with customized computer software and documentation tools that are used to collect meaningful student performance data from multiple sources on an ongoing basis. The assessment system evolved through an extensive committee process during which input from a variety of stakeholders was sought. The development process is summarized and the computer-based rating and documentation system is described, including the technical adequacy of the resulting alternate assessment.

Development of an Integrated Computer-Based Assessment System for Early Childhood Programs

This article describes the development of a computer-based assessment system for children in early childhood programs, The Indiana Assessment System of Educational Proficiencies: Early Childhood (IASEP: EC). Skills in five developmental domains (i.e., cognitive, communication, social, sensory motor, and self-help) were selected and content validated using a representative stakeholder and expert panel process. Resulting indicators were included in assessment software and piloted with 40 teachers and 128 children between the ages of 3 ½ and 6 ½ years. Analysis of core ratings yielded four separate factors: language development, social development, sensory motor development, and self-help skills. The rotated factor structure was largely consistent with the initial domain structure. Correlations between IASEP: EC domain ratings and the Speed DIAL and Picture Naming Task suggest a strong positive relationship between the developmental constructs measured by each. Initial results would lend support for the reliability and validity of the IASEP: EC rating system. Implications for using teacher ratings with portfolio evidence as a means of providing comprehensive documentation of child growth are discussed.

Electronic Educational Data Security: System Analysis and Teacher Training

This article describes the analysis of data security needs for the Indiana Assessment System of Educational Proficiencies (IASEP), a computer-based rating and documentation system that is currently being implemented statewide for students with disabilities. Analysis of the system and the creation of security architecture to protect sensitive student data are outlined. Matching security controls with security risks and selecting appropriate security software, hardware, and user guidelines are also discussed. The importance of user error in managing student data is recognized, and a teacher training protocol is described.

Validity Evidence for a Computer-Based Alternate Assessment Instrument

This study investigated the validity of a statewide alternate assessment program, IASEP (Indiana Assessment System of Educational Proficiencies) by examining supporting profile patterns on the 100 core IASEP items of individuals with significant disabilities. Participants were 5,192 students ranging in age from 7-21 years with special education diagnoses of autism, mild cognitive disabilities, moderate cognitive disabilities, multiple disabilities, and severe cognitive disabilities. A profile analysis using a repeated-measures multivariate analysis of variance (MANOVA) was performed to examine profile patterns across the five disability groups and items in each domain. Results support existing profile patterns among disability groups, indicating that IASEP can be used as a model for an alternate assessment instrument.