Hunter Gehlbach

Developing questionnaires for educational research: AMEE Guide No. 87

Abstract In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.

Measure twice, cut down error: A process for enhancing the validity of survey scales

For years psychologists across many subfields have undertaken the formidable challenge of designing survey scales to assess attitudes, opinions, and behaviors. Correspondingly, scholars have written much to guide researchers in this undertaking. Yet, many new scales violate established best practices in survey design, suggesting the need for a new approach to designing surveys. This article presents 6 steps to facilitate the construction of questionnaire scales. Unlike previous processes, this one front loads input from other academics and potential respondents in the item-development and revision phase with the goal of achieving credibility across both populations. Specifically, the article describes how (a) a literature review and (b) focus group–interview data can be (c) synthesized into a comprehensive list to facilitate (d) the development of items. Next, survey designers can subject the items to (e) an expert review and (f) cognitive pretesting before executing a pilot test.

Forecasting student achievement in MOOCs with natural language processing

Student intention and motivation are among the strongest predictors of persistence and completion in Massive Open Online Courses (MOOCs), but these factors are typically measured through fixed-response items that constrain student expression. We use natural language processing techniques to evaluate whether text analysis of open responses questions about motivation and utility value can offer additional capacity to predict persistence and completion over and above information obtained from fixed-response items. Compared to simple benchmarks based on demographics, we find that a machine learning prediction model can learn from unstructured text to predict which students will complete an online course. We show that the model performs well out-of-sample, compared to a standard array of demographics. These results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning.

Teaching social perspective taking: how educators might learn from the Army

Frequently and accurately discerning others’ thoughts and feelings is associated with multiple valued educational outcomes across an array of settings. Despite its foundational role in social interactions, it is unclear whether individuals can be taught to improve their social perspective taking capacities. This experiment assesses whether a curriculum taught to US Army personnel (N = 116) improved their social perspective taking prior to deployment. Results showed that participants improved their social perspective taking in three ways: through more accurately detecting biases in others, by generating more initial hypotheses to explain others’ behaviours, and by adapting their hypotheses in the face of new evidence. The curriculum did not affect participants’ perspective taking accuracy on a video measure. We discuss these findings with respect to their implications for other learning environments.

Making Social Studies Social: Engaging Students through Different Forms of Social Perspective Taking.

People are intrinsically motivated to connect to others socially. One of the most important mechanisms in fostering social relationships is social perspective taking (SPT)—the capacity to discern the thoughts and feelings of others. Thus, students in social studies classrooms might be motivated to engage with their subject either through taking the perspectives of their peers in class (interpersonal SPT) or through taking the perspectives of the historical and cultural figures they are studying (academic SPT). This article first provides a theoretical overview of the contrasts and similarities between these two forms of SPT. Next, it describes 3 examples of how these 2 forms of SPT might be implemented in teaching social studies.

Seven Survey Sins

As pressure builds to assess students, teachers, and schools, educational practitioners and policy makers are increasingly looking toward student perception surveys as a promising means to collect high-quality, useful data. For instance, the widely cited Measures of Effective Teaching study lists student perception surveys as one of the three key measures of teachers’ efficacy and describes these measures as a source of potentially valuable feedback for teachers (Cantrell & Kane, 2013). When one factors in the low cost with their prospective utility in assessing teachers and fostering teaching effectiveness, it seems clear that surveying students will increase dramatically in the coming years. Within this context, researchers who survey early adolescents must navigate the confluence of three tensions. First, responding to survey items requires an array of cognitive skills that early adolescents are still mastering (Downer, Stuhlman, Schweig, Martínez, & Ruzek, 2014). Second, unlike national, public opinion surveys, school-based surveys need to provide accurate data for small samples. For instance, a middle school teacher may have 100 students divided between her sixth, seventh, and eighth-grade science classes; an elementary school teacher may only have 20 students total. Third, for the full promise of surveys to be realized as a tool for improving schools, the survey scales need to be practitioner friendly: short, easy to administer, and straightforward to interpret (Hamre & Cappella, 2015, Kosovich, Hulleman, Barron, & Getty, 2014).

Learning to Walk in Another's Shoes.

Despite the enthusiasm around social-emotional learning, the vast number of skills, dispositions, and attitudes we hope to infuse into students will overwhelm even the most ambitious schools. However, a single core capacity underlies a great many social-emotional learning outcomes: social perspective taking. Recent research on this process of “reading” others — figuring out their thoughts, feelings, and motivations — suggests that we now know enough to teach this capacity to students. By helping youth engage in this process regularly as “detectives” rather than “judges” and providing them with feedback, a constellation of social-emotional learnings will blossom.

Questionnaires as interventions: can taking a survey increase teachers’ openness to student feedback surveys?

Abstract Administrators often struggle in getting teachers to trust their school’s evaluation practices – a necessity if teachers are to learn from the feedback they receive. We attempted to bolster teachers’ support for receiving evaluative feedback from a particularly controversial source: student-perception surveys. For our intervention, we took one of two approaches to asking 309 teachers how they felt about students evaluating their teaching practice. Control participants responded only to core questions regarding their attitudes towards student-perception surveys. Meanwhile, treatment participants were first asked whether teachers should evaluate administrators in performance reviews and were then asked the core items about student-perception surveys. Congruent with cognitive dissonance theory, this juxtaposition of questions bolstered treatment teachers’ support for using student surveys in teacher evaluations relative to the control group. We discuss the implications of these findings with respect to increasing teacher openness to alternative evaluation approaches, and consider whether surveys show promise as a vehicle for delivering interventions.

How an Artificially Intelligent Virtual Assistant Helps Students Navigate the Road to College

Deep reinforcement learning using convolutional neural networks is the technology behind autonomous vehicles. Could this same technology facilitate the road to college? During the summer between high school and college, college-related tasks that students must navigate can hinder successful matriculation. We employ conversational artificial intelligence (AI) to efficiently support thousands of would-be college freshmen by providing personalized, text message–based outreach and guidance for each task where they needed support. We implemented and tested this system through a field experiment with Georgia State University (GSU). GSU-committed students assigned to treatment exhibited greater success with pre-enrollment requirements and were 3.3 percentage points more likely to enroll on time. Enrollment impacts are comparable to those in prior interventions but with substantially reduced burden on university staff. Given the capacity for AI to learn over time, this intervention has promise for scaling personalized college transition guidance.

Assessing Parent Perceptions of School Fit: The Development and Measurement Qualities of a Survey Scale

Students whose school environment fits their developmental needs also typically experience academic success and increased motivation. Most investigations of school fit, which focus on teachers’ and students’ perceptions, have found a general decline in fit across the transition from elementary to middle school. However, little research has examined the school-child fit from the parent perspective. In this article, we first detail the development process behind a new survey measure of parent perceptions of school fit. Second, using three online panel-based samples of parents from across the country (n1 = 323, n2 = 188, n3 = 1,033), we evaluate the scales measurement properties and conduct exploratory analyses examining grade-level and income-based differences on reported school fit. Finally, in line with previous research, we find that parents of middle school children perceived statistically significantly worse fit than parents of elementary school children. Among parents of high school students, we found that, on average, high-income parents perceive statistically significantly better fit than low-income parents.