Kerstin Carlson Le Floch

Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research

Improving the validity and reliability of surveys is a critical part of the response to the call for improved rigor of education research, policy analysis and evaluation. Too often we create inquiry tools without validating our measures against how respondents interpret our questions, and therefore collect data of questionable quality. The purpose of this article is to demonstrate how cognitive interviews can be a useful method for improving the reliability and validity of surveys used in education research. The cognitive interviews described in this article reveal the complex thought processes in which our respondents engaged as they discussed their responses to questions concerning standards, assessments and professional development, and the unanticipated interpretations of seemingly straightforward constructs. We present a number of examples to illustrate complexities that education researchers face in trying to design large-scale surveys to answer critical educational policy questions, and the substantive insights that can be gained from the use of cognitive interviews. The use of cognitive interviews in the development of surveys has been for the most part neglected by the educational research community, and we suggest that more attention be paid to this methodology as a way of improving survey research.

The Geoscience Pipeline: A Conceptual Framework

In order to assess the effectiveness of projects intended to increase the participation of members of traditionally underrepresented groups in geoscience careers, short-term indicators of “success” must be identified and developed. Our first step in identifying these indicators was the creation of a model of the science, technology, engineering, and math (STEM) career pipeline, based on a literature review of factors associated with STEM career choice in minority populations. To validate the appropriateness of this model for the geosciences, as well as to identify factors specific to geoscience career choice, we conducted a critical incident study and further refined our pipeline model. We used the model to determine the potential efficacy of different approaches that are being employed by geoscience diversity projects and to show how it can be used for determining the effectiveness of these projects.