Collin Hitt

Comparing and Validating Measures of Character Skills: Findings from a Nationally Representative Sample

Though researchers now are aware of the potential importance of character skills, such as conscientiousness, grit, self-control, and a growth mindset, researchers struggle to find reliable measures of these skills. In this paper, we use data collected from the Understanding America Study, a nationally representative internet panel to study the validity of innovative measures of character skills based on measures of survey effort. We believe surveys themselves can be seen as a behavioral tasks and that respondents provide meaningful information about their character skills by way of the effort they put forward on surveys. In particular, we compare measures of grit, conscientiousness and other personality traits, and growth mindset, based on self-reports, and survey effort measures of character. We study the relationship across each other and their relationship with academic and life outcomes such as income and labor-market outcomes, after controlling for cognitive ability and other relevant demographic characteristics. Our results show that survey effort measures of character skills, in particular measures of careless answering in surveys, show great promise for being good proxy measures of relevant non-cognitive skills.

Comparisons of Student Perceptions of Teacher's Performance in the Classroom: Using Parametric Anchoring Vignette Methods for Improving Comparability

Self-reports are an indispensable source of information in education research but might be affected by reference group bias if the frame of reference (i.e. implicit standards), used to answer the questions, differs across students. The anchoring vignettes method was introduced, in other areas of social science, precisely to correct for this source of bias. However, studies that make use of this approach in education are rare and more research is needed to study its potential. This paper uses data from PISA 2012 to investigate the use of the parametric model of the anchoring vignettes method to correct for differential implicit standards in cross-country comparisons of student’s perceptions of an important dimension of teacher quality: teacher’s classroom management. Our results show significant heterogeneity in implicit standards across countries. We also show how correlations between countries’ average teacher classroom management levels and external variables can be improved substantially when heterogeneity in implicit standards is adjusted for. We conclude that the anchoring vignettes method shows a good potential to enhance the validity and comparability of self-reported measures in education.

Non-Cognitive Abilities and Spanish Regional Differences in Student Performance in PISA 2009

The goal of this paper is to analyze the role that non-cognitive skills and, in particular, regional differences in those skills, play on the observed differences in 15-year-old student’s academic performance, across Spanish regions, on PISA 2009. Previous research has shown the relevance of differences in student’s personal, family and school characteristics in accounting for academic differences across Spanish regions but it has also found that a sizeable part of the observed differences remained unexplained. We have found that differences in the distribution of certain non-cognitive skills associated to academic performance like focus, perseverance and resilience play a prominent role in accounting for differences in student performance in PISA 2009. We observe these skills by developing new measures of student effort on standardized tests. In particular, our estimates suggest that a standard deviation reduction in the dispersion of non-cognitive skills across Spanish regions would lead to a 25% reduction in the magnitude of the observed differences in student performance across regions. This is a relevant effect as, for example, a one standard deviation reduction in the regional dispersion of parent’s educational levels or occupational status would only lead to at most a 2% reduction in the magnitude of observed differences in performance on PISA across Spanish regions. Put plainly, a substantial portion of the regional variation in test scores appears attributable to effort on the PISA test, and not necessarily just differences in actual knowledge.

Measures of Student Non-Cognitive Skills and Political Tolerance after Two Years Of The Louisiana Scholarship Program

This report examines the short-term effects of the Louisiana Scholarship Program (LSP) on students’ non-cognitive skills and civic values. While a growing number of studies have evaluated K-12 school voucher programs along academic dimensions, few have focused on the development of non-cognitive skills and civic values. This study aims to address that gap by providing the first analysis of differences in self-reported measures of grit, locus of control, self esteem, and political tolerance associated with the LSP. Using results from a phone survey of applicants to the program, we find little evidence of differences between LSP scholarship recipients and non-recipients. Nevertheless, diagnostics assessing the precision of our instruments to detect differences between subjects indicate that several of the scales measuring non-cognitive skills performed poorly in our sample. Moreover, our relatively low survey response rate of 11 percent raises concerns about the representativeness of our sample. Given these issues, we caution that our results are best understood as descriptive and not necessarily conclusive: they do not represent reliable estimates of the causal impact of the LSP on student non-cognitive skills and political tolerance.