Brandon M. Stewart

A Model of Text for Experimentation in the Social Sciences

ABSTRACT Statistical models of text have become increasingly popular in statistics and computer science as a method of exploring large document collections. Social scientists often want to move beyond exploration, to measurement and experimentation, and make inference about social and political processes that drive discourse and content. In this article, we develop a model of text data that supports this type of substantive research. Our approach is to posit a hierarchical mixed membership model for analyzing topical content of documents, in which mixing weights are parameterized by observed covariates. In this model, topical prevalence and topical content are specified as a simple generalized linear model on an arbitrary number of document-level covariates, such as news source and time of release, enabling researchers to introduce elements of the experimental design that informed document collection into the model, within a generally applicable framework. We demonstrate the proposed methodology by analyzing a collection of news reports about China, where we allow the prevalence of topics to evolve over time and vary across newswire services. Our methods quantify the effect of news wire source on both the frequency and nature of topic coverage. Supplementary materials for this article are available online.

Use of force and civil–military relations in Russia: an automated content analysis

Russias intervention in the Georgian–South Ossetian conflict has highlighted the need to rigorously examine trends in the public debate over the use of force in Russia. Approaching this debate through the prism of civil–military relations, we take advantage of recent methodological advances in automated content analysis and generate a new dataset of 8000 public statements made by Russias political and military leaders during the Putin period. The data show little evidence that military elites exert a restraining influence on Russian foreign and defence policy. Although more hesitant than their political counterparts to embrace an interventionist foreign policy agenda, Russian military elites are considerably more activist in considering the use of force as an instrument of foreign policy.

The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums

In this study, we develop methods for computationally measuring the degree to which students engage in MOOC forums with other students holding different political beliefs. We examine a case study of a single MOOC about education policy, Saving Schools, where we obtain measures of student education policy preferences that correlate with political ideology. Contrary to assertions that online spaces often become echo chambers or ideological silos, we find that students in this case hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

TopicCheck: Interactive Alignment for Assessing Topic Model Stability.

Content analysis, a widely-applied social science research method, is increasingly being supplemented by topic modeling. However, while the discourse on content analysis centers heavily on reproducibility, computer scientists often focus more on scalability and less on coding reliability, leading to growing skepticism on the usefulness of topic models for automated content analysis. In response, we introduce TopicCheck, an interactive tool for assessing topic model stability. Our contributions are threefold. First, from established guidelines on reproducible content analysis, we distill a set of design requirements on how to computationally assess the stability of an automated coding process. Second, we devise an interactive alignment algorithm for matching latent topics from multiple models, and enable sensitivity evaluation across a large number of models. Finally, we demonstrate that our tool enables social scientists to gain novel insights into three active research questions.

Psychological and Physiological Responses following Repeated Peer Death

Objective Undergraduates at a university in the United States were exposed – directly and indirectly – to 14 peer deaths during one academic year. We examined how individual and social factors were associated with psychological (e.g., anxiety, depression, somatization) and physiological (i.e., cortisol) distress responses following this unexpected and repeated experience with loss. Method Two to three months after the final peer death, respondents (N = 122, 61% female, 18–23 years, M = 20.13, SD = 1.14) reported prior adverse experiences, degree of closeness with the deceased, acute responses to the peer deaths, ongoing distress responses, social support, support seeking, and media viewing. A subset (n = 24) returned hair samples for evaluation of cortisol responses during the previous 3 months. Results Ongoing psychological distress was associated with a) prior interpersonal trauma, b) fewer social supports, and c) media exposure to news of the deaths (ps<.05). Participants who had no prior bereavements showed, on average, high cortisol (>25 p/mg) compared to individuals with one or two prior bereavement experiences (who were, on average, within the normal range, 10 to 25 p/mg) (p<.05). Only 8% of the sample utilized available university psychological or physical health resources and support groups. Conclusions Limited research has examined the psychological and physiological impact of exposure to chronic, repeated peer loss, despite the fact that there are groups of individuals (e.g., police, military soldiers) that routinely face such exposures. Prior adversity appears to play a role in shaping psychological and physiological responses to repeated loss. This topic warrants further research given the health implications of repeated loss for individuals in high-risk occupations and university settings.

The Civic Mission of MOOCs: Engagement across Political Differences in Online Forums

Massive open online courses (MOOCs) attract diverse student bodies, and course forums could potentially be an opportunity for students with different political beliefs to engage with one another. We test whether this engagement actually takes place in two politically-themed MOOCs, on education policy and American government. We collect measures of students’ political ideology, and then observe student behavior in the course discussion boards. Contrary to the common expectation that online spaces often become echo chambers or ideological silos, we find that students in these two political courses hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

Discourse: MOOC Discussion Forum Analysis at Scale

We present Discourse, a tool for coding and annotating MOOC discussion forum data. Despite the centrality of discussion forums to learning in online courses, few tools are available for analyzing these discussions in a context-aware way. Discourse scaffolds the process of coding forum data by enabling multiple coders to work with large amounts of forum data. Our demonstration will enable attendees to experience, explore, and critique key features of the app.