Justin M. Rao

Precinct or Prejudice? Understanding Racial Disparities in New York City's Stop-and-Frisk Policy

Recent studies have examined racial disparities in stop-and-frisk, a widely employed but controversial policing tactic. The statistical evidence, however, has been limited and contradictory. We investigate by analyzing three million stops in New York City over five years, focusing on cases where officers suspected the stopped individual of criminal possession of a weapon (CPW). For each CPW stop, we estimate the ex ante probability that the detained suspect has a weapon. We find that in more than 40% of cases, the likelihood of finding a weapon (typically a knife) was less than 1%, raising concerns that the legal requirement of “reasonable suspicion�? was often not met. We further find that blacks and Hispanics were disproportionately stopped in these low hit rate contexts, a phenomenon that we trace to two factors: (1) lower thresholds for stopping individuals — regardless of race — in high-crime, predominately minority areas, particularly public housing; and (2) lower thresholds for stopping minorities relative to similarly situated whites. Finally, we demonstrate that by conducting only the 6% of stops that are statistically most likely to result in weapons seizure, one can both recover the majority of weapons and mitigate racial disparities in who is stopped. We show that this statistically informed stopping strategy can be approximated by simple, easily implemented heuristics with little loss in efficiency.

Avoiding the Ask: A Field Experiment on Altruism, Empathy, and Charitable Giving

If people enjoy giving, then why do they avoid fund-raisers? Partnering with the Salvation Army at Christmastime, we conducted a randomized field experiment placing bell ringers at one or both main entrances to a supermarket, making it easy or difficult to avoid the ask. Additionally, bell ringers either were silent or said “please give.” Making avoidance difficult increased both the rate of giving and donations. Paradoxically, the verbal ask dramatically increased giving but also led to dramatic avoidance. We argue that this illustrates sophisticated awareness of the empathy-altruism link: people avoid empathic stimulation to regulate their giving and guilt.

Ideological Segregation and the Effects of Social Media on News Consumption

Online publishing, social networks, and web search have dramatically lowered the costs of producing, distributing, and discovering news articles. Some scholars argue that such technological changes increase exposure to diverse perspectives, while others worry that they increase ideological segregation. We address the issue by examining webbrowsing histories for 50,000 US-located users who regularly read online news. We find that social networks and search engines are associated with an increase in the mean ideological distance between individuals. However, somewhat counterintuitively, these same channels also are associated with an increase in an individuals exposure to material from his or her less preferred side of the political spectrum. Finally, the vast majority of online news consumption is accounted for by individuals simply visiting the home pages of their favorite, typically mainstream, news outlets, tempering the consequences -- both positive and negative -- of recent technological changes. We thus uncover evidence for both sides of the debate, while also finding that the magnitude of the effects is relatively modest

Modeling Consumer Preferences and Price Sensitivities from Large-Scale Grocery Shopping Transaction Logs

In order to match shoppers with desired products and provide personalized promotions, whether in online or offline shopping worlds, it is critical to model both consumer preferences and price sensitivities simultaneously. Personalized preferences have been thoroughly studied in the field of recommender systems, though price (and price sensitivity) has received relatively little attention. At the same time, price sensitivity has been richly explored in the area of economics, though typically not in the context of developing scalable, working systems to generate recommendations. In this study, we seek to bridge the gap between large-scale recommender systems and established consumer theories from economics, and propose a nested feature-based matrix factorization framework to model both preferences and price sensitivities. Quantitative and qualitative results indicate the proposed personalized, interpretable and scalable framework is capable of providing satisfying recommendations (on two datasets of grocery transactions) and can be applied to obtain economic insights into consumer behavior.

Experiments as Instruments: Heterogeneous Position Effects in Sponsored Search Auctions

The generalized second price (GSP) auction allocates billions of dollars of advertising via position auctions. Theory tells us that the GSP achieves the efficiency of the Vickrey-Clarke-Groves mechanism but with greater revenue, provided better positions increase click-through-rate by the same scaling factor for all ads. Since position is endogenous, this assumption is largely untested. We develop a novel method, “experiments-as-instruments,�? to re-purpose internal business experimentation to estimate the causal impact of position for 20,000 search ads. We strongly reject the multiplicatively-separable model, position effects differ by 100% across ads, which is partially explained by advertiser attributes.

Whole page optimization: how page elements interact with the position auction

We study the trade-off between layout elements of the search results page and revenue in the real-time sponsored search auction. Using data from a randomized experiment on a major search engine, we find that having images present among the search results tends to simultaneously raise the ad click-through rate and flatten the ad click curve, reducing the premium for occupying the top slot and thus impacting bidding incentives. Theoretical analysis shows that this type of change creates an ambiguous impact on revenue in equilibrium: a steeper curve with lower total click-through rate is preferable only if the expected revenue distribution is skewed enough towards the top bidder. Empirically, we show that this is a relatively rare phenomenon, and we also find that whole page satisfaction causally raises the click-through rate of the ad block. This means search engines have a short-run incentive to boost search result quality, not just a long-run incentive based on competition between providers.

Using gaze patterns to study and predict reading struggles due to distraction

We analyze gaze patterns to study how users in online reading environments cope with visual distraction, and we report gaze markers that identify reading difficulties due to distraction. The amount of visual distraction is varied from none, medium to high by presenting irrelevant graphics beside the reading content in one of 3 conditions: no graphic, static or animated graphics. We find that under highly-distracting conditions, a struggling reader puts more effort into the text -- she takes a longer time to comprehend the text, performs more fixations on the text and frequently revisits previously read content. Furthermore, she reports an unpleasant reading experience. Interestingly, we find that whether the user is distracted and struggles or not can be predicted from gaze patterns alone with up to 80% accuracy and up to 15% better than with non-gaze based features. This suggests that gaze patterns can be used to detect key events such as user strugglefrustration while reading.

Competition and Crowd-Out for Brand Keywords in Sponsored Search

On search keywords with trademarked terms, the brand owner (“focal brand”) and other relevant firms compete for consumers. For the focal brand, paid clicks have a direct substitute in the organic links below the paid ad(s). The proximity of this substitute depends on whether competing firms are aggressively bidding to siphon off traffic. We study the returns to focal brands and competitors using large-scale experiments on Bing with data from thousands of brands. When no competitors are present, we find a positive, statistically significant impact of brand ads of 1%–4%, with larger brands having a smaller causal effect. In this case, the effective “cost per incremental click” is significantly higher than what focal brands typically pay on other keywords. When the focal brand ad is present, competitors in paid positions 2–4 can “steal” 1%–5% of the focal brand’s clicks and raise its costs by shifting traffic to the paid link. Finally, for a set of brands that face competition on their brand search but choos...

Firms' Reactions to Public Information on Business Practices: Case of Search Advertising

We use five years of bidding data to examine the reaction of advertisers to widely disseminated press on the lack of effectiveness of brand search advertising (queries that contain the firms name) found in a large experiment run by eBay (Blake, Nosko and Tadelis, 2015). We estimate that 11% of firms that did not face competing ads on their brand keywords, matching the case of eBay, discontinued the practice of brand search advertising. In contrast, firms did not react to the information pertaining to the high value and ease of running experiments -- we observe no change in the experiment-like variation in advertising levels. Further, while 72% of firms had sharp changes in advertising suitable for estimating causal effects, we find no correlation between firm-level advertising effects and the propensity to advertise in the future. We discuss how a principal-agent problem within the firm would lead to these learning dynamics.

A/B Testing

Large and thus statistically powerful A/B tests are increasingly popular in business and policy to evaluate potential innovations. We study how to optimally use scarce experimental resources to screen innovations. To do so, we propose a new framework for optimal experimentation that we call the A/B testing problem. The key insight of the model is that the optimal experimentation strategy depends on whether most gains accrue from typical innovations, or from rare and unpredictable large successes that can be detected using tests with small samples. We show that, if the tails of the (prior) distribution of true effect sizes is not too fat, the standard approach of trying a few high-powered experiments is optimal. However, when this distribution is very fat tailed, a lean experimentation strategy of trying more but smaller interventions is optimal. We measure this tail parameter using experiments from Microsoft Bings EXP platform and find extremely fat tails. Our theoretical results and empirical analysis suggest that even simple changes to business practices within Bing could dramatically increase innovation productivity.