On the Long-term Impact of Algorithmic Decision Policies: Effort Unfairness and Feature Segregation through Social Learning
OOn the Long-term Impact of Algorithmic Decision Policies:Effort Unfairness and Feature Segregation through Social Learning
Hoda Heidari * 1
Vedant Nanda * 2
Krishna P. Gummadi Abstract
Most existing notions of algorithmic fairness are one-shot : they ensure some form of allocativeequality at the time of decision making, but donot account for the adverse impact of the algo-rithmic decisions today on the long-term welfareand prosperity of certain segments of the popula-tion. We take a broader perspective on algorith-mic fairness. We propose an effort -based mea-sure of fairness and present a data-driven frame-work for characterizing the long-term impact ofalgorithmic policies on reshaping the underlyingpopulation. Motivated by the psychological lit-erature on social learning and the economic lit-erature on equality of opportunity , we proposea micro-scale model of how individuals may re-spond to decision making algorithms. We employexisting measures of segregation from sociologyand economics to quantify the resulting macro-scale population-level change. Importantly, weobserve that different models may shift the group-conditional distribution of qualifications in dif-ferent directions. Our findings raise a number ofimportant questions regarding the formalizationof fairness for decision-making models.
1. Introduction
Machine Learning tools are increasingly employed to makeconsequential decisions for human subjects, in areas suchas credit lending (Petrasic et al., 2017), policing (Rudin,2013), criminal justice (Barry-Jester et al., 2015), andmedicine (Deo, 2015). Decisions made by these algo-rithms can have a long-lasting impact on people’s livesand may affect certain individuals or social groups nega-tively (Sweeney, 2013; Angwin et al., 2016; Levin, 2016). * Equal contribution Computer Science Department, ETHZ¨urich, Z¨urich, Switzerland MPI-SWS, Saarbr¨ucken, Germany.Correspondence to: Hoda Heidari < [email protected] > . Proceedings of the th International Conference on MachineLearning , Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).
This realization has recently spawned an active area of re-search into quantifying and guaranteeing fairness for ma-chine learning (Dwork et al., 2012; Kleinberg et al., 2017;Hardt et al., 2016).Most existing notions of fairness assume a static popula-tion: they ensure some form of allocative equality at thetime of decision making, but do not account for the ad-verse impact of algorithmic decisions today on the longterm welfare and prosperity of different segments of thepopulation. For instance, consider equality of odds (Hardtet al., 2016). The notion requires that the model distributesdifferent types of error (i.e., false positives and false neg-atives) equally across different social groups. But it doesnot take into consideration the fact that for members of theadvantaged group these erroneous predictions may be easyto overturn, whereas for the disadvantaged it may take asignificant amount of effort to improve their qualificationsto obtain better algorithmic outcomes. Furthermore, in thelong run, the decision-making model may nudge differentsegments of the population to obtain very different sets ofqualifications—some of which might be socially and eco-nomically more desirable than others. This may in effectlead to further marginalization of these groups.Motivated by these concerns about existing notions of fair-ness, we argue for a broader view of algorithmic models—one that treats them as policies implemented within a socialcontext and with the potential of impacting individuals andreshaping society. Among other considerations , such viewof decision-making models necessitates a deeper understand-ing of how individual decision subjects may respond to thesemodels and how those responses may translate into adverseimpact for certain segments of the population.In this work, we propose an effort-based measure of un-fairness for algorithmic decisions. We define a data-driven,group-dependent measure of effort drawing on the economicliterature on Equality of Opportunity (Roemer & Trannoy,2015). Our effort function captures the idea that the kind of Another important factor is how a utility maximizing decisionmaker—employing the model—would respond to its predictions.For instance, they may interpret the predictions in a certain way,or update the model entirely. Prior work (Liu et al., 2018; Kannanet al., 2019) has already addressed some of these considerations. a r X i v : . [ c s . C Y ] J un n the Long-term Impact of Algorithmic Decision Policies changes required to obtain a desirable algorithmic outcome(e.g., changing one’s school type from public to private toget a better prediction for SAT score) is often significantlymore difficult to make for members of the disadvantagedgroup compared to the advantaged. Building on this notionof effort, we formulate effort unfairness as the inequality inthe amount of effort required for members of each group toobtain their desired outcomes.To formulate the long-term impact of algorithmic policies onthe underlying population, we specify a micro-scale modelof how individuals respond to algorithmic decision-makingmodels, taking inspiration from the psychological literatureon social learning (Bandura, 1962; 1978). We posit thatindividuals observe and imitate the qualifications of their social models —someone who has received a better algo-rithmic outcome from the decision-making model—if bydoing so, they can obtain higher rewards (Bandura, 1962;Apesteguia et al., 2007). More precisely, we model an in-dividual’s response to be the decision-making algorithm byfirst selecting a social model for him/her; the individual isthen assumed to exert effort to attain his/her model’s qual-ifications if and only if doing so improves his/her overallutility. With this individual-level behavioral model in place,we can simulate decision subjects’ responses and quantifythe macro-scale impact of algorithmic policies on reshapingthe underlying populations. We employ existing measuresof segregation from sociology and economics (Massey &Denton, 1988) to characterize how the distribution of quali-fication for each group changes in response to the deployedmodel. Importantly, we observe that different models mayshift the group-conditional distribution of qualifications invastly different directions.Our work raises a number of important questions about algo-rithmic policies and the formulation of fairness: What is theultimate purpose of a fair predictive model—to guaranteeallocative equality today, or to ensure similar distributionsof qualifications in the long-run? With respect to short termallocative equality, are all error created equal, or should wetake into account the disparity in the effort it takes for differ-ent groups to obtain their desired predictions? In the longrun, what are the type of changes that different predictivemodels impose on the society? Which ones are desirable,and which ones should be watched out for? Is it ethicallyand economically acceptable to nudge different segmentsof the population toward obtaining different qualifications?If not, how can we prevent this without employing a modelwhose decisions may be perceived as unfair today? Theseare all critical questions that must be carefully analyzedbefore determining which model is fair and best-suited tomake consequential decisions for humans. Addressing suchethical challenges is outside the scope of this paper—andarguably intradisciplinary Machine Learning research. Wehope that our work serves as a reminder to the ML com- munity that to formalize fairness appropriately we need tofirst formalize the processes and dynamics through whichalgorithmic decisions impact their subjects and society inthe long run. Most existing notions of algorithmic fairness are one-shotand require that a particular error metric is equal across allsocial groups. Different choices for the metric have led todifferent fairness criteria; examples include demographicparity (Kleinberg et al., 2017; Dwork et al., 2012; Corbett-Davies et al., 2017), disparate impact (Zafar et al., 2017;Feldman et al., 2015), equality of odds (Hardt et al., 2016),and calibration (Kleinberg et al., 2017). Prior notions failto capture the disparity in the effort it takes members of dif-ferent social groups to improve their algorithmic outcome.We propose a group-dependent, data-driven measure of ef-fort, inspired by the literature on Equality Of Opportunity(EOP) (Roemer & Trannoy, 2015; Heidari et al., 2019). (Inparticular, the effort it takes individual i to improve theirfeature k value from x to x (cid:48) is proportional to the differencebetween the rank/quantile of x (cid:48) and x in the distribution offeature k in i ’s social group.)Social (or observational) learning (Bandura, 2008) is a typeof learning that occurs through observing and imitatingthe behavior of others. This type of learning requires asocial model (or role model)—someone of higher statusin the environment. According to the social learning the-ory, observers recreate their role model’s behavior only ifthey have sufficient motivation (Bandura, 1962; Apesteguiaet al., 2007)—this often comes from the observation thatthe model is rewarded for their actions. In our model, anindividual recreates their role model’s qualification if bydoing so he obtains a positive utility, where utility is definedas reward minus effort. Furthermore, it has been shown thatobservers learn best from models that they identify with and find it within their capability to imitate them (Bandura,1962). These points are captured by our effort function.Social learning explicitly captures the role model implica-tions of decision making policies. This echos research insociology and economics which has already established therole model effects of affirmative action policies (Chung,2000). We note that imitation dynamics have been exten-sively studied in population and evolutionary games (see,e.g., (Sandholm, 2010), Chapters 4 and 5).Several recent papers study the impact of decision-makingmodels and fairness interventions on society and individuals(see, e.g., (Liu et al., 2018; Kannan et al., 2019)). Unlikeprior work, our focus is on how subjects respond to algorith-mic policies by improving/updating their (mutable) qual- Social identity is a person’s sense of who they are based ontheir group membership(s) (Tajfel et al., 1979). n the Long-term Impact of Algorithmic Decision Policies ifications . We don’t make any case-specific assumptionsabout how the world changes in response to the deployedmodel, rather allow our micro-scale behavioral model to de-rive the macro-level change. We emphasize that our modelis not meant to perfectly capture all the behavioral nuancesinvolved; rather our primary goal is to highlight the poten-tial role of behavioral dynamics and human responses inshaping the long-term impact of algorithmic models.Also related but orthogonal to our work is a recent lineof research on strategic classification —a setting in whichdecision subjects are assumed to respond strategically andpotentially untruthfully to the choice of the classificationmodel, and the goal is to design classifiers that are robust tostrategic manipulation (Dong et al., 2018; Hu et al., 2019;Milli et al., 2019).
2. Setting
We consider the standard supervised learning setting: Alearning algorithm receives the training data set D = { ( x i , y i ) } ni =1 consisting of n instances, where x i ∈ X specifies the feature vector for individual i and y i ∈ Y ,the ground truth label for him/her. We use s i to refer tothe sensitive feature value (e.g., race, gender, or their in-tersection) for individual i . For the ease of notation, wewill use z i to denote the example ( x i , y i ) . We assume z i fully characterizes individual i with respect to the task athand. The training data is sampled i.i.d. from a distribution P on Z = X × Y . For simplicity, throughout we assumethere exists an unknown function f : X → Y such that forall i , y i = f ( x i ) . Unless specified otherwise, we assume X ⊆ R K , where K denotes the number of features. Thegoal of a learning algorithm is to use the training data tofit a (regression) model (or hypothesis) h : X → Y thataccurately predicts the label for new instances. Let H bethe hypothesis class consisting of all the models available tothe learning algorithm. A learning algorithm receives D asthe input; then utilizes the data to select a model h ∈ H thatminimizes some notion of loss, L . For instance, in regres-sion the empirical mean squared loss of a model h on D isdefined as L D ( h ) = (cid:80) i ∈ D ( y i − ˆ y i ) , where ˆ y i = h ( x i ) .The learning algorithm outputs the model h ∈ H that mini-mizes the empirical loss; i.e., h = arg min h (cid:48) L D ( h (cid:48) ) .We assume there exists a benefit function b : X , Y ×Y → R that quantifies the benefit an individual with feature vector x and ground truth label y receives if the trained modelpredicts label ˆ y for them. Throughout this work we willfocus on benefit functions that are only functions of y, ˆ y andare linear in ˆ y (e.g., b ( y, ˆ y ) = ˆ y − y + 1 or b ( y, ˆ y ) = ˆ y ). Forsimplicity and ease of interpretation, all illustrations in themain body of the paper are performed with ˆ y as the benefitfunction. Throughout, we assume higher predicted labelsare considered more desirable from the point of view of individual decision subjects (e.g., this is the case when thetask is to predict students’ grade to decide who is admittedto a top school). Let h specify the deployed predictive model. Consideran individual characterized by z = ( x , y ) . Let R h ( z , z (cid:48) ) specify the reward or added benefit he/she obtains as theresult of changing his/her characteristics from z to z (cid:48) : R h ( z , z (cid:48) ) = b ( h ( x (cid:48) ) , y (cid:48) ) α − b ( h ( x ) , y ) α , where α > is a constant specifying the individual’s de-gree of risk aversion. This parameter can be adjusted tomodel diminishing returns to added benefit. Unless oth-erwise specified, in our illustrations we take α = 1 . Let E h ( z , z (cid:48) ) specify the effort it takes the individual to updatetheir qualifications and make the change from z to z (cid:48) (wewill shortly elaborate on how effort can be quantified). Theoverall utility of the individual is denoted by U h ( z , z (cid:48) ) andfor simplicity, we assume it takes on a a linear form: U h ( z , z (cid:48) ) = R h ( z , z (cid:48) ) − E h ( z , z (cid:48) ) . That is, utility is simply reward minus effort. When clearfrom the context, we drop the subscript h .Throughout, we focus on effort functions that only dependon x , x (cid:48) , s . For simplicity, we assume the effort function isadditively separable across features—that is, the total effortrequired to change x to x (cid:48) is a linear combination of theeffort needed to change each feature separately: E ( z , z (cid:48) ) = E s ( x , x (cid:48) ) = c s + 1 K K (cid:88) k =1 c s,k (cid:15) s,k ( x k , x (cid:48) k , ) , where (cid:15) s,k ( x k , x (cid:48) k , ) denotes the effort it takes to change thevalue of feature k from x k to x (cid:48) k for an individual belongingto group s (defined below). For group s , c s is a group-dependent constant specifying the minimum effort requiredto make any change. Similarly for k = 1 , · · · , K , c s,k ’sare constant weights which allow us to specify the relativedifficulty of change across different features. For simplicity,throughout we assume c s,k ≡ for all k, s .We define (cid:15) s,k ( x k , x (cid:48) k , ) as follows—depending on the fea-ture type and our domain knowledge about the feature: Depending on our domain knowledge of how features relateto one another, we may find a different aggregating operator (e.g.,max) to be more appropriate than summation. For instance, if twofeatures automatically change together, no extra effort is requiredfor changing both of them simultaneously. Throughout our illus-trations, for simplicity we focus on additively separable functions,but our results can be readily produced for more complicated effortfunctions. See the Appendix for a complete description of the effortfunction. n the Long-term Impact of Algorithmic Decision Policies
Suppose feature k is numerical and monotone (Duivesteijn& Feelders, 2008), that is, we expect an increase in its valueto monotonically increase the predicted label—everythingelse being equal. Monotonicity implies that there is a cleardirection of change that is considered desirable. As an exam-ple in the education context, consider the number of hoursof study : we expect an increase in this feature to increase thean student’s predicted grade. Without loss of generality, weassume higher values of feature k are expected to increasethe predicted label (we can ensure this by preprocessing thedata and negating feature k values if necessary). We define (cid:15) s,k ( x k , x (cid:48) k , ) as follows: (cid:15) s,k ( x k , x (cid:48) k , ) = max { , Q s,k ( x (cid:48) k ) − Q s,k ( x k ) } . Above, Q s,k ( x ) specifies the quantile/rank of value x in theempirical distribution of feature k among individuals whobelong to group s .The above effort function is inspired by the line of work onequality of opportunity (Roemer & Trannoy, 2015): Notethat the distribution of feature k values can be differentacross different social groups (e.g., men and women, orAfrican-Americans and Whites). We take the view that thisis potentially not the result of one group being inherentlyinferior to another (in terms of feature k ). Rather, it is mostlikely due to the underlying socio-economic circumstancesthat the privileged group can achieve higher values of fea-ture k with less effort. To account for this in our effortfunction, we measure the effort it takes a person i in group s to change their feature k value from x k to x (cid:48) k by comparingthe rank/quantile of x (cid:48) k and x k in the empirical distributionof feature k within group s . This implies that if for most peo-ple in group s , the value of feature k is equal or better than x (cid:48) k , we consider it relatively easy for individual i to makethe change from x k to x (cid:48) k . If however, very few in group s have ever been able to achieve x (cid:48) k , then it is considered verydifficult for i to make this change.Using the effort function defined above, in Section 3 we pro-pose a new effort-based measure of algorithmic unfairness.In Section 4, we will utilize our effort function to computeindividual utilities and subsequently specifying the socialmodels.
3. Effort-based Measures of Fairness
Existing formulations of fairness are concerned with howerrors are distributed among various social groups, but theydo not account for the fact that even if errors are distributedsimilarly, the effort required to fix those errors and improveone’s prediction may be significantly higher for the disad-vantaged subpopulation. Building on our notion of effortintroduced in Section 2, in this section we formulate a newmeasure of algorithmic unfairness, called the effort unfair-ness . At a high level, effort unfairness is the disparity in the average effort members of each group have to exert toobtain their desired outcomes—by imitating the appropriaterole models. We propose three different formalizations of effort unfair-ness: bounded-effort unfairness, threshold-reward unfair-ness, and effort-reward unfairness. Each notion correspondsto a distinct/salient way in which decision subjects mayevaluate fairness and respond to their predictions.
Effort ( δ ) A v e r a g e R e w a r d Linear (Women)Linear (Men)NN (Women)NN (Men)DT (Women)DT (Men)
Figure 1: The average reward obtainable by members ofeach group as the result of exerting (at most) δ units ofeffort. For each model, the bounded effort unfairness is thedifference between the dashed and solid curves. Bounded-effort unfairness is the inequality in the averagereward members of each group can obtain by exerting afixed level of effort. More precisely:
Definition 1 (Bounded-effort Unfairness)
Given a con-stant δ , the δ -bounded-effort unfairness of a predictivemodel h is the inequality of the following metric acrossdifferent groups: E i ∼ P : s i = s (cid:18) max z ∈Z R h ( z i , z ) s.t. E h ( z i , z ) ≤ δ (cid:19) . (1)The bounded-effort formulation is motivated by the litera-ture on bounded willpower in behavioral economics (Mul-lainathan & Thaler, 2000), which at a high level posits thatthere is an upper bound on the level of effort people can beexpected to exert.To compute the bounded-effort unfairness in practice, wepropose replacing the expectation in Equation 1 with theempirical mean, and taking the maximum over the availabledata set D . The latter not only simplifies the optimization, Note that algorithmic unfairness has many different aspects.We introduce a new dimension along which algorithmic decisionsdisparately affect different subpopulation, but this is not to under-mine the importance of all other dimensions of unfairness (such aserror disparities). In particular, we do not claim that if a model isfair according to our notion, then it cannot be unfair according toother criteria. n the Long-term Impact of Algorithmic Decision Policies it also has a natural interpretation in terms of social mod-els (see Section 4): Precisely, we estimate Equation 1 by n s (cid:80) i : s i = s (max z ∈ D R h ( z i , z ) s.t. E h ( z i , z ) ≤ δ ) where n s is the number of subjects in group s .Figure 1 illustrates the bounded effort unfairness calcu-lated over the student performance data set (Cortez & Silva,2008). (Information about the data set, our pre-processingsteps, and the trained models can be found in the Ap-pendix. The code used to generate the plots in this papercan be found at https://github.com/nvedant07/effort_reward_fairness .) Note that according tothe bounded-effort measure, different models may be dis-criminatory against different groups. Also depending onthe choice of δ the measure may rank the three modelsdifferently. Threshold-reward unfairness is the inequality in the averageeffort member of each group need to exert to reach a certainlevel of reward. More precisely:
Definition 2 (Threshold-reward Unfairness)
Given aconstant δ , the δ -threshold-reward unfairness of a predictivemodel h is the inequality of the following metric acrossdifferent groups: E i ∼ P : s i = s (cid:18) min z ∈Z E h ( z i , z ) s.t. R h ( z i , z ) ≥ δ (cid:19) . (2)This formulation is motivated by the capability view offairness (Sen, 1993): Sen conceptualizes fairness as theequality of capability, where at a high level, capability isa person’s ability to reach valuable states of being (in ourcase, a certain level of reward).Figure 2 illustrates the average effort unfairness on the stu-dent performance data set. Note that depending on thechoice of δ the measure may rank the three models differ-ently. Also interestingly, depending on the choice of δ thesame model (i.e., linear model) may be considered unfairtoward men, unfair toward women, or perfectly fair!The previous two formulations of effort unfairness—whilewell-motivated—may give us different rankings across thesame set of alternatives depending on our choice of δ . Thefinal formulation, which we call the effort-reward unfairness,resolves this issue by comparing the highest utility membersof each group can possibly achieve by exerting additionaleffort. More precisely: Definition 3 (Effort-reward Unfairness)
For a predictivemodel h , the effort-reward unfairness is the inequality of thefollowing metric across different groups: E i ∼ P : s i = s max z ∈Z U h ( z i , z ) . (3)Figure 3 contrasts the effort-reward measure with existing Reward ( δ ) A v e r a g e E ff o r t Linear (Women)Linear (Men)NN (Women)NN (Men)DT (Women)DT (Men)
Figure 2: The average effort required from members of eachgroup to obtain (at least) δ units of reward. For each model,the threshold-reward unfairness is the difference betweenthe dashed and solid curves.notions of algorithmic unfairness (these measures are pre-cisely defined in the Appendix.) As evident in Figure 3,on the student performance data set none of the existingfairness notions fully captures the effort disparity. See theAppendix for a numerical example further illustrating whyand how our measure of effort unfairness may not be cap-tured by existing notions. LinearReg NeuralNet DTreeReg
Model −1 D i s p a r i t y MSE Effort-Reward Positive Residual Negative Residual MAE
Figure 3: Comparison of the effort-reward unfairness withseveral existing notions of (un)fairness. Effort-reward un-fairness ranks the models differently.
4. How Algorithmic Policies Re-shape Society
To formulate the long-term impact of algorithmic policieson the underlying population, in this section we propose amicro-scale model of how individuals may respond to algo-rithmic policies, taking inspiration from the psychologicalliterature on social learning (Bandura, 1978). We posit thatindividuals observe and potentially imitate the behavior oftheir so-called social models . A social model is anotherdecision subject who has received a higher level of benefitas the result of being subject to the decision-making model.Our social learning model captures settings in which sub-jects don’t know the inner workings of the decision making n the Long-term Impact of Algorithmic Decision Policies
Linear DT NN1.000951.000961.000971.000981.00099 A C I Impacted PopulationInitial Population
Linear DT NN0.880.900.920.940.960.981.001.021.04 A I Impacted PopulationInitial Population
Linear DT NN0.500.550.600.650.70 C I Impacted PopulationInitial Population
Figure 4: Segregation measures computed over the initial and impacted population for 3 models. Segregation may increaseor decrease as the result of the behavioral dynamics.model, but can infer how to improve their standing by ob-serving the decisions it makes for people similar to them(i.e., their social models).With the behavioral dynamics specified, we can quantify themacro-scale long-term impact of the model on reshapingthe underlying populations. We adapt existing measuresof segregation from sociology and economics (Massey &Denton, 1988) to characterize how the distribution of quali-fications for each group changes in response to the deployedmodel. Measures of segregation quantify how separate thetwo subpopulations are in terms of distribution of qualifi-cations. We believe such model-independent measures areimportant to consider, because the decision making modelitself may change over time, but its impact on the underlyingpopulation may be long lasting.
At a high level, we simulate every individual’s response tothe predictive model by selecting a social model for themfrom the training data set; the individual is then assumedto exert effort to attain his/her model’s qualifications if andonly if doing so improves her overall utility.Our micro-scale model is meant to capture two importantnuances pointed out by the social learning theory: First, ac-cording to the theory observers recreate their social model’sbehavior only if they have sufficient motivation and this mo-tivation often comes from observing that the social model isrewarded for their actions. We capture this by assuming thatan individual recreates his/her social model’s qualificationif by doing so he/she is sufficiently rewarded and obtainsa positive utility. Second, it has been shown that observerslearn best from social models that they identify with andfind it within their ability to emulate. These points are cap-tured through our notion of effort and utility . If a potentialsocial model belongs to a different group than that of the in-dividual, the effort it takes to recreate his/her actions is veryhigh, therefore the individual won’t find sufficient utility inimitating him/her. Assuming that the training data is a representative sampleof the population , we select the social models from amongthe individuals present in the data set. In particular, for anindividual z , the model (denoted by z (cid:48) ) is another decisionsubject in D whose imitation would maximize z ’s utility.That is, z (cid:48) = arg max z (cid:48)(cid:48) ∈ D U h ( z , z (cid:48)(cid:48) ) . Two remarks are in order. First, note that each one of ourfairness notions corresponds to a criterion for choosing thesocial model. Depending on the context, one criterion maybetter reflect the human response. In this paper, we deliber-ately focus on utility maximization. This choice is primarilyfor ease of illustration—it allows us to forgo specifying δ —but our analysis can be replicated for the other two criteria,as well.Second, one may ask how can individuals be expected to findthe right social model (in particular, the utility-maximizingone)? One concrete way is through actionable or coun-terfactual explanations (Wachter et al., 2017; Ustun et al.,2018). When an individual fails to receive their desired pre-diction, such explanations lay out the optimal change he/shecan make to improve their outcome. (We re-emphasize thatwe do not consider our model to be a perfect reflection ofextremely nuanced human behavior in the real world. Wedo, however, consider it to be a reasonable approximationof certain aspects of the process. Our primary goal withthis model is to illustrate the role of behavioral dynamics indriving the societal impact of a decision making model.)Through our proposed dynamics, we can simulate how sub-jects respond to the predictive model. We then obtain anew data set D (cid:48) representing the impacted population ’squalifications. Next, we adopt measures of segregation tocompare the initial and impacted populations and quantifythe long-term macro-scale impact of the predictive modelon the underlying population. Our interest in measures ofsegregation is rooted in the observation that unfairness usu-ally emerges as a concern when there is a clear separation We cannot always make this assumption—the sampling pro-cess can become biased in numerous ways. n the Long-term Impact of Algorithmic Decision Policies between different segments of the population (in terms ofqualifications and/or outcomes). If people belonging to twosocially salient groups are fully mixed—in terms of theirfeatures and outcomes—-such concerns are unlikely to arise.We emphasize that segregation does not always imply un-fairness (for instance, segregation could be the consequenceof specialization : different segments of the population maywillingly invest in different sets of qualifications). But un-fairness often comes with some form of segregation. We,therefore, propose measures of segregation as an effectivetest for potential unfairness.
Ethnic and racial segregation is a well-studied phenomenonin sociology. At a high level, segregation is the degree towhich two or more groups live separately from one another.A long line of work in sociology has been concerned withmeasuring segregation. In their highly influential article,Massey & Denton (1988) break down residential segrega-tion into five distinct axes of measurement: centralization,evenness, clustering, exposure, and concentration. Below,we overview these measures and show how three of them canbe utilized to measure the macro-scale impact of decision-making models on the distribution of qualifications.
Evenness measures how unevenly the minority group isdistributed over areal units. Evenness is maximized whenall units have the same relative number of minority andmajority members as the whole population. More precisely,for an area/neighborhood i , let t i denote its total population, m i the number of minority residents, and M i the numberof majority residents of the neighborhood. Also, let p i = m i /t i specify the percentage of minority residents in thearea. Let T and P specify the total population size andminority proportion of the whole population. Suppose thereare N areal units in total. The Atkinson Index (AI) is aparticular measure of evenness satisfying several desirableproperties. For a constant < β < , the Atkinson indexmeasures the inequality of (1 − p i ) /p i (i.e. the number ofmajority residents per minority resident in neighborhood i )computed across all individuals belonging to the minoritygroup: − P − P (cid:16) N (cid:80) Ni =1 (1 − p i ) − β p βi t i /T P (cid:17) / (1 − β ) . Centralization is the degree to which a group is spatiallylocated near the center of an urban area. (Because of certainurban development policies in the past, central areas ofmost cities across the U.S. are declining residential areas.)The degree of centralization can be measured by comparingthe percentage of minority residents living in the centralareas of the city. The Centralization Index (CI) is preciselydefined as follows: (cid:80) i central m i m where m is the total minority It satisfies the transfer principle , compositional invariance , population invariance , and organizational equivalence . population. Clustering measures the extent to which areal units inhab-ited by minority members adjoin one another, or cluster, inspace. For example, the
Absolute Clustering Index (ACI)“expresses the average number of [minority] members innearby [areal units] as a proportion of the total population inthose nearby [areal units]” (Massey & Denton, 1988). ACIis defined as follows: (cid:104)(cid:80) Ni =1 (cid:80) Nj =1 c i,j m i m m j (cid:105) − (cid:104)(cid:80) Ni =1 (cid:80) Nj =1 c i,j N mN (cid:105)(cid:104)(cid:80) Ni =1 (cid:80) Nj =1 c i,j m i m t j (cid:105) − (cid:104)(cid:80) Ni =1 (cid:80) Nj =1 c i,j N mN (cid:105) . For any two areas i and j , c i,j specifies the closeness be-tween their corresponding centers.Residential exposure refers to the degree of potential con-tact, or the possibility of interaction, between minority andmajority group members within geographic areas of a city. Concentration refers to relative amount of physical spaceoccupied by a minority group in the urban environment.There are several notable differences between our settingand that of residential segregation. First, in our settingthere are no predefined notions of “area” or “neighbor-hoods” that individuals belong to. Second, individuals aredescribed by multi-dimentional feature vectors—as opposedto a 2-dimensional vector specifying their residential loca-tion. Third, it is not immediately clear how distances andsimilarity between individuals should be defined. Next, wewill address these issues for evenness, centralization, andclustering.Throughout this section, we will focus on the mutable fea-ture subspace. We take the distance between two individ-uals i (belonging to group s ) and j (belonging to group s (cid:48) ) as follows: d ( i, j ) = max { d s ( i, j ) , d s (cid:48) ( i, j ) } , where d s ( i, j ) is defined as follows: max { , Q s ( y j ) − Q s ( y i ) } + (cid:80) k ∈ mutable (cid:15) s,k ( x k , x (cid:48) k ) . We will use Atkinson index tomeasure evenness. We will specify areas through what wecall focal points —these are feature vectors in the mutablefeature subspace that at least one subject in the original pop-ulation imitates to improve their utility. Each focal pointcorresponds to a neighborhood, and an individual belongs tothe neighborhood of their nearest focal point. We measurethe degree of centralization by comparing the percentageof minority individuals whose predictions are above theaverage (e.g., ˆ Y > . ). We measure ACI at the individ-ual level—that is, we assume each individual correspondsto a neighborhood. We define the similarity between twoneighborhoods as follows: c i,j = e − d ( i,j ) .Figure 4 illustrates our measures of segregation both forthe initial population (depicted in blue) and the impactedpopulation—after individuals respond to the model by ad-justing their qualifications (depicted in red). Segregation n the Long-term Impact of Algorithmic Decision Policies τ A C I Impacted PopulationInitial Population (a) Clustering vs. τ τ A I Impacted PopulationInitial Population (b) Evenness vs. τ τ C I Impacted PopulationInitial Population (c) Centralization vs. τ Figure 5: The change in various measures of segregation as a function of enforcing fairness contraints with strength τ .can change in counter-intuitive ways through imitation dy-namics.
5. The Role of Fairness Interventions
In this Section, we investigate the effect of enforcing fair-ness constraints—at the time of training—on the long termpopulation-level impact of the deployed model. We focuson the case of linear regression. We train a model by mini-mizing the mean squared error while imposing the welfareconstraints proposed by Heidari et al. (2018).Figure 5 shows the effect of imposing fairness constraintson various measures of segregation (all computed by takingfemales as the minority/protected group). One might expectthat these constraints would always reduce segregation inthe long run. As illustrated in Figure 5, this is not always thecase. For a small value of τ , enforcing fairness constraintscan significantly reduce the degree of clustering (see Fig-ure 5a). Larger values of τ can reverse this effect and lead toa population that is more heavily clustered/segregated com-pared to the original population. Evenness remains relativelyunchanged regardless of the value of τ (see Figure 5b).These findings highlight an important insight about fairnessconstraints: they can affect segregation in two competingways. On the one hand by automatically assigning a desir-able label to some members of the disadvantaged group, themodel incentivizes these members not to make any change.On the other hand, these members can serve as social mod-els for the rest of the disadvantaged group, nudging moreof them to improve their qualifications and obtain betterlabels. Which force is more powerful? One can only answerthis by simulating the dynamics on the particular data set athand. We see clear parallels between our observations andthe prior work on affirmative action policies. Advocates ofaffirmative action often argue that a larger representationof minorities in desirable positions can lead to role modelswho encourage other minorities in their investment deci-sions (see e.g., (Chung, 2000)). At the same time, critics argue that affirmative action quota may indirectly harm thedisadvantaged group members by reducing their incentivesto invest in qualifications (Coate & Loury, 1993a;b). Similarto our work, economic results on the long-term impact ofaffirmative action policies is mixed and context-specific.We end this section with a remark on fairness-restoringinterventions. While we focused on algorithmic interven-tions, we must emphasize that changing the decision-makingmodel is not the only mechanism through which segrega-tion and unfairness can be alleviated. Instead of artificiallychanging the decision boundary, it may be socially more de-sirable to address unfairness before people are subjected toalgorithmic decision making. For instance, one could designand implement policies that make it easier for disadvantagedgroup members to obtain certain qualifications. We leavethe analysis of such feature interventions as a promisingdirection for future work.
6. Conclusion & Future Directions
We presented a data-driven framework for studying the po-tential long-term impact of predictive models on decisionsubjects and society. We proposed a micro-model of humanresponse to algorithmic policies rooted in psychology andseveral macro-level measures of change borrowed from soci-ology and economics. Our work suggests several immediatedirections for future work, including but not limited to (a)human subject experiments to investigate the viability ofour behavioral model; (b) designing an efficient mechanismfor bounding effort-reward unfairness.
Acknowledgements
K. P. Gummadi is supported in part by an ERC Ad-vanced Grant “Foundations for Fair Social Computing” (no.789373). n the Long-term Impact of Algorithmic Decision Policies
References
Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machinebias.
Propublica , 2016.Apesteguia, J., Huck, S., and Oechssler, J. Imitation—theory and experimental evidence.
Journal of EconomicTheory , 136(1):217–235, 2007.Bandura, A. Social learning through imitation. 1962.Bandura, A. Social learning theory of aggression.
Journalof communication , 28(3):12–29, 1978.Bandura, A. Observational learning.
The internationalencyclopedia of communication , 2008.Barry-Jester, A., Casselman, B., and Goldstein, D. Thenew science of sentencing.
The Marshall Project , August2015.Calders, T., Karim, A., Kamiran, F., Ali, W., and Zhang,X. Controlling attribute effect in linear regression. In
Proceedings of the International Conference on DataMining , pp. 71–80. IEEE, 2013.Chung, K.-S. Role models and arguments for affirmative ac-tion.
American Economic Review , 90(3):640–648, 2000.Coate, S. and Loury, G. Antidiscrimination enforcement andthe problem of patronization.
The American EconomicReview , 83(2):92–98, 1993a.Coate, S. and Loury, G. C. Will affirmative-action policieseliminate negative stereotypes?
The American EconomicReview , pp. 1220–1240, 1993b.Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., and Huq,A. Algorithmic decision making and the cost of fairness.In
Proceedings of the 23rd ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining ,pp. 797–806. ACM, 2017.Cortez, P. and Silva, A. M. G. Using data mining to predictsecondary school student performance. 2008.Deo, R. C. Machine learning in medicine.
Circulation , 132(20):1920–1930, 2015.Dong, J., Roth, A., Schutzman, Z., Waggoner, B., and Wu,Z. S. Strategic classification from revealed preferences. In
Proceedings of the 2018 ACM Conference on Economicsand Computation , pp. 55–70. ACM, 2018.Duivesteijn, W. and Feelders, A. Nearest neighbour classifi-cation with monotonicity constraints. In
Joint EuropeanConference on Machine Learning and Knowledge Dis-covery in Databases , pp. 301–316. Springer, 2008. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel,R. Fairness through awareness. In
Proceedings of theInnovations in Theoretical Computer Science Conference ,pp. 214–226. ACM, 2012.Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C.,and Venkatasubramanian, S. Certifying and removingdisparate impact. In
Proceedings of the InternationalConference on Knowledge Discovery and Data Mining ,pp. 259–268. ACM, 2015.Hardt, M., Price, E., and Srebro, N. Equality of opportunityin supervised learning. In
Proceedings of the 30th Con-ference on Neural Information Processing Systems , pp.3315–3323, 2016.Heidari, H., Ferrari, C., Gummadi, K. P., and Krause, A.Fairness behind a veil of ignorance: A welfare analysis forautomated decision making. In
Proceedings of the 32ndConference on Neural Information Processing Systems ,2018.Heidari, H., Loi, M., Gummadi, K. P., and Krause, A. Amoral framework for understanding of fair ml througheconomic models of equality of opportunity. In
Proceed-ings of the 2nd ACM Conference on Fairness, Account-ability, and Transparency , 2019.Hu, L., Immorlica, N., and Vaughan, J. W. The disparateeffects of strategic manipulation. In
Proceedings of the2nd ACM Conference on Fairness, Accountability, andTransparency , 2019.Kannan, S., Roth, A., and Ziani, J. Downstream effects ofaffirmative action. In
Proceedings of the 2nd ACM Con-ference on Fairness, Accountability, and Transparency ,2019.Kleinberg, J., Mullainathan, S., and Raghavan, M. Inherenttrade-offs in the fair determination of risk scores. In
Inproceedings of the 8th Innovations in Theoretical Com-puter Science Conference , 2017.Levin, S. A beauty contest was judged by AI and the robotsdidn’t like dark skin.
The Guardian , 2016.Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., and Hardt,M. Delayed impact of fair machine learning. In
Proceed-ings of the International Coference on Machine Learning ,2018.Massey, D. S. and Denton, N. A. The dimensions of resi-dential segregation.
Social forces , 67(2):281–315, 1988.Milli, S., Miller, J., Dragan, A. D., and Hardt, M. Thesocial cost of strategic classification. In
Proceedings ofthe 2nd ACM Conference on Fairness, Accountability,and Transparency , 2019. n the Long-term Impact of Algorithmic Decision Policies
Mullainathan, S. and Thaler, R. H. Behavioral economics.Technical report, National Bureau of Economic Research,2000.Petrasic, K., Saul, B., Greig, J., and Bornfreund, M. Algo-rithms and bias: What lenders need to know.
White &Case , 2017.Roemer, J. E. and Trannoy, A. Equality of opportunity. In
Handbook of income distribution , volume 2, pp. 217–300.Elsevier, 2015.Rudin, C. Predictive policing using machine learning todetect patterns of crime.
Wired Magazine , August 2013.Retrieved 4/28/2016.Sandholm, W. H.
Population games and evolutionary dy-namics . MIT press, 2010.Sen, A. Capability and well-being.
The quality of life , 30,1993.Sweeney, L. Discrimination in online ad delivery.
Queue ,11(3):10, 2013.Tajfel, H., Turner, J. C., Austin, W. G., and Worchel, S. Anintegrative theory of intergroup conflict.
Organizationalidentity: A reader , pp. 56–65, 1979.Ustun, B., Spangher, A., and Liu, Y. Actionable recourse inlinear classification. In
Proceedings of the 2nd ACM Con-ference on Fairness, Accountability, and Transparency ,2018.Wachter, S., Mittelstadt, B., and Russell, C. Counterfactualexplanations without opening the black box: Automateddecisions and the GDPR. 2017.Zafar, M. B., Valera, I., Gomez Rodriguez, M., and Gum-madi, K. P. Fairness constraints: Mechanisms for fairclassification. In
Proceedings of the 20th InternationalConference on Artificial Intelligence and Statistics , 2017.
A. The Effort Function
We define (cid:15) s,k ( x k , x (cid:48) k , ) —depending on the type of feature k —as follows: • Non-monotone numerical feature:
Suppose feature k is numerical, but it is not clear which direction ofchange should increase the probability of the instance x being labeled as positive. An example of this type offeature in the education context is extracurricular activ-ities —depending on other factors this may be increaseor decrease one’s performance in school. For this typeof feature, we assume change in either direction re-quires effort, and define (cid:15) s,k ( x k , x (cid:48) k , ) as follows: (cid:15) s,k ( x k , x (cid:48) k ) = | Q s,k ( x (cid:48) k ) − Q s,k ( x k ) | . • Ordinal feature:
We define (cid:15) s,k ( x k , x (cid:48) k , ) similar tonumerical features—depending on whether we con-sider the attribute monotone or not. • Categorical feature:
Suppose feature k is categori-cal and can take on n k different values { v , · · · , v n k } (example: marital status). We define (cid:15) s,k via n K con-stants, c i,j for ≤ i, j ≤ n k , with c i,j specifying theeffort required to change the value of feature k from v i to v j . Throughout our simulations and for simplicity,we assume there exists a constant c such that c i,j ≡ c for all ≤ i, j ≤ n k . • (Conditionally) immutable feature: We call feature k (conditionally) immutable if there exist two values x k (cid:54) = x (cid:48) k , where the change from x k to x (cid:48) k is consid-ered impossible. For example, race is an immutablefeature (one cannot be expected to change their race).Age is conditionally immutable (one cannot be ex-pected to become younger). In this case we define oureffort function as follows: (cid:15) s,k ( x k , x (cid:48) k ) = ∞ . B. The Student Performance Data Set
The student performance data set (Cortez & Silva, 2008)contains information about student achievement in sec-ondary education of two Portuguese schools. The dataattributes include student grades, demographic, social andschool related features. The data set consists of 649 in-stances/student, with each instance consisting of 32 features.The task is to predict the student’s final grade (value from0 to 20) in Portuguese. Out of the 32 features, we chooseonly features that are considered mutable in at least onedirection, that is, the student can exert effort and change thefeature value. We dropped all immutable features—exceptgender—to be able to find a social model for every student.(Since the data set is very small, this would not have beenpossible had we kept the immutable features). This resultsin a total of 23 features out of which 10 are binary and therest are numerical. We then perform a 70:30 train-test split,with the train set consisting of 454 instances and the test setconsisting of 195 instances.
C. Trained Models
We trained the following models on the student performancedata set: • Neural network:
A shallow neural network with onehidden layer (ReLu activation) containing 100 nodes.Loss function with L2 regularization with regulariza-tion strength = 10. Regularization strength and numberof nodes in the hidden layer were found using gridsearch by doing a 3 fold cross validation and taking the n the Long-term Impact of Algorithmic Decision Policies s c h oo l a dd r e ss t r a v e l t i m e s t u d y t i m e s c h oo l s u p f a m s u p p a i d a c t i v i t i e s h i g h e r i n t e r n e t r o m a n t i c f a m r e l f r ee t i m e g oo u t D a l c W a l c h e a l t h a b s e n c e s I n t e r c e p t Ridge Regression,Regularization Const. = 0.1 1.4407 0.3136 -0.2481 1.6016 -1.4664 0.0881 -0.7904 0.1965 2.5093 0.5606 -0.4103 0.5203 -0.9247 -0.0061 -1.0606 -0.5881 -0.8108 -1.5757 8.9708Ridge Regression,Regularization Const. = 200 0.6065 0.2795 -0.0926 0.4125 -0.2505 0.0571 -0.1035 0.1211 0.6293 0.2433 -0.2105 0.1164 -0.2048 -0.1341 -0.2757 -0.3072 -0.2371 -0.1182 10.5303 (a) Mutable feature weights (and intercept) common to both models. F e d u F j o b ( a t _ h o m e ) F j o b ( h e a l t h ) F j o b ( o t h e r ) F j o b ( s e r v i c e s ) F j o b ( t e a c h e r ) M e d u M j o b ( a t _ h o m e ) M j o b ( h e a l t h ) M j o b ( o t h e r ) M j o b ( s e r v i c e s ) M j o b ( t e a c h e r ) P s t a t u s a g e f a il u r e s f a m s i z e g u a r d i a n ( f a t h e r ) g u a r d i a n ( m o t h e r ) g u a r d i a n ( o t h e r ) nu r s e r y r e a s o n ( c o u r s e ) r e a s o n ( h o m e ) r e a s o n ( o t h e r ) r e a s o n ( r e p u t a t i o n ) s e x Ridge Regression,Regularization Const. = 200 0.2781 -0.0262 0.0007 0.0048 -0.1533 0.174 0.2966 -0.2095 0.1352 -0.1355 0.0145 0.1953 0.0362 -0.0753 -0.5692 -0.1352 0.13 -0.0402 -0.0899 0.0053 -0.1258 0.0601 -0.217 0.2827 0.3614 (b) Immutable feature weights; only applicable to Ridge Regression with regularization const = 200.
Figure 6: Weights assigned by 2 models, one trained with only mutable features (top row in Figure 6a) and the other trainedwith both mutable and immutable features (bottom row in Figure 6a and the model in Figure 6b).parameters that resulted in the maximum average testaccuracy. • Linear regressor:
Least squares solver. Finds param-eters B such that L2 norm of | Bx − Y | is minimized. • Decision Tree:
Decision Tree Regressor with maxi-mum depth of 5 to avoid overfitting. Max depth pa-rameter was chosen using grid search by doing a 3-fold cross validation and choosing the parameter thatmaximised the average test set accuracy. Criterion forsplitting was minimization of MSE.
D. Fairness Notions for Regression
Positive residual difference (Calders et al., 2013) is com-puted by taking the absolute difference of mean positiveresiduals across groups: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) | G +1 | (cid:88) i ∈ G max { , (ˆ y i − y i ) } − | G +2 | (cid:88) i ∈ G max { , (ˆ y i − y i ) } (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) . Negative residual difference (Calders et al., 2013) is com-puted by taking the absolute difference of mean negativeresiduals across groups: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) | G − | (cid:88) i ∈ G max { , ( y i − ˆ y i ) } − | G − | (cid:88) i ∈ G max { , ( y i − ˆ y i ) } (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) . E. Why Existing Notions of Fairness Fail toCapture Effort-Reward Disparity
Figure 6 shows an example of 2 ridge regressions, bothtrained on the student performance dataset (described insection B), but one has access to only mutable features andthe other has access to both mutable and (conditionally)immutable features. For simplicity, let’s call them “muta-ble model” and “combined model” respectively. Both the “mutable model” and “combined model” have similar er-ror distributions on the dataset with Mean Averaged Errors(MAE) of . and . on the entire population. Theyalso have similar errors across sub-groups defined basedon the value of sensitive feature s (for the student dataset, s corresponds to gender); with MAEs for the sub-groupwith s = 1 (females) being . and . and MAEsfor sub-group with s = 0 (males) being . and . for “mutable model” and “combined model” respectively.Lastly, both the models also have comparable measures ofexisting fairness notions defined in section D with positiveresiduals of . and . and negative residuls of . and . respectively.However, when evaluated for the effort-reward unfairness,“mutable model” and “combined model” perform differentlywith measures of . and . respectively. One of thereasons for such contrasting values is the different weightseach model assigns to the mutable features (shown in Fig-ure 6a). For example, consider a student at a benefit levelof b initial (assuming benefit function = predicted value bythe model) subject to predictions by the “mutable model”(top row in Figure 6a), were to imitate a role model havingvalue of the continuous feature, “studytime”, greater by 1unit. Assume, for simplicity, that all other feature valuesof the role model and the student are same. Say the effortexerted to make this change is e which brings the studentto a benefit level of b (=new predicted value by the model),thus making utility, u mutable = b − b initial − e . Now say thesame student were subject to predictions by the “combinedmodel” (bottom row in Figure 6a) and were to immitatethe same role model (having “studytime” greater by 1 unitand having all other features same as the student) as in theprevious case. Since both the models have similar predic-tion errors, we can assume that the student has a similarprediction value as in the previous case (thus being at thesame benefit level of b initial ). The effort is independent ofthe model, so effort in this case remains e . However, since n the Long-term Impact of Algorithmic Decision Policies the weight assigned by “combined model” to “studytime”is . x the weight assigned by “mutable model” (see Fig-ure 6a), increasing “studytime” by 1 unit will result in a newbenefit level of b (cid:48) ( < b ) . Thus utility in this case, u combined = b (cid:48) − b initial − e . Since b initial , b , b (cid:48) and e are all positivevalues, u mutable > u combinedcombined