Network Structure, Efficiency, and Performance in WikiProjects
NNetwork Structure, Efficiency, and Performance in WikiProjects
Edward L. PlattUniversity of [email protected] Daniel M. RomeroUniversity of [email protected]
Abstract
The internet has enabled collaborations at a scale never beforepossible, but the best practices for organizing such large collabo-rations are still not clear. Wikipedia is a visible and successful ex-ample of such a collaboration which might offer insight into whatmakes large-scale, decentralized collaborations successful. We an-alyze the relationship between the structural properties of WikiPro-ject coeditor networks and the performance and efficiency of thoseprojects. We confirm the existence of an overall performance-efficiency trade-off, while observing that some projects are higherthan others in both performance and efficiency, suggesting the ex-istence factors correlating positively with both. Namely, we find anassociation between low-degree coeditor networks and both highperformance and high efficiency. We also confirm results seen inprevious numerical and small-scale lab studies: higher performancewith less skewed node distributions, and higher performance withshorter path lengths. We use agent-based models to explore possi-ble mechanisms for degree-dependent performance and efficiency.We present a novel local-majority learning strategy designed tosatisfy properties of real-world collaborations. The local-majoritystrategy as well as a localized conformity-based strategy both showdegree-dependent performance and efficiency, but in opposite di-rections, suggesting that these factors depend on both networkstructure and learning strategy. Our results suggest possible ben-efits to decentralized collaborations made of smaller, more tightly-knit teams, and that these benefits may be modulated by the partic-ular learning strategies in use.
Introduction
The problem with Wikipedia is that it only works inpractice. In theory, it’s a total disaster.—Gareth Owen [10]
The internet has enabled collaborations at a global scale.Wikipedia, a free encyclopedia that invites anyone to editarticles, is one of the most successful and visible exam-ples of such a collaboration. Organizing groups without top-down control is notoriously difficult [12], and yet Wikipedia,with millions of self-organized editors, has produced a high-quality encyclopedia [14, 24]. A better theoretical under-standing of projects like Wikipedia is highly desirable as itcould help inform the design of new collaborative projects. We focus on one aspect of a large-scale decentralized col-laboration: its network structure [31]. How does Wikipedia’snon-hierarchical structure relate to its success?We look at WikiProjects on the English-languageWikipedia. WikiProjects are collections of thematically re-lated articles, each with their own standards and norms.When measuring the quality of collaborative projects, thereare at least two distinct measures to consider. The first mea-sure is short-term: how effective a unit of work is at im-proving the collaboration’s output, which we call efficiency .The other measure is long-term: the highest quality typicallyreached by an output, which we call performance . These twoterms are often used interchangeably, but we find it fruit-ful to distinguish between the two. We find that Wikipediaexhibits an overall trade-off between performance and effi-ciency. However, some WikiProjects surpass others in bothefficiency and performance, suggesting the existence of fac-tors that correlate positively with both.Our study focuses on the coeditor networks of eachWikiProject: which editors have edited at least one articlein common? These relationships represent the possible flowof information. We focus specifically on mean degree, de-gree skewness, and path length. High-degree editors havemore collaborators, which can increase diversity and accessto information at the possible expense of higher coordina-tion costs [20, 15]. Highly skewed degree distributions canamplify the biases of high-degree editors while reducing theneed for explicit coordination [22]. Networks with shorterpath lengths allow information to travel more quickly at thepossible expense of less local diversity [28, 3].In addition to our empirical study, we use agent-basedmodeling to examine the consequences of specific assump-tions on networked collaboration. We model individual be-havior using a social learning strategy that assumes agents1. can only access a fraction of the model’s state, 2. interactwith others who share their concerns, and 3. integrate theirpreferences into a single state. Our model is the first we areaware of to incorporate these assumptions, which are presentacross many real-world collaborations, including Wikipedia.1 a r X i v : . [ c s . S I] A p r ur main findings are: • Despite an overall performance/efficiency trade-off,WikiProjects with low-degree coeditor networks tendto have both higher performance and higher efficiency; • Short paths are associated with higher performance,consistent with a conformity-based learning strategy; • Structural inequality, as measured by degree skewness,is associated with lower performance; • Our agent-based model shows that the efficiency andperformance of collaborations can depend on networkdegree, and that the direction of that dependence varieswith social learning strategy.Our findings shed light on the importance of networkstructure for successful collaboration. These findings mightbe informative for future interventions that recommend tasksbased on how they will influence network structure, or for in-terventions that seek to encourage behaviors complementaryto existing network structure.
Background and Related Work
The present paper investigates the relationship betweensocial networks and collaboration outcomes. This connec-tion has been explored by a number of theoretical, numeri-cal, and small-scale lab studies in the field of social learning .We contribute to this literature with a large-scale, empiricalfield study. In much of the existing literature, degree dis-tribution correlates with outcome measures. But aside fromthe naive Bayes case, it is unknown whether the correlationis explained best by degree or by another structural property,such as characteristic path length. In the empirical networkswe study, unlike artificial networks, the structural proper-ties vary independently, making it easier to isolate individualnetwork properties that correlate with outcome variables.
Social Learning. In networked social learning , agents arerepresented by nodes on a network and can interact onlywith their neighbors. Social learning tasks can be dividedinto cases where agents have generated signals (indepen-dently noisy estimates of a true value) and those whereagents have interpreted signals (solutions based on differ-ent selections of available data) [19]. The behavior of in-dividual agents is described by their social learning strat-egy . For generated signals, a naive Bayesian approach con-verges to the truth when all agents have the same degree,while the speed of convergence depends on the spectral gap between the two largest eigenvalues of the network’s in-teraction matrix [9, 15]. Complex social learning tasks canalso be modeled as the problem of maximizing an objectivefunction with many local maxima, referred to as a ruggedlandscape [26, 28, 27, 17, 3]. Numerical simulations haveshown that efficient networks (those with short paths be-tween nodes) can result in faster convergence at the costof a less optimal solution, due to less time for exploration [28, 17]. However, when conformity-based social learningstrategies are used, efficient networks can sometimes findmore optimal solutions than inefficient ones [3]. Using anagent-based model, Hong and Page [20] found that diversegroups can outperform groups composed of the best individ-ual problem-solvers. Lab experiments.
Lab-based experiments on networkedcollaboration suggest a complex interaction between net-work topology and other factors. While groups of net-worked human subjects perform very well on difficult graph-coloring tasks, the best performing network architectures(e.g., fully-connected vs. small-world) vary from task to task[22]. The same studies found that while human subjectstend to perform well on many networks, they perform worston self-organized networks, possibly due to higher struc-tural inequality (degree skewness). Similarly, some networktopologies are able to reach faster decisions in the presenceof more information, while others show the opposite effect[23]. Based on lab experiments, Fowler and Christakis [11]suggest that individual decisions towards altruism are con-ditional on their neighbor’s behavior and “contagious” upto three degrees away. Later experiments by Suri and Watts[36] confirmed the existence of conditional altruism, butconcluded that altruism influences only first-degree neigh-bors.
Digital Communities.
Research on digital communitieshas also examined the role of diversity and inequality in col-laborative work and decision-making. In sociology, researchhas focused on the relationship between network structureand social capital. Powerful individuals are often “brokers”who act as exclusive intermediaries between disconnectedportions of the social network [35]. Similarly, successful in-novation in organizations often occurs in “structural holes”between groups [16].For Wikipedia specifically, Robert and Romero [33]found that larger group sizes yield higher article ratingswhen the groups are diverse and experienced. Kittur andKraut found that different types of coordination have a com-plex effect on the quality of Wikipedia articles [25]. Bothexplicit and implicit coordination result in higher quality ar-ticles, with explicit coordination being especially central inthe early life of an article. Shaw and Hill [34] found thatbehavior in online wiki communities is consistent with the“iron law of oligarchy,” which states that earlier membersof a group will, over time, gain disproportionate decision-making power and act increasingly out of self-interest ratherthan the good of the group [29]. Similarly, Halfaker et al.[18] attributed decreasing participation on Wikipedia to poorretention of new users. Looking specifically at Wikipediapolicies determined by editor consensus, Keegan and Fiesler[24] found a trend from flexible rule-making towards lessflexible maintenance and deliberation. Using content analy-sis, Morgan et al. [30] found WikiProjects to be more looselyorganized than traditional teams.2cross the broad range of work discussed above, a fewkey themes emerge. Both the efficiency and the performanceof a collaboration are important considerations and vary de-pending on both network structure and type of task [22].While generated signal models of social learning predict norelationship between the two [15], contagion-style innova-tion models predict a trade-off [27, 3]. Such a trade-off hasbeen observed in simulations and lab experiments on collab-oration [22, 17].
WikiProjects
Many articles on Wikipedia belong to one or moreWikiProjects. WikiProjects are groups of thematically-related articles (e.g., articles related to Philosophy). Infor-mation about an article’s associated WikiProjects can beviewed on that article’s talk page (Figure 1). Each WikiPro-ject has its own page and talk page, containing informationabout conventions within the project as well as discussionsabout individual articles. WikiProjects are thus distinct com-munities, with distinct norms and processes. These commu-nities are the fundamental units of analysis in this paper.One of the main roles of a WikiProject is to evaluate thequality of its articles. Quality assessments are made throughconsensus-based deliberation on the WikiProject talk page.Within a WikiProject, assessments are typically made us-ing the following assessment classes (in order of increas-ing quality): Stub, Start, C, B, A. Different WikiProjectscan assign different quality assessments to the same article.Differences between quality assessments could reflect dif-ferent quality standards, different grading systems, differentresponsiveness to changes in an article, etc.In addition to the above assessment classes, articles onWikipedia can be tagged as “good article” (GA) or “fea-tured article” (FA) quality. FA and GA determinations aremade using a Wikipedia-wide consensus, independently ofWikiProject-based evaluations. FA articles are “the best arti-cles Wikipedia has to offer” [8]. GA articles meet “a core setof editorial standards“ but are “not featured article quality”[7]. When an article is assigned GA or FA status, WikiPro-ject quality assessments are often updated to reflect that sta-tus. For example, the article
Mewtwo was assessed as GAstatus on October 5, 2009 and shortly afterwards its qualityassessment was changed from B to GA within both
WikiPro-ject Pok´emon and
WikiProject Video Games . This examplealso illustrates a quirk of conventions on Wikipedia: veryoften, articles pass to GA or FA directly from B, skipping A.The majority of WikiProjects rarely use the A class qualityassessment.
Data
Our analysis combines multiple data sets from theEnglish-language Wikipedia [32]. For information aboutedit history, we used a publicly-available data set contain-ing metadata (time, article id, user) about all edits from July Figure 1: From Wikipedia
Knitting talk page. Two WikiPro-jects have assessed the article as B-class quality.12, 2006 to December 2, 2015. We used a custom scriptto scrape article quality assessments from logs produced byWP 1.0 Bot for 2279 unique WikiProjects between May 4,2006 and December 2, 2015. Finally, we used a publicly-available database dump of page events (including renameevents) to reconstruct the article id for each title mentionedin the assessment logs.
Efficiency and Performance
When individuals collaborate to solve a problem, thereare many ways to gauge their success. One possibility is ef-ficiency : how quickly they find a solution. Another is perfor-mance : how good their solution is. Evidence from numeri-cal simulations [26, 28, 27, 17, 3], lab studies [22], and fieldobservations [13] all suggest a trade-off between efficiencyand performance. While common, this trade-off is not abso-lute, suggesting it is sometimes possible to simultaneouslyincrease performance and efficiency. The identification offactors associated with both higher efficiency and higherperformance has obvious practical importance. In this pa-per, we focus on how network structure relates to efficiencyand performance within WikiProjects.For a WikiProject, efficiency quantifies how much partic-ipants can raise the assessed quality of an article for a fixedamount of work. We measure work by the number of revi-sions made. Quality assessments are made through consen-sus of the project participants themselves. Different projectscan have different standards and practices for assessing arti-cle quality, so the efficiency is not a measure of how quicklysome objective measure of quality improves, but rather ofhow quickly the project participants can reach consensuson the improvements that need to be made and make thoseimprovements. Because our definition relies on assessmenttransitions, we must define efficiency variables for each of3he project-level quality assessments: A, B, and C. For a par-ticular grade G , we desire our definition of efficiency to meetthe following conditions: • Strictly increasing in the number of articles reachinggrade G (with revision count fixed); • Strictly decreasing in the number of revisions (withtransition count fixed); • Independent of WikiProject size: not affected by addingan article having the same efficiency.We now define an efficiency measure which meets theabove criteria. Let T ( W, G ) be the set of article assessmenttransitions from below grade G to grade G or higher inproject W . Let N ( W, G ) be the number of articles in project W which ever transition from below grade G to grade G (orhigher). Given a transition t , let r ( t ) be the number of revi-sions to the article since its previous grade transition, and let g ( t ) be the number of grade levels crossed bt t . We quantifythe efficiency E ( W, G ) as the inverse of the mean numberof revisions per transition: E ( W, G ) = N ( W, G ) (cid:88) t ∈ T ( W,G ) r ( t ) g ( t ) − , (1)where the g ( t ) term accounts for assessments that raise arti-cle quality by several grades by dividing the revisions evenlybetween all grade levels achieved.For performance, we wish to quantify how good articlestend to be when they reach a stable state. Measuring per-formance is difficult for two reasons: there is no objectivemeasure of article quality available, and articles are alwayschanging, making it difficult to know which articles shouldbe considered complete or stable. We use an extremely sim-ple performance measure that gives surprisingly consistentresults. In addition to per-project quality assessment, arti-cles can be given “featured article” or “good article” sta-tus. The criteria for these statuses are consistent across allof Wikipedia, and any editor can participate in the discus-sion and decision to award good or featured status. In otherwords, the good and featured statuses are less subjective thanper-project assessments.Our performance measure P ( W ) is defined simply as thepercentage of articles in project W which have reached goodor featured status: P ( W ) = f ( W ) + g ( W ) n ( W ) , (2)where f ( W ) and g ( W ) are the numbered of featured andgood articles respectively, and n ( W ) is the total number ofarticles. Coeditor Networks
We would like to determine how the social networkstructure of Wikipedia—the pattern of who interacts withwhom—relates to efficiency and performance. There areseveral types of interactions we could focus on, including:coediting, user talk messages, and talk page replies. Wechoose to focus on coediting: when two editors have madechanges to the same article or talk page. While editors cancommunicate directly through user talk messages, the num-ber of such messages is small compared to the number ofedits to article and talk pages. We also could have consid-ered direct replies between editors on article talk pages, butthese replies are typically seen (and intended to be seen) byeveryone reading the talk page, and are part of larger conver-sations. When an editor views a page, they are potentiallyviewing content from and interactions between all editorswho came before them, motivating our choice to focus onthe social network structure of coeditors.The coeditor network of a WikiProject consists of nodesrepresenting editors and edges connecting pairs of editorswho have edited the same article. The edges are directed,with the direction representing plausible information flow ;an edge from Alice to Bob exists if Alice edited an articleand then Bob edited the same article at a later time. Notethat edges can exist in both directions. We make the simpli-fying assumption of unit weight for all edges. We focus onthree structural properties: degree, characteristic path length,and min-cut. Degree and characteristic path length have beenshown to correlate with performance and efficiency in somesocial learning settings [15, 28, 17], while min-cut can be in-terpreted as a measure of decentralization, common featureof peer-produced communities such as Wikipedia [4].The degree distribution is the simplest network propertywe analyze. The in-degree (out-degree) of a node is the num-ber of edges to (from) that node. Taking the average of ei-ther in-degree or out-degree gives the same value: the meandegree of the network. In our context, the mean degree rep-resents, on average, how many other editors each editor hascollaborated with. We also consider the skewness of the in-degree and out-degree distributions. A large positive degreeskewness for a WikiProject coeditor network implies that asmall number of editors have a very large number of collab-orators, while a small positive value implies that the editorshaving the most collaborators don’t have many more than atypical editor.We also calculate the characteristic path length for eachWikiProject coeditor network. The distance from node s tonode t is the distance of the shortest path from s to t . The characteristic path length (or just path length ) is the meandistance between all editor pairs, excluding unconnectedpairs. To account for unconnected nodes, we also measurethe connected fraction : the fraction of ordered node pairswith a directed path from source to sink. The path lengthrepresents how quickly information can move through the4etwork. Networks with longer paths require more interac-tions for information to propagate, which has been shown toreduce efficiency in some settings [28, 3].Our final network measure quantifies the connectivity of aproject’s coeditor network using min-cut size. The minimum st -cut between nodes s and t is the set of edges that mustberemoved for no path exists from s to t . The minimum cut(min-cut) of a graph is the smallest minimum st -cut over allnode pairs st . The size of the graph min-cut quantifies theconnectivity of a graph, but only incorporates informationabout edges lying on paths crossing the min-cut. Instead, weuse the mean size of all minimum st -cuts, which we referto as the mean min-cut . This measure quantifies the num-ber of redundant paths information can take through the net-work. Networks with higher redundancy are more resilientto errors on one path [1] and allow innovations to propagatethrough complex contagion, in which innovations are onlyadopted after multiple exposures [6].The mean path and min-cut are computationally intensive,requiring distance and minimum st -cut calculations for allnode pairs. For larger projects, these calculations are im-practical and we thus employed sampling to determine meanpath length and mean min-cut. For mean path length, sourcenodes were sampled, and path length was calculated to alldestination nodes from each of these. For min-cut, nodepairs were sampled. In both cases, stratification was used toensure the same number of nodes were were sampled fromeach of 12 node degree quantiles. We estimated the error dueto sampling by determining true values for a medium-sizedproject, and calculating error as a function of sample-size.Sample sizes were chosen such that relative error was be-low 10%. Even with sampling, however, it was impracticalto calculate these properties for the largest projects, so weexclude the 183 largest projects from the analysis. Empirical Results
We find that both efficiency and performance are highlyright-skewed, with a small number of projects having val-ues much higher than the average. After log-transformingthe values, both the efficiency and the performance have aunimodal distribution with low skew (see Figure 2). Ourfindings confirm the trade-off between performance and effi-ciency observed in many other settings (Figure 3). However,when looking at specific projects, some are higher in bothperformance and efficiency, suggesting the existence of fac-tors which correlate positively with both.We also find that mean min-cut is highly correlated withdegree ( r = 0 . , p < . ), so we exclude min-cut fromregression models to prevent collinearity. The high correla-tion between mean degree and min-cut implies that, in mostcases, the minimum st -cut is simply the set of edges from s or the set of edges to t . The rarity of non-trivial min-cutssuggests that WikiProject coeditor networks have very fewcentral bottlenecks and are thus highly decentralized. A-Efficiency050100150200 P r o j e c t c o un t B-Efficiency050100150200250 P r o j e c t c o un t C-Efficiency0100200300 P r o j e c t c o un t Performance050100150 P r o j e c t c o un t Figure 2: Histograms of WikiProject efficiency and per-formance. Both measures are highly right-skewed, but formunimodal distributions with low skewness after log transfor-mation. B-Efficiency10 P e r f o r m a n c e r = -0.12, p<0.001 Figure 3: WikiProject performance is anticorrelated with B-level efficiency, with Pearson r of -0.12. Results are similarfor other grade levels. On average, highly efficient WikiPro-jects are under-performing, but when looking at specificWikiProjects, some are higher than others in both perfor-mance and efficiency.To study the relationship between network structure, ef-ficiency, and performance, we model the performance andefficiency of WikiProjects using ordinary least-squares lin-ear regression. Each WikiProject is a single observation.The models include each project’s coeditor network proper-ties as independent variables. We also include the followingproject-level variables to control for confounding factors.
C-efficiency (performance only). Quantifies how quickly aWikiProject improves articles. Efficiencies for differentgrades are highly correlated, so we include only one.
Connected fraction.
Fraction of coeditor pairs connectedby a path.
Talk fraction.
Fraction of total revisions made to talk5 erf † A-Eff † B-Eff † C-Eff † Mean degree † -0.7 ∗∗∗ -0.8 ∗∗∗ -0.6 ∗∗∗ -0.3 ∗ Out degree skew † -0.4 ∗∗∗ -0.5 ∗∗ -0.3 ∗ -0.06Mean path length † -0.33 ∗∗∗ -0.09 -0.05 -0.09C-Efficiency † -0.08 ∗ — — —Connected frac. 0.01 0.09 ∗ ∗∗∗ † † ∗∗ -0.03 0.01 0.02Mean editors/art. † ∗∗ ∗ † -0.4 0.7 ∗ ∗∗ ∗∗ Editor count † ∗∗ ∗∗ ∗ Revision count † ∗ -1 ∗∗ -1.1 ∗∗∗ -1 ∗∗∗ First assessment 0.05 0.11 ∗∗ ∗∗∗ ∗∗∗ Mean article age -0.03 -0.04 -0.01 -0.05 ∗ N 1179 966 1260 1415R adj † Log-transformed. * p < . . ** p < . . *** p < . . Table 1: Standardized coefficients for OLS models.pages.
Mean similarity.
Mean Jaccard similarity (by article) withother WikiProjects; a measure of topical complexity.
Mean editors/article.
Mean number of editors collaborat-ing on each article in a WikiProject.
Article count.
Total number of articles in the WikiProject.
Editor count.
Total number of editors working on articleswithin a WikiProject.
Revision count.
Total number of revisions to articles in aWikiProject.
First assessment.
Timestamp of first assessment; a mea-sure of how long a WikiProject has been active.
Mean article age.
Mean age of articles within a WikiPro-ject.Our models are summarized in Table 1. Min-cut is ex-cluded from all models to avoid collinearity, as it is highlycorrelated with degree. In-degree and out-degree skew-ness were also highly correlated, so we only include out-degree skewness (results are similar for in-degree skewness).Heavy-tailed variables are log-transformed. To test the ro-bustness of our results, we also computed models using cuberoot instead of logarithmic transformations, and using onlytop- and high-importance articles. The results were quali-tatively similar results for all variables, except for degree-skewness, which had an inconsistent sign across models.We see that B-efficiency and C-efficiency have very sim-ilar models, but that A-efficiency behaves differently in itsdependence on degree skewness and connectivity. The dif-ferent behavior of A-efficiency is likely explained by theobservation that the A-Class quality is infrequently used inpractice. The A-Class quality level is usually passed when an article reaches good or featured article status, which fol-low deifferent a consensus process from other ratings.The negative dependence of performance on C-efficiencysuggests there is generally a trade-off between performanceand efficiency. However, low degree is correlated with bothhigher efficiency and higher performance, suggesting that itis possible to improve both simultaneously. Much of the ex-isting numerical work on networked social learning focuseson path length rather than degree, so we explore this resultfurther using simulations in the next section.For path length, we find that longer lengths correspondto lower performance, contrary to the conjecture that longerpath lengths allow more exploration [28] but consistent witha conformity-based social learning strategy [3].We also observe that high degree skewness is correlatedwith lower performance and lower A-efficiency, suggestingthat articles in projects with decentralized coeditor networksreach featured or good status more efficiently, and reachhigher quality ratings in general.
Agent-Based Model
In addition to our empirical study, we use a simple agent-based model of collaboration to better understand the rela-tionship between node degree, efficiency, and performance.Numerical models allow us to determine the effect of chang-ing a single variable (e.g., network structure, learning strat-egy), which is impractical in the empirical setting. It is im-portant to note that the goal of our model is not to simulateall the intricacies of Wikipedia or any other specific plat-form. Rather, our goal is to determine whether the correla-tions we observe between degree and outcome variables onWikipedia can be reproduced in a more general setting.Past work in the field of social learning typically mod-els collaboration as an optimization problem: finding a stateof the world which maximizes some objective function[26, 28, 27, 3]. Wikipedia itself can be regarded as an opti-mization problem. On Wikipedia, editors are generally seek-ing to improve the quality of articles and have some personalpreference over possible states of an article. When editorsdo not agree on the optimal state of an article, the conflict isresolved through a consensus-based deliberation. This con-sensus process can be regarded as a social choice function [2, 5] which maps individual preferences to community pref-erences. Wikipedia can thus be thought of as a group of ed-itors with individual preferences for article states, collabo-rating to optimize articles according to community prefer-ences. Note that these community preferences do not assumethe existence of any ground truth, other than the preferencesthemselves.To simulate collaboration, we need a model problem forcollaborators to solve. Following existing literature on sociallearning, we use the NK model [21] to create NP-hard, non-linear optimization problems. The NK model produces anobjective function with a rugged landscape , i.e., many local6 ame Social stage Individual stage Limited concern Unknown objective Single truthBest+I Best neighbor GlobalConf+I Conformity Global ! Best+LI Best neighbor Local ! Conf+LI Conformity Local ! !
LMaj+LI Local majority Local ! ! !
Table 2: Definitions and properties of social learning strategies. Each consists of a social stage and an individual stage. Indi-vidual stages use hill-climbing based on either the global state, or the agent’s local concern.optima. The ruggedness of the model can be tuned throughthe parameters N (the dimensionality of the solution space)and K (the level of inderdependence between dimensions).Formally, the NK model produces an objective function F mapping a binary string S of length N to a real value in [0 , . Model state is divided into N loci , with locus i hav-ing a binary state S i and a value f i ( S ) dependent on its ownstate and on the state of K random other loci. The functions f i ( S ) are created by selecting a random value in [0 , foreach possible state of locus i and its K neighbors. The valueof the model F ( S ) is the mean of all locus values f i ( S ) . Inour simulations, agents iteratively search for a bit string S that maximizes F ( S ) .In a typical social learning model, a set of agents eachmaintain an estimate of the optimal state and iteratively up-date that estimate based on information available from otheragents, according to some learning strategy . In networkedsocial learning, agents are associated with the nodes of a net-work and share information only with their neighbors. Wedefine efficiency and performance in terms of the solutionvalues for each time step (averaged over many trials). We de-fine the performance to be the mean solution value after theprocess has converged, while the efficiency is the reciprocalof the number of steps required to converge. We measure thetime to convergence as the number of steps required to reach99% of the maximum mean solution value.Without additional constraints, the above model is miss-ing several key properties of real-world collaborations. Indesigning our agent-based model, we paid attention to thefollowing properties. Limited concern.
Agents are concerned only with a subsetof the entire state when making decisions and determin-ing preferences. (On Wikipedia, editors typically inter-act with a small subset of the articles.)
Concern-based network.
Agents interact with otheragents who share a common concern over some subsetof the state. (On Wikipedia, editors interact with otherswho share interests in the same articles.)
Unknown objective.
Agents rank states in order of pref-erence, but do not have access to the objective func-tion. (On Wikipedia, there is no ground truth measureof quality.)
Single source of truth.
At any given time, the system is ina single state and agent preferences are based on localmodifications to that state. (At any point in time, thereis only one current version of Wikipedia.)
Concern-Based Networks
On Wikipedia, editors interact by editing articles and talkpages. Thus, the editors who interact with each other are ex-actly those who care about the same content. Rather thanusing arbitrary networks, we devise a network structure in-spired by the above observation. We do so by associatingagents with particular loci in the NK model. We also wishto study the effect of varying network degree, which weachieve through a rewiring process described below.Our concern-based networks are generated directly fromthe structure of the NK model. The value of each NK locusdepends on its own state and the state of K other loci. Foreach locus, we define an agent and assign these K + 1 locias its concern. Next, an agent-agent co-affiliation networkis created by connecting two agents if they share at leastone locus in their concerns. This process is analogous to ourconstruction of WikiProject coeditor networks.To create a tunable degree, we duplicate each agent and itsconcern, then randomly rewire a fraction of agent concernsbefore creating the agent-agent network. With no rewiring,the duplication process creates a high overlap between agentconcerns. This overlap results in redundant links to a smallnumber of agents, rather than unique links to a large num-ber of agents, and therefore to an agent-agent network withsmall average degree. By randomly rewiring the agent con-cerns, the redundancy is reduced and the average degree ofthe agent-agent network is increased. Networked Learning Strategies
Learning strategies determine how agents update theirpreferences based on available information [3]. Agents canengage in individual learning by applying a hill-climbing al-gorithm to their current solution. In each iteration, one bitof the NK solution string is flipped to maximize the solutionvalue. If no change improves the value, the original solutionis kept. The above strategy relies only on rankings of states,satisfying the unknown objective assumption. However, itrelies on information about the entire state, violating the lim-ited concern assumption. In order to satisfy this assumption,7e also define a local variant in which only a subset of bits inthe NK solution string are considered. This variant reflectsa more realistic style of collaboration, in which individualagents focus on sub-problems.In social learning, agents can also incorporate informationfrom other agents they are connected to by an edge. Whileindividual learning always converges to the local maximumrelative to the starting point, social learning strategies al-low agents to “jump” to drastically different solutions withhigher local maxima. In our model, we use both the con-formity and best-neighbor strategies from [3]. In the best-neighbor strategy, each agent compares its solution to a sam-ple of neighbors, and chooses the solution with the highestvalue. In order to compare solutions between neighbors, theexact value of the objective function must be known for eachsolution, so this strategy does not satisfy the unknown ob-jective assumption or the limited concern assumption. In the conformity strategy, agents simply choose the most commonsolution among their neighbors (ties are broken uniformlyat random). This strategy does not rely on solution value atall, so clearly satisfies the unknown objective and limitedconcern assumptions. In both cases, a single iteration of in-dividual learning is performed after each social learning it-eration. Because each agent maintains a separate estimateof the solution, neither strategy satisfies the single source oftruth assumption.
Local Majority Strategy
To satisfy the single source of truth assumption, we intro-duce a new strategy: local majority . In local majority, agentsall begin with the same starting state and apply individuallearning to their concern to generate possible improvementsto the solution. Next, a new solution is constructed by con-sidering each locus of the NK solution individually. Everyagent concerned with a locus votes for its state based on theirpreferred new solution and the majority state is chosen. Theresult of this process is that all agents integrate their solu-tions into a single state, which forms the basis for the nextiteration. This strategy more realistically reflects collabora-tions like Wikipedia: at any given time, a Wikipedia articlehas a single state, determined by consensus, but editors mayhave differing opinions on how to improve that article.
Simulation results
We simulated 100 trials for rewiring values of 0.0, 0.167,0.333, 0.5, 0.667, 0.833, and 1.0. For each trial we gener-ated an NK model with N = 250 and K = 7 , generated aconcern-based network, and ran each social learning strat-egy (Table 2) for 300 iterations. For conformity and best-neighbor strategy, we used a sample size of 3, following[3]. We confirmed that all trials converged to their maxi-mum value before reaching the last iteration. Networks hadmean degree 116.6 with 1.3 standard deviation, and meanpath length of 1.766 with 0.0027 standard deviation. The Iteration V a l u e P[rewire] = 0.00
Best+IBest+LIConf+IConf+LILMaj+I 0 200
Iteration V a l u e P[rewire] = 1.00
Best+IBest+LIConf+IConf+LILMaj+I
Figure 4: Mean agent solution value over time, averagedover 100 trials. Strategies are defined in Table 2.
Strategy Performance EfficiencyBest+I 0.722 ± ± ± ± ± ± ± ± ± ± Table 3: Simulated Performance and Efficiency. Resultsshown for 100 trials with P[rewire] = 0. Strategies are de-fined in Table 2. Local strategies are less efficient than theirnon-local counterparts. Local best-neighbor out-performsglobal, while local conformity is the worst performer in allcases. The local majority strategy is both most efficient andmost performant.coefficient of variation for degree is approximately 10%,while only 1% for mean path length, confirming that therewiring process has a stronger influence on degree than onpath length.Figure 4 shows how agents’ solutions improve afterrepeated applications of different learning strategies andrewiring values. Each curve represents an average over 100trials, each with 250 agents. The mean performance and effi-ciency are reported in Table 3. For all rewiring values, localstrategies are less efficient and more performant than theirnon-local counterparts. For the best-neighbor strategy, lo-cal outperforms global. Local conformity performs notablyworse than all others. Local majority is both more efficientand more performant than others, with its performance in-creasing with higher rewiring. This implies that, at least ina simple collaboration model, performance and efficiency
Strategy Perf. Std. Coeff. Eff. Std. Coeff.Best+I -4.2 × − × − Conf+I 2.7 × − × − Best+LI -9.6 × − ∗∗ × − Conf+LI -1.5 × − ∗∗∗ × − ∗ LMaj+LI 1.2 × − ∗∗ -0.038 ∗∗∗ * p < . . ** p < . . *** p < . . Table 4: Degree regression coefficients for simulations.8
15 120
Degree E ff i c i e n c y Best+I
115 120
Degree E ff i c i e n c y Best+LI
115 120
Degree E ff i c i e n c y Conf+I
115 120
Degree E ff i c i e n c y Conf+LI
115 120
Degree E ff i c i e n c y LMaj+I
115 120
Degree P e r f o r m a n c e Best+I
115 120
Degree P e r f o r m a n c e Best+LI
115 120
Degree P e r f o r m a n c e Conf+I
115 120
Degree P e r f o r m a n c e Conf+LI
115 120
Degree P e r f o r m a n c e LMaj+I
Figure 5: Efficiency and Performance of social learning strategies vs. mean network degree. Each point represents a singletrial of 300 iterations. Strategies are defined in Table 2. The local best-neighbor strategy shows decreased performance at highdegree, with no significant change in efficiency. Local conformity shows decreased performance and increased efficiency athigh degree. Local majority shows the opposite behavior: increased performance and decreased efficiency at high degree, withthe efficiency showing the largest effect size of all strategies.can be simultaneously increased. Furthermore, performanceand efficiency are potentially affected by both the choice oflearning strategy and the average degree of the agents’ socialnetwork.The effects of degree on performance and efficiency areshown in Figure 5 and Table 4. For non-local versions ofboth conformity and best-neighbor strategy, there is no sig-nificant effect of degree on performance or efficiency. Thelocal best-neighbor strategy shows reduced performancewith increasing degree, but no change in efficiency. Localconformity and local majority show opposite behavior as de-gree increases: with local conformity gaining efficiency atthe expense of performance, while local majority increasesin performance and decreases in efficiency. The largest ef-fect size is achieved for efficiency in the local majority sim-ulation, which is consistent with the efficiency behavior ob-served in WikiProjects. However, the performance behaviorfor local majority is opposite that observed on Wikipedia.These agent-based models confirm that network degree hasthe potential to influence the performance and efficiency ofcollaborations. Furthermore, this influence can be drasticallydifferent depending on the strategies used by collaborators.
Discussion
While existing research into the role of network struc-ture in collaboration has focused on numerical simulationsand lab experiments, analysis of large real-world systems isan important next step. Our empirical analysis contributesseveral findings towards a better understanding of large, de-centralized, real-world collaboration. We observe several re- sults consistent with previous work: a trade-off between per-formance and efficiency [28, 17], higher performance forshorter path lengths in a conformity setting [3], and a re-duction in performance with increased structural inequality[22]. By using real-world networks, we were also able toanalyze network properties independently. While most ex-isting work has focused on the importance of path length,our findings suggest degree distribution may be just as, ormore, important. The association of low degree with bothhigh performance and high efficiency is compelling, as itsidesteps the usual trade-off between performance and effi-ciency. In low-degree networks, agents have more repeatedinteractions with smaller groups of collaborators, suggestingthat small team sizes could be beneficial for large collabora-tions. Similarly, the observation that performance is higherin projects with less structural inequality suggests that, if thechallenges of egalitarian organizing are overcome, decen-tralized collaborations may produce better outcomes thanthose with centralized, top-down structures.Our agent-based models offer a several insights. We ob-serve degree-dependent performance and efficiency for bothlocal conformity and local majority strategies. However,these two strategies have opposite degree dependence, sug-gesting that different strategies may be preferable for high-degree and low-degree networks. Our local majority strat-egy, designed to satisfy several properties found in real-world collaborations, shows the strongest effects on per-formance and efficiency as network degree changes. Forthe local majority strategy, the relationship between degreeand efficiency is consistent with our empirical observations9n Wikipedia, suggesting one possible mechanism underly-ing that efficiency dependence. However, the performancedependence of this strategy is opposite that observed onWikipedia, suggesting that either the local majority strategyis incompatible with actual behavior on Wikipedia or thatother factors outweigh the contribution of mean degree.Our work has several limitations. Our empirical analysisis purely correlative and cannot be used to draw conclusionsabout the causal influence of network structure on collabora-tion. However, the consistency of our results with other lab-based and numerical studies suggests that the causal link isworthy of further study. Similarly, our study focuses entirelyon a single online community, and while the results are sug-gestive, they do not necessarily generalize. We have focusedon structure, ignoring content-related variables. For simplic-ity, we have assumed unweighted edges and measured workby revision counts rather than bytes changed.Our work suggests several directions for future work. Isthe correlation between network structure, performance, andefficiency causal? A time-dependent analysis of our datacould offer insight. Are similar relationships observed inother large-scale collaborations? Does varying degree in-dependently of path length influence performance and ef-ficiency in a controlled lab setting?A better understanding of the relationship between net-work structure and collaboration outcomes has practical ap-plications. Online communities using recommender systemscould make recommendations guided by desirable networkproperties. Similarly, network structure could be used toidentify under-performing groups in need of an interven-tion. The relationship between network structure and learn-ing strategy suggests that behaviors interact with networkstructure, which could be used to encourage behaviors com-plementary to existing network structure.
Conclusion
In this paper, we have described the relationship betweenthe structural properties of WikiProject coeditor networks,their performance, and their efficiency. As in other stud-ies, we see a trade-off between performance and efficiency.However, some properties, such as low degree, are associ-ated with both higher performance and higher efficiency.We also find that the correlations between path length andperformance are consistent with a conformity-based sociallearning strategy, but not a greedy best-neighbor strategy.We observe improved performance in more decentralizedprojects, as has been seen in small-scale lab experiments. Wehave also proposed a novel local majority learning strategythat is more realistic, more efficient, and higher performancethan existing strategies. While most previous social learn-ing simulations focus on path length, we observe degree-dependent performance and efficiency in both the local ma-jority strategy and a localized version of the conformitystrategy. We find that the direction of that dependence varies with the specific strategy being used. While additional workis needed to determine causal relationships and the general-izability of our results, we have shown evidence that severalphenomena predicted by numerical and small-scale lab ex-periments are present in a large, real-world collaboration.Our results suggest that the success of large-scale collabo-rations may be aided by greater decentralization, consensusor conformity-based decision-making, and more tightly-knitcollaborations between smaller teams.
Acknowledgments
Thanks to Ceren Budak, Scott E. Page, Yan Chen, TanyaRosenblat, anonymous reviewers, and the attendees of theMay 25, 2017 MIT Center for Civic Media lab meeting andthe Berkman-Klein Center Cooperation Working Group forhelpful feedback; and to Danielle Livneh and Karthik Ra-manathan for help collecting the data. This research waspartly supported by the National Science Foundation underGrant No. IIS-1617820.
References [1] Albert, R., Jeong, H., and Barab´asi, A.-L. (2000). Er-ror and attack tolerance of complex networks.
Nature ,406(6794).[2] Arrow, K. J. (2012).
Social choice and individual val-ues , volume 12. Yale University Press.[3] Barkoczi, D. and Galesic, M. (2016). Social learningstrategies modify the effect of network structure on groupperformance.
Nature , 7.[4] Benkler, Y. (2006).
The wealth of networks: how socialproduction transforms markets and freedom . Yale Uni-versity Press, New Haven [Conn.].[5] Brandt, F., Conitzer, V., and Endriss, U. (2012). Com-putational social choice.
Multiagent systems , pages 213–283.[6] Centola, D. and Macy, M. (2007). Complex contagionsand the weakness of long ties.
Am J Sociol , 113(3).[7] Contributors (2017). Wikipedia:Good articles.[8] Contributors (2018). Wikipedia:Featured articles.[9] DeGroot, M. H. (1974). Reaching a consensus.
Jour-nal of the American Statistical Association , 69(345):118–121.[10] Elsharbaty, S. (2016). Editing Wikipedia for a decade:Gareth Owen.[11] Fowler, J. H. and Christakis, N. A. (2010). Coopera-tive behavior cascades in human social networks.
PNAS ,107(12):5334–5338.1012] Freeman, J. (1972). The tyranny of structurelessness.
Berkeley Journal of Sociology , 17:151–164.[13] Gentry, M. E. (1982). Consensus as a form of decisionmaking.
J. Soc. & Soc. Welfare , 9:233.[14] Giles, J. (2005).
Internet encyclopaedias go head tohead . Nature Publishing Group.[15] Golub, B. and Jackson, M. O. (2010). Naive learningin social networks and the wisdom of crowds.
Am EconJ-Microecon , 2(1):112–149.[16] Granovetter, M. S. (1973). The strength of weak ties.
Am J Sociol , pages 1360–1380.[17] Grim, P., Singer, D. J., Fisher, S., Bramson, A., Berger,W. J., Reade, C., Flocken, C., and Sales, A. (2013).scientific networks on data landscapes: question diffi-culty, epistemic success, and convergence.
Episteme ,10(04):441–464.[18] Halfaker, A., Geiger, R. S., Morgan, J. T., and Riedl,J. (2013). The rise and decline of an open collaborationsystem: How Wikipedias reaction to popularity is causingits decline.
Am Behav Sci , 57(5).[19] Hong, L. and Page, S. (2009). Interpreted and gener-ated signals.
Journal of Economic Theory , 144(5):2174–2196.[20] Hong, L. and Page, S. E. (2004). Groups of diverseproblem solvers can outperform groups of high-abilityproblem solvers.
PNAS , 101(46):16385–16389.[21] Kauffman, S. and Levin, S. (1987). Towards a generaltheory of adaptive walks on rugged landscapes.
J TheorBiol , 128(1):11–45.[22] Kearns, M. (2012). Experiments in social computation.
Communications of the ACM , 55(10):56–67.[23] Kearns, M., Suri, S., and Montfort, N. (2006). An ex-perimental study of the coloring problem on human sub-ject networks.
Science , 313(5788):824–827.[24] Keegan, B. and Fiesler, C. (2017). The Evolution andConsequences of Peer Producing Wikipedia’s Rules. In
ICWSM .[25] Kittur, A. and Kraut, R. E. (2008). Harnessing the wis-dom of crowds in wikipedia: quality through coordina-tion. In
CSCW . ACM.[26] Lazer, D. and Friedman, A. (2007). The network struc-ture of exploration and exploitation.
Admin Sci Quart ,52(4).[27] Mason, W. and Watts, D. J. (2012). Collaborativelearning in networks.
PNAS , 109(3). [28] Mason, W. A., Jones, A., and Goldstone, R. L. (2008).Propagation of innovations in networked groups.
Journalof Experimental Psychology: General , 137(3):422.[29] Michels, R. (1999).
Political parties a sociologicalstudy of the oligarchical tendencies of modern democ-racy . Transaction Publishers, New Brunswick, N.J.,U.S.A.[30] Morgan, J. T., Gilbert, M., McDonald, D. W., andZachry, M. (2013). Project talk: Coordination work andgroup membership in WikiProjects. In
OpenSym . ACM.[31] Newman, M. E. (2003). The structure and function ofcomplex networks.
SIAM Review , 45(2):167–256.[32] Platt, E. L., Livneh, D., Ramanathan, K.,and Romero, D. M. (2018). English WikiPro-ject coeditor networks and quality assessments.https://dx.doi.org/10.7302/Z2610XJB.[33] Robert, L. and Romero, D. M. (2015). Crowd size,diversity and performance. In
CHI . ACM.[34] Shaw, A. and Hill, B. M. (2014). Laboratories of oli-garchy? How the iron law extends to peer production.
JCommun , 64(2):215–238.[35] Silverman, S. F. (1965). Patronage and community-nation relationships in central Italy.
Ethnology , 4(2):172–189.[36] Suri, S. and Watts, D. J. (2011). Cooperation andcontagion in web-based, networked public goods experi-ments.