Facilitating Asynchronous Participatory Design of Open Source Software: Bringing End Users into the Loop
FFacilitating Asynchronous Participatory Design of Open SourceSoftware: Bringing End Users into the Loop
Jazlyn Hellman [email protected] UniversityMontreal, Quebec, Canada
Jinghui Cheng [email protected] MontrealMontreal, Quebec, Canada
Jin L.C. Guo [email protected] UniversityMontreal, Quebec, Canada
ABSTRACT
As open source software (OSS) becomes increasingly mature andpopular, there are significant challenges with properly accountingfor usability concerns for the diverse end users. Participatory design,where multiple stakeholders collaborate on iterating the design,can be an efficient way to address the usability concerns for OSSprojects. However, barriers such as a code-centric mindset andinsufficient tool support often prevent OSS teams from effectivelyincluding end users in participatory design methods. This paperproposes preliminary contributions to this problem through theuser-centered exploration of (1) a set of design guidelines thatcapture the needs of OSS participatory design tools, (2) two personasthat represent the characteristics of OSS designers and end users,and (3) a low-fidelity prototype tool for end user involvement inOSS projects. This work paves the road for future studies abouttool design that would eventually help improve OSS usability.
CCS CONCEPTS • Human-centered computing → Participatory design ; Com-puter supported cooperative work ; •
Software and its engineering → Open source model . KEYWORDS open source software, usability, participatory design, asynchronouscollaboration
ACM Reference Format:
Jazlyn Hellman, Jinghui Cheng, and Jin L.C. Guo. 2021. Facilitating Asyn-chronous Participatory Design of Open Source Software: Bringing EndUsers into the Loop. In
CHI Conference on Human Factors in ComputingSystems Extended Abstracts (CHI ’21 Extended Abstracts), May 8–13, 2021,Yokohama, Japan.
ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3411763.3451643
With the increasing maturity and popularity of open source soft-ware (OSS), ongoing efforts to increase the usability of the softwaredeveloped under the open source model are growing in impor-tance [16]. Usability refers to the attributes which determine the
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
Conference’17, July 2017, Washington, DC, USA © 2021 Association for Computing Machinery.ACM ISBN 978-1-4503-8095-9/21/05...$15.00https://doi.org/10.1145/3411763.3451643 ease, error-prevention, efficiency, and pleasantness for an end userwhen interacting with a software [2, 8]. In traditional softwaredevelopment, usability concerns rest in the hands of design teamsperforming activities that involve various stakeholders in the itera-tive lifespan of a project to achieve a system design that respondsto the needs of the end users. Much of this practice relies on theprinciples of participatory design, which places emphasis on theinvolvement and collaboration of all stakeholders, especially endusers [6]. While user participation in the design process is vital forachieving successful usability of an OSS, it is often pushed asideas teams adapt to asynchronous, remote working and focus on aproject’s code and functionality [13, 16].Take for example GitHub, one of the most successful OSS hostingand development platform. Due to its affinity for supporting variousasynchronous activities, GitHub has enabled a vibrant communityfor remote collaboration and development of OSS projects. Whilethere are many tools that can potentially be used for monitoringusability interests on GitHub OSS’s, such as Issue tracking, thereis currently no mature method for end user community collabora-tion [13, 15, 16]. Consequently, if any design decisions are madethrough input from end users, there are difficulties with properlycommunicating these decisions to other team members due to alack of design artifacts clearly documenting the process. Many OSSsystems, as a result, are designed without direct input from endusers. While prior work explored the pitfalls of end user communityinvolvement in OSS design [1, 12, 13, 16], as of yet, there have beenlittle contribution to the open source community offering a solutionto this problem.In this paper, we address this gap by exploring the design ofa low-fidelity prototype to facilitate asynchronous participatorydesign in large-scale OSS projects. Our design decisions were origi-nally informed by related work on challenges of addressing OSSusability from the developers’ perspective and later iterated throughpreliminary user studies with both OSS designers and end users.Through this study, we contribute an initial set of personas anddesign guidelines for designing platforms that promote end users’participation in asynchronous, participatory OSS design, as well asa preliminary tool to achieve this primary design objective. Thiswork lays the foundation for future studies of more full-fledgedtools that would eventually improve OSS usability.
Usability in the context of OSS has been a topic of interest fordecades and is constantly shifting in nature. OSS has certainlygarnered certain reputation for being ‘by developers for developers’;prior work explored by Nichols and Twidale (2002) has stated asense of ‘elitism’ among the OSS developers where pride was drawn a r X i v : . [ c s . H C ] F e b onference’17, July 2017, Washington, DC, USA Jazlyn Hellman, Jinghui Cheng, and Jin L.C. Guo Table 1: Design Guidelines for OSS Designers (* indicates challenges gathered during user studies.)
No. Description Targeted Challenges
D1 OSS-D are able to easily engage with end users and understand their needs. User Needs [16]D2 The system integrates design practices into an existing development pipeline. Mindset [16]; Pipeline*D3 OSS-D are able to generate artifacts from interacting with end users. User Diversity [16]; Tracing Artifacts*D4 Design for a project can happen at any stage of an OSS project’s lifecycle. Development [16]D5 The system encourages community-wide involvement in an OSS project. Mindset [16]; User Needs [16]
Table 2: Design Guidelines for OSS End Users (* indicates challenges gathered during user studies.)
No. Description Targeted Challenges
EU1 OSS-EU should easily learn how to collaborate on a project. Inclusivity*, Learnability*EU2 OSS-EU should interact with the OSS team through asynchronous discussions. Development [16], Mindset [16]EU3 OSS-EU should be able to draw attention to current usability issues they face. Mindset [16], User Needs [16]EU4 OSS-EU contributions should be recognized. Transparency*, Recognition*EU5 OSS-EU should feel motivated to collaborate on a project. Inclusivity*, Transparency*
In this section, we detail how we explored the tool design, present-ing the target user groups, the usability goals and heuristics, andthe ideation and overall design process.
To understand the context of when and where issues will be facedduring asynchronous participatory design as well as the nature ofthose issues, we defined two target user groups of our intendedsystem: (1) OSS contributors focusing on the design aspects of OSSprojects (
OSS-D ) and (2) non-technical end users of OSS projects(
OSS-EU ).We selected OSS contributors focused on design (referred to asOSS designers from here on) as the first User Group because: (1)these individuals are the ones most impacted by the lack of stan-dardization in OSS design practices; (2) they would have the mostinsight on current barriers facing participatory design integrationin asynchronous OSS work. In regards to the second user group,we are interested in the case where the end users of the OSS arenot the technical contributors to the OSS project, but rather thedirect user of the end product. These target audiences were used to(1) direct our design decisions and (2) guide participant selectionfor the user studies (see Section 4).
Once the target audiences were finalized, we created a set of de-sign guidelines to support the exploration of the system design.Defining these guidelines was an iterative process. The initial setof guidelines were distilled from Wang et al.’s [16] recent workabout challenges in addressing OSS usability. Then the guidelineswere revised as more information was gleaned through the userstudies of preliminary prototypes (see Section 4). When creatingthese guidelines, we particularly concentrated on the aspects thatcan facilitate asynchronous participatory design in the OSS context.The final design guidelines for the system are summarized in Tables1 and 2, as well as the corresponding challenges identified eitherby Wang et al [16] or our user study.
We design the preliminary system that can be integrated into ex-isting OSS hosting services, such as GitHub . While those servicesinclude various asynchronous collaboration features that are widely https://github.com acilitating Asynchronous Participatory Design of Open Source Software: Bringing End Users into the Loop Conference’17, July 2017, Washington, DC, USA (a) End user interactions with user tagsand user collaboration portal (b) Communication messages designwith pipeline integration (c) Design decision artifact creation and as-signment to contributors Figure 1: Initial Sketches of the System Design adopted by OSS communities, they generally fall short of deliveringthe same level of benefits for asynchronous, social design .Designing the prototype of the system had three overlappingphases. First the research team brainstormed, sketched, and dis-cussed the system design based on the initial design guidelines.Some initial sketches are shown in Figure 1. These sketches focusedon the following main features that reflected the design guidelines. • End User Involvement (see Figure 1a) - This feature captures theguidelines D1, D5, EU1, EU2, EU4, and EU5 and provides a portal,separated from the rest of the code management system, fordesigners to define target end users and their roles, for end usersto sign-up for collaborations, and facilitate direct and intuitiveinteractions between designers and end users. • Pipeline Integration (see Figure 1b) - This feature captures theguidelines D2, D4, and EU3 and provides ability to integrate theproposed system within existing OSS development pipelinesthrough the creation of associated ‘Issues’ from discussions ofusability concerns with end users. • Collaborative Design Artifact Generation (see Figure 1c) - Thisfeature captures the guidelines D3 and EU4 and facilitates the co-creation and feedback collection of design artifacts with the endusers. This feature can be utilized to achieve
Pipeline Integration
We conducted a preliminary user study with two OSS designersand two OSS end users to get feedback on the initial sketches andto iterate the early versions of the prototype. This user study wasapproved by the Ethics Board of the institution of the researchers.
Participants were were recruited from both personal networks andthe Open Source Design forum; Open Source Design is a communityof designers and developers committed to improving design inOSS . The characteristics of the participants are summarized inTable 3. Among the four participants, one was non-binary, one wasfemale, and two were male. Both OSS designers had considerableexperience with OSS design projects; all their latest design projectswere utilized by non-technical end users.We used two different sets of interview questions for the de-signers and end users, respectively, during the one-on-one videoconferencing calls. The OSS Designer interview focused on the par-ticipants’ (a) experience and roles, (b) current tools and processes https://discourse.opensourcedesign.net/ onference’17, July 2017, Washington, DC, USA Jazlyn Hellman, Jinghui Cheng, and Jin L.C. Guo Table 3: User Study Participants’ Characteristics
Participant ID Target Group Experience Current Occupation
P1D OSS Designer Decades working in OSS in the public health sector Design Director/LecturerP2D OSS Designer About one decade working in OSS as lead designer in social enterprises Design LeadP3EU OSS End User Casual OSS end user of Mozilla Firefox, Audacity Project Coordinator AssistantP4EU OSS End User OSS end user Notepad++, Audacity, Gimp PhD Candidate/Lecturer used to achieve OSS participatory design, (c) current challengesfaced in making design decisions, and (d) open ended feedback onthe sketches. Similarly, the OSS End User interview included specificquestions regarding (a) general experience and roles, (b) utilizingOSS, (c) becoming involved in OSS development and challengesthat were faced, and (d) open ended feedback on the sketches.The interviews were recorded and then inductively coded toanalyze themes [5]. Following the analysis, a summary of the maininsights was drafted and used to (1) modify the design guidelines ofthe proposed system, (2) evolve the design of the proposed system,and (3) create the personas to capture the user needs.
In this section, we summarize the key insights from our user stud-ies. We begin with the insights from the interviews with the OSSdesigners:
Designer Insight 1:
There are no current standards for addressingusability concerns with OSS end users and community mem-bers. P2D mentioned that each project they worked on uti-lized different design methods to practice participatory de-sign. Both P1D and P2D utilized many different platforms andtools to practice participatory design (e.g., Figma, Airtable,screen recorded demos, Twitter, emails, Slack, Discourse,video conferencing, etc.).
Designer Insight 2:
Two major consequences resulting from
De-signer Insight 1 were bottlenecks in the pipeline from ad-dressing usability and difficulties in communicating ideas todifferent groups of people on various tasks. Especially withthe asynchronous culture in which P2D works, there wasn’ta set process for providing design artifacts and necessarychanges on the software design were not always accom-plished.
Designer Insight 3:
The OSS designers positively received the ideaof moving usability discussions and interactions betweendesigners and end users out of ‘Issues’, which is already anoverloaded feature on OSS platforms [15]. P2D mentionedthat the developers they collaborated with frequently ig-nored issues flagged as usability concerns and argued withP2D against proposed changes resulting from end user dis-cussions. P2D believed that decluttering the ‘Issues’ wouldlead to less aggravation of developers, potentially resultingin a mindset shift towards more receptive to usability issues.
Designer Insight 4:
Participants discussed that integrating and trac-ing the visual design artifacts in the OSS collaboration plat-forms such as GitHub would be helpful. P1D said: “
Feels likea no-brainer to have a more visual asset visualizer... on GitHub.Our team would greatly benefit in SEEING artifacts that areinherently visual. ” Designer Insight 5:
Participants also emphasized the importanceto “ keep the core GitHub repo concepts ” where “ the DesignLayer should not "interfere" with core engineering activities ”(P1D). They expressed that any additional feature should beconsistent with the existing engineering culture and use ofthe collaboration platforms.Interviews with the OSS End Users brought to light many impor-tant distinctions for our system design that had previously goneunconsidered. Listed below are several key insights.
Eng User Insight 1:
From the end user’s perspective, a major barrierto contributing to an OSS was not knowing how. WhileP4EU had found the GitHub repository for one OSS throughtheir website, they expressed difficulty in understanding howGitHub works and how to word their comments to ensuremutual understanding. For this, P4EU stated GitHub wasoverwhelming and intimidating. P3EU felt an ideal way to beincluded in the design would be through surveys, feedback,or being contacted for participating in a conversation.
Eng User Insight 2:
To address the above issue, the onboarding pipe-line for end users to the proposed system needs to prioritizesimplicity. After viewing the initial sketches, P3EU felt over-whelmed by the onboarding pipeline starting from landingon the main page of a GitHub repository while P4EU ex-pressed minimizing the initial steps an OSS end user wouldneed to perform to provide feedback is crucial to maximizeend user involvement (such as not requiring an account, seeFigure 1a).
Eng User Insight 3:
It is important for the tool design to indicate toOSS end users that they are welcome and that their opin-ions/experiences are valid and helpful. Several key featuresin the sketches can help to accomplish this, as indicated byP4EU, including the ‘end user tag’ features (see Figure 1a) andthe informality of the conversation format (see Figure 1b).Following on this, P4EU also discussed the importance interminology used throughout the system design; in manycases, P4EU felt that the tone of a selected word would deterthem from participating in feedback. For example, the useof ‘Contribute’ (see Figure 1a) was received negatively.
Eng User Insight 4:
Both end user participants felt that it is valuableto see that concrete actions are taken by developers basedon their conversations on usability concerns. They valuedthe feature of linking design artifacts to the discussions (see1c), considering that it would enhance their experience incollaborating and would motivate them for continued efforts.
Eng User Insight 5:
The two interviews illuminated necessary up-dates to the original target audience description for OSSEnd Users. Particularly, this group needs to possess charac-teristics demonstrative of early adopters and innovators of acilitating Asynchronous Participatory Design of Open Source Software: Bringing End Users into the Loop Conference’17, July 2017, Washington, DC, USA (a) OSS Designer Persona Dakota (b) OSS End User Persona Enrique
Figure 2: Personas Created Through the Study technology [3, 4]. This emphasis is important because theseuser groups traditionally are the one’s which provide criticalusability feedback in non-OSS contexts.
During the exploration of the initial design of the system, we de-veloped two personas, one for OSS designers and one for OSS endusers, and iterated them based on the user study results (see Fig-ure 2). The OSS Designer persona is Dakota, a lead designer whostruggles to manage the various platforms for interacting with theirend user community and communicating the decision outcomesfrom those interactions to the rest of their OSS team. The OSSEnd User persona is Enrique, a community coordinator for a lo-cal non-governmental organization (NGO) who is using an OSSto coordinate volunteers; Enrique has experienced issues with theOSS when coordinating volunteers with special needs and he facesdifficulties communicating these issues with the OSS team. We en-vision that these personas can support the design of this and similarsystems that aim at facilitating OSS user-designer communication.We will use them in the next iterations of system design.
The user feedback on the sketches resulted in a few notable changesto the prototype design. The workflow of the latest version of theinteractive prototype is summarized in Figure 3. To emulate a morefamiliar conversational interaction for the OSS end users follow-ing insights from P4EU, the design of interaction between OSSdesigners and end users resembles platforms such as Slack andother asynchronous messaging platforms with features to react toa comment and start a thread (Guidelines D1, D5, EU1, EU2, EU4,EU5); see Figure 3 step 5. The addition of an ‘End User Collaborator’section underneath the standard ‘Contributors’ list for a repositoryshould encourage activity due to visible recognition of their effortsand display of OSS end users as valued members of the OSS commu-nity (Guidelines D5, EU4, EU5); see Figure 3 step 3. Clear buttons to create new software issues from resulting design artifacts of aconversation now incorporated for seamless integration into theexisting pipeline of an OSS project; this feature provides the abilityto directly link the relevant artifacts in an issue and indicate in theconversation who has been assigned to work on the new issue toensure transparency and traceability (Guidelines D2, D3, D4, D5,EU4, EU5). The system design is also guided by several usabilityheuristics (e.g. consistency, minimalist design, and flexibility [9])to achieve an effective and efficient interaction.
Overall, the feedback received during the user studies were over-whelmingly supportive towards the goals of understanding keybarriers experienced by OSS end users taking part in participa-tory design efforts and of creating a tool to facilitate successful,asynchronous participatory design. Our user study highlights theneed for a standardized method to support participatory designinteractions between OSS end users and designers. While the cur-rent prototype still needs to undergo thorough usability testing todemonstrate if it successfully achieves these goals, the user teststhat have been conducted thus far support the creation of a sin-gular tool that follow the guidelines and the three core featuresestablished in Sections 3.2 and 3.3.Despite overall positive feedback on the sketches and early pro-totype, the proposed design poses some challenges illuminated bythe study participants. Participant P2D indicated that every time anew platform for communication is introduced to their end usercommunity, switching costs are incurred as not all end users “ want ”to learn a new technology. (In this example, P2D explained thechallenges with moving from Discourse to Slack where now bothare active.) It is important to further explore this dimension withthe the next round of user studies. In particular, exploring how toeffectively integrate this system with existing tools and workflowswould be important. Furthermore, the current design of the systemfor a new end user to join or start a conversation solely through onference’17, July 2017, Washington, DC, USA Jazlyn Hellman, Jinghui Cheng, and Jin L.C. Guo
Figure 3: Overview of the Prototype System Design the OSS repository might prove to go against some of the usabilityconcerns, as was indicated by both OSS end user participants’ am-bivalence towards the GitHub-influenced design. A possible changeto the system might need to include a dedicated OSS end user-facingportal or interface to make it as easy as possible for diverse userbases to join participatory design efforts.In summary, this paper contributes a set of design guidelines andtwo personas through an user-centered exploration of a prototypetool for OSS designers and end users in collaboratively addressingusability concerns. Our preliminary user study confirmed a need fora standardized method to facilitate user-designer interactions thatprioritizes efficient communication of usability concerns to otherOSS team members. Needs also emerged for an interface design thatuses concepts and terminology that is inclusive of OSS end userswith diverse technical backgrounds. Our design guidelines andpersonas focused on capturing these and other previously identifiedneeds for promoting OSS usability. We are currently iterating theprototype with additional user studies to improve the design.
We thank our participants for their time and valuable feedback.This work was partially supported by the Discovery Grant of theNatural Sciences and Engineering Research Council of Canada.
REFERENCES [1] Morten S. Andreasen, Henrik V. Nielsen, Simon O. Schrøder, and Jan Stage.2006. Usability in open source software development: opinions and practice.
Information Technology and Control
35, 3A (2006), 303–312.[2] Jinghui Cheng and Jin L.C. Guo. 2018. How Do the Open Source CommunitiesAddress Usability and UX Issues? An Exploratory Study. In
Extended Abstractsof the 2018 CHI Conference on Human Factors in Computing Systems (MontrealQC, Canada) (CHI EA ’18) . Association for Computing Machinery, New York, NY,USA, 1–6. https://doi.org/10.1145/3170427.3188467[3] Joseph Feller and Brian Fitzgerald. 2000. A Framework Analysis of the OpenSource Software Development Paradigm. In
Proceedings of the Twenty First In-ternational Conference on Information Systems (Brisbane, Queensland, Australia) (ICIS ’00)
Commun. ACM
36, 6 (June 1993), 24–28. https://doi.org/10.1145/153571.255960[7] David Nichols and Michael Twidale. 2003. The Usability of Open Source Software.
First Monday
8, 1 (Jan. 2003), 10. https://doi.org/10.5210/fm.v8i1.1018[8] Jakob Nielsen. 1994.
Usability Engineering
Proceedings of the 33rd Annual ACM Conference on Human Factors inComputing Systems (Seoul, Republic of Korea) (CHI ’15) . Association for Comput-ing Machinery, New York, NY, USA, 3413–3422. https://doi.org/10.1145/2702123.2702441[12] Arif Raza, Luiz F. Capretz, and Faheem Ahmed. 2010. Improvement of OpenSource Software Usability: An Empirical Evaluation from Developers’ Perspective.
Advances in Software Engineering
Proceedings acilitating Asynchronous Participatory Design of Open Source Software: Bringing End Users into the Loop Conference’17, July 2017, Washington, DC, USA of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Geor-gia, USA) (CHI ’10) . Association for Computing Machinery, New York, NY, USA,999–1008. https://doi.org/10.1145/1753326.1753476[14] Dzhavat Ushev. 2020. My thoughts on GitHub Discussions. https://dzhavat.github.io/2020/04/04/my-thoughts-on-github-discussions.html[15] Wenting Wang, Deeksha Arya, Nicole Novielli, Jinghui Cheng, and Jin L.C. Guo.2020. ArguLens: Anatomy of Community Opinions On Usability Issues UsingArgumentation Models. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) . Association forComputing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376218[16] Wenting Wang, Jinghui Cheng, and Jin L. C. Guo. 2020. How Do Open SourceSoftware Contributors Perceive and Address Usability? Valued Factors, Practices,and Challenges.