ColVis: Collaborative Visualization Design Workshops for Diverse User Groups
CColVis: Collaborative Visualization Design Workshops for Diverse User Groups
Damla C¸ ay
KUARKoc¸ UniversityIstanbul, TurkeyEmail: [email protected]
Till Nagel
Faculty of Computer ScienceMannheim University of Applied SciencesMannheim, GermanyEmail: [email protected]
Asım Evren Yantac¸
KARMA, KUARKoc¸ UniversityIstanbul, TurkeyEmail: [email protected]
Abstract —Understanding different types of users needs caneven be more critical in todays data visualization field, asexploratory visualizations for novice users are becoming morewidespread with an increasing amount of data sources. Thecomplexity of data-driven projects requires input from includ-ing interdisciplinary expert and novice users. Our workshopframework helps taking design decisions collaboratively withexperts and novice users, on different levels such as outlin-ing users and goals, identifying tasks, structuring data, andcreating data visualization ideas. We conducted workshopsfor two different data visualization projects. For each project,we conducted a workshop with project stakeholders who aredomain experts, then a second workshop with novice users. Wecollected feedback from participants and used critical reflectionon the process. Later on, we created recommendations on howthis workshop structure can be used by others. Our maincontributions are, (1) the workshop framework for designingdata visualizations, (2) describing the outcomes and lessonslearned from multiple workshops.
Keywords -User centered design, Participatory design, Infor-mation visualization
I. I
NTRODUCTION
Today, the increasing amount of data sources like sensorsand activity trackers make data visualization an everydaylife topic for many types of users. Thus designers arecreating compelling visualizations that would be both usefuland interesting for diverse groups of users including bothnovices and domain experts. Domain experts are definedas researchers who perform complex data analyses usingvisualization tools [1], whereas novice users are new orinexperienced to certain tasks[2]. Designing visualizationsthat target different types of users, can become challengingwhen there are unclear project goals, ambiguous tasks,unstructured data, or many different stakeholders.Collaborative user-centered design methods can be ben-eficial to understand the above mentioned critical datavisualization aspects[3], [1]. Collaborative methods havebeen used in the data visualization field, however currentapproaches often emphasize data first, so user needs can beneglected[4], [5]. On the other hand, design thinking is acollaborative approach that focuses on users first to tacklecomplex problems[6]. Typically, the two main objectives of adesign thinking workshop are identifying the problem spaceand the solution space. In the first, participants understand the users and reframe the problem. During the second phase,participants ideate on solutions, build and test prototypes.Aligned with this approach, we suggest a workshopframework where the problem space deals with the goals,questions, and tasks of the users, while the solution spaceconsiders the data and visualization possibilities. In theproblem space phase, participants define the target users,identify project goals, collect and prioritize questions andtasks according to these goals. If there are ambiguous termsin questions that are not directly linked to the data at hand,they define proxies to resolve the ambiguity. In the solutionspace phase, participants link the questions and tasks todata. They identify the necessary data, discuss available data.Lastly, participants have a general discussion on what kind ofinformation is needed at what kind of granularity to supportspecific tasks.In this paper, we describe the workshop we conceived andimplemented. We organized four workshops, two workshopseach for two projects from different domains. The firstworkshop of each project had expert users, and the secondhad novice users. Then we collected feedback from theparticipants, and critically reflected upon our experiencesto identify challenges and opportunities. We report recom-mendations for applying collaborative design workshops fordesigning visualizations for diverse user groups. Collabora-tive design for data visualization is a challenging subjectconsidering the level of expertise required for building datavisualizations. Our findings indicate that these workshopscan enable collaboratively taking design decisions that re-flect the values of different stakeholders.II. R
ELATED L ITERATURE
A. Participatory and User-Centered Design
Design knowledge is the knowledge embedded in the de-sign of an artifact or service [7]. Creating design knowledgethrough end-user participation has been an important aspectof participatory design[8], [9]. As Schn [10] defines, partici-patory design is the process of mutual understanding, investi-gating, reflecting between participants where designers learnthe realities of users and users articulate their aims. Usefulmethods for participatory design include workshops, culturalprobes [11], ethnography, and cooperative prototyping [12]. a r X i v : . [ c s . C Y ] S e p mong these methods, the workshop technique has beenwidely used for human-computer interaction (HCI) research.Different workshop methods and tools are developed fordifferent aims and settings. Organizations like IDEO andStanford d.school have been successfully implementing user-centered design thinking workshops for business solutionsand social innovation [6], [13]. These workshops typicallyinclude hands-on divergent and convergent activities withusers to explore and prioritize possibilities [14].Collaborative and user-centered design methods arerapidly gaining popularity among data visualization re-searchers and practitioners as well[15], [16], [17], [18]. Heand Adar [19] express that design thinking could be usefulfor information design cases because of the wickedness ofthe data visualization design studies. Wicked problems arewithout definitive limits or conditions to the design problem[20], and visualization design studies can be defined aswicked problems due to the iterative nature of the designstudy research, as elaborated by Meyer and Dykes [21].With this perspective, we believe pursuing a design thinkingapproach with divergent and convergent activities would beuseful for data visualization cases. B. Frameworks for Data Visualization Design
Existing data visualization design frameworks formulatesteps to take when designing interactive data visualizations.Munzner [22] proposed the nested model. The model iden-tifies four nested decision-making levels which are; (1)domain problem characterization, (2) data/operation abstrac-tion design, (3) encoding /interaction technique design and(4) algorithm design. Another well-established frameworkfrom Sedlmair et al. [23] describes nine visualization designstages as learn, winnow, cast, discover, design, implement,deploy, reflect, and write.Consistently with these frameworks in the literature, theframework that we suggest starts with defining the problemspace first and then focusing on data, while different stagesof our workshop framework generate information that sup-ports both of the above-mentioned taxonomies by covering;domain problem definition, data/operation abstraction designand encoding/interaction technique design of the 4 stagesand understand, ideate, winnow, cast and discover stages ofthe nine-stage framework.
C. Co-design methods for Data Visualization
User-centered and collaborative methods are getting popu-larized. Koh et al. [24] propose a user-centered visualizationdesign approach where the process starts with familiariz-ing users with visualization methods through collaborativeactivities. Collaboration for data visualization used to takeplace between visualization researchers and other typesof researchers. When fields like economics, business, andhumanities started to use data visualizations increasingly,they were included in participation as domain experts [1]. Today, data visualizations are not only used as analysis toolsfor researchers and experts but also for data explorationby novice users[25], [16]. This requires the collaborationsphere to expand to novices[26]. Heer et al. [3] characterizethe visualization user base as expert, savvy or novice users.They identify a new research goal of supporting novice usersto specify their needs for a visualization. Our workshopframework enables these different types of users to specifyand prioritize their data visualization needs.As opposed to sequential visualization design frameworks,Wood et al. [27] perform simultaneous user studies withdifferent user types for a visualization case study. Theauthors state that this technique enables them to gain richinsights to guide the visualization design process. Theyuse various methods like public releases, talks, workshops,stakeholder meetings to gain insights through a three yearslong period. Hall et al. [28] learn from users throughimmersive exchanges between visualization researchers anddomain experts. Our approach is similar in the sense ofgaining rich insights from different types of users throughdifferent activities over time. However, we aim to initiatethis process in a shorter period.Kerzner et al. [29] define guidelines for workshops of datavisualization opportunities. Authors argue for participantsto adapt to a visualization mindset and recommend differentdesign activities for different purposes, where we focus on asystematic yet flexible structure which allows faster planningand execution. Differently, our approach starts with solelyfocusing on the problems and need, then focusing on visu-alization solutions. Additionally, our workshop frameworkenables exploring real data as well, and find solutions withdata in mind. Our work builds upon the existing literatureof collaborative practices in visualization and adapts it to asetting where different user types including non-experts cantake part in the data visualization process.III. C
OLLABORATIVE V ISUALIZATION W ORKSHOPS
Based on the challenges we experienced during earlierstudies on building visualization tools with interdisciplinaryteams, we opted for collaborative design workshops. Westarted working on a workshop structure to create a gen-eralizable framework to use as a guideline for planning andexecuting workshops, reporting outputs to gather problemsand ideas for a specific data-driven real-world context fordiverse stakeholders more richly and creatively.In this section, we explain the workshop framework wehave conceived and refined iteratively. Later on, we reporton the lessons learned through performing the workshops forvisualization projects from different domains. We criticallyreflected upon our experiences during the workshops andanalyzed feedback from the participants. After refining theframework using these insights, we observed how the work-shop worked independently by collecting feedback from amoderator who used the workshop for their project.n the following section, we will present four workshopsconducted for two different projects. For both projects, wefirst conducted a workshop with project stakeholders who aredomain experts, then a second with novice users. At least onevisualization expert was present at the workshop and theymoderated the workshops. The purpose of the visualizationexpert in all workshops was to translate the discussions intodesign decisions at the design phase. Finally, an additionalworkshop on a third visualization case was conducted by amoderator who is not a part of the team.
A. Projects and Participants
The first visualization project, The City Walls is a collab-oration between Archaeology and Design departments thataims to create a geographical visualization of data relatedto the city walls of Istanbul. The data the archaeologyteam collected includes historical information about thecity walls from primary historical sources, historical imagesand footage, architectural details of the walls, and a photoarchive created by the team that includes detailed images ofeach gate, tower, and wall.For the first workshop with experts, we invited all theproject stakeholders. The workshop participants were 6archaeologists, 1 photographer responsible for creating thephoto archive, 2 designers and a developer (3 female, 7male). The workshop lasted for 5. 5 hours. For the secondworkshop with novice users, we announced to the networkof a co-working space in Istanbul. One game developer, onearchitect, one interior architect, and one visualization expertattended the workshop (1 female, 3 male). The workshoplasted for 2. 5 hours.The second visualization project called ”Hope Archive”,is a geospatial video archive about non-governmental or-ganizations (NGO) activities. The project aims to makethe NGO activities visible, establish spatial or contextualconnections among different NGOs. The initial motivationfor the video database started with the activities of the DzceHope Homes. These videos documented the struggles of the1999 earthquake victims and the participatory process ofredesigning and rebuilding a living space for them. The datarelated to the videos are actors, themes of the NGOs, andtools for the activity.For the first workshop with experts, we invited all projectstakeholders to the workshop. 8 NGO employees, 1 doc-umentarist, 2 designers, 1 developer (6 female, 6 male).The workshop lasted for 6 hours. For the second workshopwith novice users, we announced it to the network of aco-working space in Istanbul. One project coordinator, oneservice and UX designer, one community coordinator andone visualization expert (4 females) attended the workshop,and it lasted for 3 hours.
B. Procedure
In the workshops, we employed the general structure of adesign thinking workshop where participants first define theproblem space and then define the solution space throughdivergent and convergent activities.The workshop has four phases as demonstrated on Figure 1:1. User and Goal (Problem space)2. Questions and Tasks (Problem space)3. Data (Solution space)4. Visualization (Solution space)The User and Goal phase starts with an open discussion.Participants discuss and list the potential users of futurevisualization. Then, they elect the core and extended usersof visualization using the dot voting method to prioritizeuser types (Figure 1, Define Core and Extended Users).Next, participants discuss and list the goals of the prioritizedpossible users, then vote for the most important and relevantgoals.In the Questions and Tasks phase, participants discuss andlist the questions to ask to the visualization, considering thegoals they define in the previous phase[30]. Participants thistime, vote the questions that are the most relevant to thegoals or interesting. Participants create one or more tasksout of each selected question. Then they identify ambiguouscomponents (not directly addressable by the dataset). Partic-ipants define proxies until all tasks are actionable [31]. Thisworkshop phase aims to form clear tasks from ambiguousquestions.After this, participants continue to data phase where theaim is to identify links with questions and data. If there isalready collected data, participants explore the data set toidentify the links between questions and data. If there is noor partially collected data, participants discuss which data isneeded to answer the questions. In this phase, participantsuse methods like card sorting, affinity diagramming, mindmapping, and dot voting to organize and prioritize data.In the visualization phase, we present different visualiza-tion functions (Distribution/ Time / Compare / Geospatial/ Part-to-Whole / Relationship) and interaction styles (se-lecting / filtering / brushing / hovering / highlighting). Thenwe demonstrate examples of the explained concepts. Afterthis, participants discuss which dimensions of data shouldbe visualized. Then they ideate on alternative visualizationideas. This activity can be conducted as a group or indi-vidual activity depending on the number of participants andparticipant preference. If it is conducted as an individualor small group activity, at the end of the workshop, theparticipants present and give feedback to the ideas. If itis conducted as a group activity with all participants ofthe workshop participants, it can be implemented as groupideation followed with a reflective discussion. There are twooutputs of this phase that can inform the design of the igure 1. The workshop has four phases :1. User and Goal, 2. Questions and Tasks, 3. Data, and 4. Visualization. The lines represent possible relationsbetween consecutive phases. Detailed guide of the process and other materials can be achieved at https://github.com/colvis2019/ColVis-Workshop visualization. Firstly, the visualization ideas that receivedpositive feedback can inform abstraction and interactiondesign. Secondly, critique and discussions reveal the finaldesign requirements.
C. Methodology
After the workshops, we collected the notes from theworkshops, answers to the post-workshop open-ended ques-tions, our notes from the oral feedback participants gaveafter the workshops and the critical reflections [32], [33] ofthe authors who participated as visualization experts at theworkshops.IV. R
EFLECTIONS ABOUT W ORKSHOPS
In this section, we briefly present an overview of eachworkshop’s process. Then we present our critical reflectionsabout the process and highlight the important points of theparticipant feedback.
A. W1. The City Walls Map with Experts
After a brief overview of the project’s main goals andstatus, we started the workshop with users and goals. Theparticipants prioritized novice citizens as the primary usertype and exploring the city walls as the primary goal.In the next phase Questions and Tasks, the participantsgenerated and prioritized questions. For the data phase,we printed out samples from the visual content and excelsheets before the workshop. During the data phase, first, theparticipants identified the links between the questions andexisting data. Some questions required data other than thealready collected data. These were also identified during thediscussion. After this, participants sorted and prioritized thedata samples using card sorting and dot voting methods.At the visualization phase, first, participants discussed the different visualization possibilities over examples. Then theycreated data sketches and the workshop finished with a groupdiscussion about the results.Before the workshop, the project stakeholders had a vaguedefinition of the project goal. They started collecting visualdata with an archival motivation and wanted to create ageographical visualization from this archive. They statedthat such a tool can be useful for remote researchers.However, during the workshop, the necessity of definingand prioritizing the user and goal made the stakeholdersrealize that they were prioritizing citizens as users overresearchers. During the design process before the work-shop, the project stakeholders and the visualization teamhad several meetings. These meetings included differentcombinations of stakeholders at once, due to availability.Discussions during the workshop revealed that differentstakeholders had different visions of and expectations fromthe visualization tool. The workshop structure enabled themto create a unified goal.Post-workshop feedback from the participants reflects thatthey were overall pleased with the workshop at the end.One participant stated, At the beginning, I wasn’t quite surewhere it will all lead but I was impressed with the resultswe ended up with. One participant expressed the need formore breaks. Some participants felt like one stakeholderdominated the discussions for some phases of the workshop.
B. W2. The City Walls Map with Novice Users
In the second workshop of the same project, the partic-ipants identified students as the prioritized user type and,exploration and research as the primary goal. In the Ques-tions and Tasks phase, the questions participants generatedwere related to the main entry paths to the city, and themodern socio-cultural surrounding of the walls. Even thoughome questions that were generated in the workshop weresimilar to the ones from the expert workshop, the prioriti-zation differed. Novice users focused more on gates thanother architectural elements like walls or towers. They alsoprioritized contemporary information like the communitieslived and still lives around the city walls. At the dataphase, the group was presented the same material from theexpert workshop, including architectural data, historical data,and visual material. However, they had trouble linking theexisting data to some of the questions and proposed new datatypes instead. At the visualization phase, the participantsproposed visualization ideas for different levels of detail, asa group.Participants felt that the workshop had a casual andrelaxed environment. Even though we explained the over-arching aim of the project at the beginning and presentedthe collected data, some participants expressed that theyfelt uninformed about the project. One participant foundthe discussions too free-form. One participant found thediscussions to be too abstract, another enjoyed the dialogand discussion itself. Several participants expressed that theworkshop’s interdisciplinary nature helped to create fruitfuldiscussions. One of them found, The difference of partic-ipants in terms of background and discipline enables eachother to see new aspects and create a cohesive contribution.Several participants felt like the collaboration took placein the form of building upon each other’s ideas. One par-ticipant said, The act of sharing all our individual ideas ontopics was itself the collaboration. Another participant foundthe use of post-its enabled the discussion to be more visibleand this helped to trigger their participation.
C. W3. Hope Archive with Experts
Following the same structure, participants prioritized theResearcher/Student using the visualization for research. Thesecond user type was NGOs, using the visualization tolearn best practices and networking. The third user typewas journalists, using the visualization to find stories. Atthis point, one participant opened the discussion of contentcreation around the questions like, if the platform will beopen to the public, will it be moderated or unmoderated, orwill it be a closed platform where people can apply withtheir content.In the Questions and Tasks phase, the prioritized questionswere, what type of activities are NGOs engaged in? Wheredo these activities take place? What are the methods theyuse? What are the NGO activities with a higher impact?After the questions are set, the data dimensions related to thequestions were, video, story, location, actors, theme, method,the amount of content, latest content. The visualizationdecisions included having a simpler base map, improvingthe visual connection between the map view and list viewon the existing tool, functional suggestions like connectingYouTube channels to the website and automating the video upload process. Other suggestions were related to fixing theusability issues of the existing tool.After the workshop, we identified two important pointswhile critically reflecting on the process. Firstly, the finaldiscussion did not involve visualization solutions accordingto the identified data types. They were functional enhance-ments for the existing tool. Secondly, during the workshop,one participant initiated discussions repeatedly on who willproduce the content of the platform and how. This repetitioncaused a loss of focus and prevented the discussion frommoving forward at times. The qualitative feedback we col-lected after the workshop included recurring themes. Someparticipants expressed that the workshop helped them clarifygoals and discuss the tool thoroughly. Additional positivecomments stated that the workshop created awareness of theproblems and awareness of the necessity to use more user-centered methods. In terms of teamwork, two participantsstated that the workshop was more like a place to share in-dividual ideas rather than teamwork. The negative commentswere related to the repetition of discussions. Some partici-pants felt like the workshop structure was unsystematic, thediscussions were too broad and there was no clear result atthe end of the workshop.
D. W4. Hope Archive with Novice Users
The workshop started with an introduction where themoderator explained the goals and motivations of theproject. The prioritized user types were students/academics,NGOs/collectives, and local governments. The identifiedgoals were: researching for students/researchers, archivingtheir projects and networking for NGOs/collectives, findingproject stakeholders for local governments. The questionsgenerated were, who are the people doing similar work toour NGO? What type of methodologies they use? What hasbeen done on a specific topic? When was it done? Doesanyone have data that I can use? Related to the questions,the data types identified by participants were the location ofthe NGO, topic, methods, photos, videos, publications anddate of each activity and references. At the visualizationphase, for the overview, participants proposed a networkvisualization where users can see the links between NGOsand topics. At this level, they wanted to see the NGO name,topic, stakeholders, starting and last active dates.Our critical reflections on the workshop process includetwo important points. Firstly, the discussions were moreclear and fruitful than the expert workshop. Outcomes of theworkshop were more suited to guide the visualization design.Secondly, to make the goal and the content clear, we showedthe existing prototype. However, this limited the participants,to the point where they can only identify the usability issues.After they are reminded to think freely without limitingthemselves with the existing tool, they started to ideate.From the post-workshop questions, one common positivecomment was about the flow of the discussion and the mod-rators guidance. One participant stated that the moderatorsuccessfully guided the discussion when it was out of focusand another commented on the moderator synthesized andframed the outcomes effectively. Another positive aspect wasabout visualization awareness. One participant stated thatthe workshop enabled them to think about their data-relatedprojects more clearly. Others were glad to be aware of a localproject that might be of interest. Overall, they felt like it wassuccessful and enjoyable teamwork. On the other hand, someparticipants commented on the negative impact of seeingthe existing tool. One participant stated that it limited thediscussion. Another negative point that one participant feltnot informed enough at the beginning of the workshop.
E. Case Study: Sonic Memories
After four workshops moderated by visualization expertswho also author this paper, we wanted to have an additionalworkshop with a non-team member as a moderator, totest and improve the workshop framework. We prepared adetailed moderators guide that included the phases and stepsto conduct the workshop independently. One researcher whowas starting with a new data visualization project used theframework, whose project deals with visualizing personalmemories related to city sounds. After they conducted theworkshop, we interviewed the moderator and two workshopparticipants.Overall, the researcher found the workshop to be usefulin terms of identifying and justifying data needs. The mod-erator stated, ”The data phase was useful. I collected sampledata from the participants, everyone wrote memories aboutcity sounds. Then we extracted data dimensions from those.The dimensions were similar to what I had in mind beforethe workshop. So my ideas were supported in this phase.There were additional ideas about the functionality, which Ihaven’t thought before.”Generating questions that are related to the prioritizeduser goals was a challenge. The moderator said, ”Somequestions generated in the questions and tasks phase werenot questions about interacting with data. I had to interveneand re-direct a lot here. Additionally, one participant said,Questions and Tasks phase was good but it wasn’t clearwhich questions relate to which goals of which user type.We tend to forget about the user in this phase. We generateda lot of questions and some of them weren’t related to adefined goal. In addition to generating questions, prioritizingthem using dot voting was also unclear and challenging. Oneparticipant stated, When selecting questions with dot voting,I observed that people tended to select the ones that areeasy to understand rather than interesting ones. Similarly,another participant said, ”The selection process of questionswas hard. I wasn’t clear on the selection criteria. We couldhave selected the wild questions but we didn’t.The last important point both mentioned by the moderatorand a participant was about the sketching part of the visual- ization phase. The moderator stated that the participants whoweren’t designers struggled when sketching. The moderatorstated, ”Maybe they can communicate their ideas differently.
F. Design and Development of the Hope Archive and theCity Walls projects
Even though the aim of this paper is not to discuss thedesigned tools extensively, we would like to briefly givean overview of the tools. Both projects were designed anddeveloped using the decisions from the workshops(Figure 2).On the City Walls project’s main page, information relatedto gates, towers and walls are visualized. Glyph for gatesare bigger because they were stated as more important atboth workshops (Figure 3, top left). When clicked, tagsappear on the left side of the screen as suggested in theexpert workshop and more detailed information about theunit is presented (Figure 3, top right). At the Hope Archive’smain page, the information is presented geographically asthe experts suggested (Figure 3, bottom left). Same data canbe viewed in a node-link diagram as the novices suggested(Figure 3, bottom right).V. D
ISCUSSION
A. Maintaining the focus for informing visualization design
One of the biggest challenges during the workshops waskeeping the focus on designing visualization and guidingthe discussions in a way that will create useful outcomesfor designing visualizations. Here we share the patterns weidentified from our critical reflections and post-workshopfeedback and recommendations to overcome problems thatcan occur.In every workshop session, discussions shifted to subjectsthat were not directly about the visualization itself. Weobserved these shifts were longer and deeper in expert work-shops. It was harder to focus back on the visualization. Eventhough there may be discussion around the topic of interest,the main focus should be on the visualization. Another prob-lem we encountered several times, was discussion shifting toa topic that is not directly related to the workshop phase. Forinstance, repeated discussions on the data collection methodon every phase of the Hope Archive expert workshop. Ourapproach depends on starting with the user in mind, thenmoving towards questions and finding links between thosequestions and data. Each phase creates the outcome of thenext one. This structure makes it important to focusing onlyon one phase at a time. To overcome these problems, werecommend moderators to selectively take notes by onlyhaving related keywords noted, and bring attention to thecurrent workshop phase as necessary.For both expert workshops, there were long and insis-tent discussions about project-related, but not visualizationrelated topics. These long discussions in each workshopwere dominated by one participant, who were both projectstakeholders. Aside from elongating the workshop period, igure 2. Hope Archive and The City Walls projects are realized and online. The links are provided in the supplementary files at https://github.com/colvis2019/ColVis-Workshop this also affected other participants negatively. Dominantparticipants were mentioned negatively by other participantsin the post-workshop survey. One participant even suggestedthat the moderator should decide who will talk when. Toovercome this challenge, workshops can be divided to haveup to 5 participants, as small workshops enable everyoneto take part more comfortably. Another solution during theworkshop could be, having a quick round around the table,asking everyone their individual idea, and then making ashort, concluding group discussion.During the novice workshops, we showed a work-in-progress version of the visualization. Our aim was notto influence their visualization choices but to show theavailable data types. These unfinished visualizations causeddistraction and unnecessary discussions about the usabilityproblems of the interactive visualizations. During the noviceworkshop of the Hope Archive project, showing work-in-progress caused the divergent phases to be more limited.After the workshop, participants stated that the discussionswere more productive after the moderator reminded themto think as if the work-in-progress did not exist. Since theworkshop aims to reveal design requirements, we recom-mend not showing any work-in-progress material.We arranged the workshop set up in a way that outcomesof the previous phase were visible either on a wall or table.However, during the case study, some participants and themoderator mentioned they had a hard time with the questionsand tasks phase and some questions were not directly related to the prioritized user type and goal. We recommend visuallyand orally highlighting the prioritized outcomes of eachphase, and intervene every time an unrelated input occurs,remind the participants the overall goal and the process ofthe workshop.We wanted to include the dot voting method when pri-oritization is needed as it is commonly used in designthinking workshops as a quick way to understand the grouptendencies. When implementing this method for our datavisualization workshops, sometimes if fell short for our needfor prioritizing with important criteria in mind. These criteriawere about how relevant, important, interesting or feasiblesomething is. For the Sonic Memories case study, someparticipants stated that voting created confusion during thequestions and tasks phase as they were not aware of whythey voted, and each participant was voting for a differentreason. Instead of using the same element (dot) as feedback,we recommend separating vote types visually, by eithercolor-coding or writing the feedback types on the votes.
B. Nature of participation differs for designing data visual-ization
Even though the participants of the two workshops weredifferent people with entirely different levels of domainknowledge and involvement in the project, the groups gener-ated similar questions during the Questions and Tasks phasefor the City Walls project. On the other hand, experts andnovices prioritized different questions. This reveals that ourorkshop framework, when applied to different groups ofusers, can be a way to understand the most important tasksfor a visualization. Seeing different prioritizations can revealdifferent design requirements for different user types.During the visualization phase, some participants re-frained from creating sketches. This can be common amongparticipants who are not from a design background. Designthinking workshops have special activities to encourage peo-ple to sketch. However, this might be hard to apply becauseof the time limit, and also unnecessary since the ultimategoal is to make design decisions that are based on needs anddata. One solution might be creating collages in this phase[34], [35], or having pre-made visual examples of basicvisualization methods as sheets or cards for participants tocommunicate their design decisions easier.The City Walls project had more complex data typescompared to the Hope project. Regarding the data com-plexity, the data phase of the City Walls expert workshoptook the longest. Besides, experts had an easier time sortingdata since they have expertise on the subject. On the otherhand, both expert and novice users can identify interestingdata types for projects aimed at diverse user groups. Ourreflections on the process and feedbacks indicate that thedepth of data and participant type affect the process, and theworkshop should be applied considering these differences.
C. Challenges of organizing and moderating a data visual-ization workshop
Every data-visualization project has its unique challengesrelated to the data itself. Data might be unavailable, missing,unclear, or complex. We envisioned the workshop to workeffectively with different amounts of existing data. If theproject does not have any data, the data phase can be used toidentify the needed data types and how they can be achieved.If there are data, but the project stakeholders are open tosuggestions, then a similar discussion on data types can befollowed by browsing existing data, sorting and prioritizingand finally identifying links between questions and data. Atthe Sonic Memories workshops data phase, the moderatorwho was also the project owner decided to collect sampledata by asking participants to write a memory about a place.Then they were able to identify the data dimensions that thememories included and continued the rest of the data phaseusing these samples. After the workshop, the moderatorstated that data phase was very useful for the project. Whenapplying the workshop, we recommend adjusting the dataphase according to the needs and circumstances of theproject.The space that the workshop happens in is an importantelement that affects the nature of participation. Since theavailable options to host a workshop might be limited, weenvisioned the workshop to be applicable in a variety ofspaces. However, there are two essential elements. The firstone is the visibility of generated and prioritized keywords and how they relate to other phases. The second one ishaving enough space to display data and perform hands-onactivities. A table or wall can be used for these purposes.In small workshops with up to 4 participants, a small tablecan be suitable to arrange post-its and data since everyonewill be able to see and reach the material. However, abigger workshop might require an empty wall, and enoughspace in front of the wall to place and organize the post-its. Additionally, the table can be used to organize data andcreate visualization ideas. Space should be considered alongwith the number of participants. Overcrowded spaces withmore participants than the table can afford, can hinder hands-on participation. VI. C
ONCLUSION
We presented ColVis Workshop Toolkit, that enablescreating human-centered data visualization solutions collab-oratively with diverse user groups like novice and expertusers. We designed this workshop to include users early intothe data visualization process starting from defining usersand goals, identifying and prioritizing tasks, identifying ex-isting and needed data, and creating data visualization ideasaccording to the defined requirements. We applied the work-shop framework to two projects, two workshops for eachproject, one with expert users and the other one with noviceusers as participants. Additionally, an external researcherimplemented the workshop for their project. Based on ourcritical reflections and qualitative feedback of participantsand an external researcher, we find that ColVis workshopstructure provides data visualization design directions ondifferent levels, in a user-centered way. We provide therecommendations and the material and hope they can be usedand developed further to enable deeper user participation inthe data visualization field.R
EFERENCES [1] R. M. Kirby and M. Meyer, “Visualization collaborations:What works and why,”
IEEE computer graphics and appli-cations , vol. 33, no. 6, pp. 82–88, 2013.[2] J. Fisher, “Defining the novice user,”
Behaviour & Informa-tion Technology , vol. 10, no. 5, pp. 437–441, 1991.[3] J. Heer, F. Van Ham, S. Carpendale, C. Weaver, and P. Isen-berg, “Creation and collaboration: Engaging new audiencesfor information visualization,” in
Information visualization .Springer, 2008, pp. 92–133.[4] S. Huron, P. Gourlet, U. Hinrichs, T. Hogan, and Y. Jansen,“Let’s get physical: Promoting data physicalization in work-shop formats,” in
Proceedings of the 2017 Conference onDesigning Interactive Systems . ACM, 2017, pp. 1409–1422.[5] S. Huron, S. Carpendale, J. Boy, and J.-D. Fekete, “Usingviskit: A manual for running a constructive visualizationworkshop,” 2016.6] T. Brown and J. Wyatt, “Design thinking for social innova-tion,”
Development Outreach , vol. 12, no. 1, pp. 29–43, 2010.[7] C. Frauenberger, J. Good, G. Fitzpatrick, and O. S. Iversen,“In pursuit of rigour and accountability in participatorydesign,”
International journal of human-computer studies ,vol. 74, pp. 93–106, 2015.[8] E. Brandt, “Designing exploratory design games: a frameworkfor participation in participatory design?” in
Proceedingsof the ninth conference on Participatory design: Expandingboundaries in design-Volume 1 . ACM, 2006, pp. 57–66.[9] P. Ehn, “Participation in design things,” in
Proceedings of thetenth anniversary conference on participatory design 2008 .Indiana University, 2008, pp. 92–101.[10] D. A. Sch¨on,
The reflective practitioner: How professionalsthink in action . Routledge, 2017.[11] B. Gaver, T. Dunne, and E. Pacenti, “Design: cultural probes,” interactions , vol. 6, no. 1, pp. 21–29, 1999.[12] J. Simonsen and T. Robertson,
Routledge international hand-book of participatory design . Routledge, 2012.[13] H. Plattner, C. Meinel, and U. Weinberg,
Design-thinking .Springer, 2009.[14] A. Osborn,
Applied Imagination-Principles and Proceduresof Creative Writing . Read Books Ltd, 2012.[15] A. K. Munk, A. Meunier, and T. Venturini, “Data sprints: Acollaborative format in digital controversy mapping,” digital-STS: A Field Guide for Science & Technology Studies , p. 472,2019.[16] A. Arcia, N. Suero-Tejeda, M. E. Bales, J. A. Merrill, S. Yoon,J. Woollen, and S. Bakken, “Sometimes more is more: iter-ative participatory design of infographics for engagement ofcommunity members with varying levels of health literacy,”
Journal of the American Medical Informatics Association ,vol. 23, no. 1, pp. 174–183, 2015.[17] ¨O. Genc¸, D. C¸ ay, and A. E. Yantac¸, “Participatory explo-rations on a location based urban information system,” in
International Conference of Design, User Experience, andUsability . Springer, 2015, pp. 357–367.[18] D. C¸ ay and A. E. Yantac¸, “Learning from the users for spatio-temporal data visualization explorations on social events,” in
International Conference of Design, User Experience, andUsability . Springer, 2016, pp. 124–135.[19] S. He and E. Adar, “V iz i t c ards: A card-based toolkit forinfovis design education,”
IEEE Transactions on Visualizationand Computer Graphics , vol. 23, no. 1, pp. 561–570, 2016.[20] R. Buchanan, “Wicked problems in design thinking,”
Designissues , vol. 8, no. 2, pp. 5–21, 1992.[21] M. Meyer and J. Dykes, “Criteria for rigor in visualization de-sign study,”
IEEE transactions on visualization and computergraphics , 2019. [22] T. Munzner, “A nested model for visualization design andvalidation,”
IEEE transactions on visualization and computergraphics , vol. 15, no. 6, pp. 921–928, 2009.[23] M. Sedlmair, M. Meyer, and T. Munzner, “Design studymethodology: Reflections from the trenches and the stacks,”
IEEE transactions on visualization and computer graphics ,vol. 18, no. 12, pp. 2431–2440, 2012.[24] L. C. Koh, A. Slingsby, J. Dykes, and T. S. Kam, “Devel-oping and applying a user-centered model for the design andimplementation of information visualization tools,” in .IEEE, 2011, pp. 90–95.[25] T. Nagel, M. Maitan, E. Duval, A. V. Moere, J. Klerkx,K. Kloeckl, and C. Ratti, “Touching transport-a case studyon visualizing metropolitan public transit on interactive table-tops,” in
Proceedings of the 2014 International WorkingConference on Advanced Visual Interfaces . ACM, 2014,pp. 281–288.[26] S. McKenna, A. Lex, and M. Meyer, “Worksheets for guid-ing novices through the visualization design process,” arXivpreprint arXiv:1709.05723 , 2017.[27] J. Wood, R. Beecham, and J. Dykes, “Moving beyond se-quential design: Reflections on a rich multi-channel approachto data visualization,”
IEEE transactions on visualization andcomputer graphics , vol. 20, no. 12, pp. 2171–2180, 2014.[28] K. W. Hall, A. J. Bradley, U. Hinrichs, S. Huron, J. Wood,C. Collins, and S. Carpendale, “Design by immersion: Atransdisciplinary approach to problem-driven visualizations,”
IEEE transactions on visualization and computer graphics ,2019.[29] E. Kerzner, S. Goodwin, J. Dykes, S. Jones, and M. Meyer,“A framework for creative visualization-opportunities work-shops,”
IEEE transactions on visualization and computergraphics , vol. 25, no. 1, pp. 748–758, 2018.[30] H. Lam, M. Tory, and T. Munzner, “Bridging from goals totasks with design study analysis reports,”
IEEE transactionson visualization and computer graphics , vol. 24, no. 1, pp.435–445, 2017.[31] D. Fisher and M. Meyer,
Making Data Visual: A PracticalGuide to Using Visualization for Insight . ” O’Reilly Media,Inc.”, 2017.[32] S. Brookfield, “Critically reflective practice,”
Journal of Con-tinuing Education in the Health Professions , vol. 18, no. 4,pp. 197–205, 1998.[33] M. Meyer, J. Dykes, and M. Tory, “Reflection on reflection inapplied visualization research,”
IEEE computer graphics andapplications , vol. 38, no. 6, pp. 9–16, 2018.[34] K.-l. Chen, M. D¨ork, and M. Dade-Robertson, “Exploringthe promises and potentials of visual archive interfaces,” iConference 2014 ProceedingsiConference 2014 Proceedings