Insights From Experiments With Rigor in an EvoBio Design Study
Jen Rogers, Austin H. Patton, Luke Harmon, Alexander Lex, Miriah Meyer
TTo appear in IEEE Transactions on Visualization and Computer Graphics.
Insights From Experiments With Rigor in an EvoBio Design Study
Jen Rogers, Austin H. Patton, Luke Harmon, Alexander Lex, Miriah Meyer
Abstract — Design study is an established approach of conducting problem-driven visualization research. The academic visualizationcommunity has produced a large body of work for reporting on design studies, informed by a handful of theoretical frameworks, andapplied to a broad range of application areas. The result is an abundance of reported insights into visualization design, with anemphasis on novel visualization techniques and systems as the primary contribution of these studies. In recent work we proposeda new, interpretivist perspective on design study and six companion criteria for rigor that highlight the opportunities for researchersto contribute knowledge that extends beyond visualization idioms and software. In this work we conducted a year-long collaborationwith evolutionary biologists to develop an interactive tool for visual exploration of multivariate datasets and phylogenetic trees. Duringthis design study we experimented with methods to support three of the rigor criteria:
ABUNDANT , REFLEXIVE , and
TRANSPARENT . As aresult we contribute two novel visualization techniques for the analysis of multivariate phylogenetic datasets, three methodologicalrecommendations for conducting design studies drawn from reflections over our process of experimentation, and two writing devices forreporting interpretivist design study. We offer this work as an example for implementing the rigor criteria to produce a diverse range ofknowledge contributions.
Index Terms —Methodologies, Application Motivated Visualization, Guidelines, Life Sciences Visualization, Health, Medicine, Biology,Bioinformatics, Genomics
NTRODUCTION
Design study is an established approach to problem-driven visualizationinquiry that emphasizes designing visual analysis tools in close collab-oration with domain experts [66]. Within a design study, visualizationresearchers build an understanding of a problem domain and translatethat understanding into a visualization design, iteratively refining boththeir understanding of the problem and their visual analysis solutionthrough close work with domain collaborators. Researchers conductingdesign studies draw from a host of theoretical constructs to guide theinquiry process, from process models [23, 39, 42, 44, 66] to design deci-sion models [46, 50], guiding scenarios [65], educational models [72],and collaboration roles [69, 78]. As a result, an increasing numberof reports describe effective design studies within a broad range ofapplication areas [9, 26, 32, 40, 41, 52, 56, 82].Historically, design study papers have emphasized novel visual anal-ysis systems and techniques as primary knowledge contributions [44].Many of these papers also cite domain characterizations and abstrac-tions [50] as contributions under the reasoning that they are importantfor judging the validity of technical design artifacts and for buildinga body of visual analysis requirements that others can design against.The original definition of design study also includes lessons-learned asa potential knowledge contribution stemming from reflection, but scantguidance is available on how to generate knowledge of this sort [43].In Meyer & Dykes [44] we proposed a new, interpretivist view ofvisualization design study to produce a more diverse range of knowl-edge contributions. As a critique of the software-centric view of designstudy, this new perspective emphasizes the potential for using designstudy to acquire a more diverse range of knowledge, including knowl-edge about the visualization design process as well as about people’srelationship with data and technology more broadly. This work recom-mends six rigor criteria for guiding the design study process towardacquiring new knowledge:
INFORMED , REFLEXIVE , ABUNDANT , PLAUSIBLE , RESONANT , and
TRANSPARENT . These criteria provide an opportunity for • Jen Rogers, Alexander Lex, Miriah Meyer are with the University of Utah,E-mail: [email protected], [email protected], [email protected].• Austin H. Patton is with Washington State University, E-mail:[email protected].• Luke Harmon is with the University of Idaho, E-mail: [email protected] is the authors preprint version of this paper. Please cite the followingreference: Jen Rogers, Austin H. Patton, Luke Harmon, Alexander Lex, MiriahMeyer. Insights From Experiments With Rigor in an EvoBio Design Study. IEEETransactions on Visualization and Computer Graphics, to appear, 2020. researchers to rethink how to conduct effective design studies, learningnew things along the way.In the work we present here we experimented with methods to sup-port three of the rigor criteria:
ABUNDANT , REFLEXIVE , and
TRANSPARENT .Our experimentations took place within the context of a one-year de-sign study with evolutionary biologists. We employed techniques suchas an immersive, three-month field study; structured and systematicreflection; and careful curation of documents and other design artifacts.Through a period of collaborative, critical reflection, we identifiedseveral methodological insights that emerged from our experiments.The resulting contributions from this inquiry are diverse, includingboth technical and methodological insights. More specifically, thecontributions include:• Two new visualization techniques for supporting the analysisof multivariate trees: (1) a trait view that visualizes node-valuedistributions under uncertainty for associated characteristics alongmultivariate subtrees; and (2) a pattern view that aids in thediscovery and visualization of patterns in value trajectories forattributes across paths in a tree.• Three methodological recommendations for conducting interpre-tivist design study: (1) establish systematic reflective practicesthat include reflexive notes, reflective transcriptions, and arti-fact curation; (2) build and maintain a trace of diverse researchartifacts; and (3) argue for rigor from evidence, not just methods.• Two experimental writing devices for reporting on interpretivistdesign study: (1) inclusion of direct links to research artifactsto transparently provide an abundance of evidence; and (2) em-bedding of a design study paper within a methodological one tohighlight the diversity of our research contributions.This work serves as an example of how researchers can consider the
ABUNDANT , REFLEXIVE , and
TRANSPARENT criteria in practice, as well asthe diverse types of knowledge contributions possible through theirconsideration.We first provide the theoretical backdrop for our methodologicalwork in Section 2, followed by a description of our research meth-ods in Section 3. Section 4 is a design study paper-within-a-paper,emphasizing the technical aspect of this work; our methodologicalrecommendations follow in Section 5. Throughout the paper we in-clude direct links to our abundant collection of research artifacts — forexample [T45] — to transparently provide evidence for our claims.
HEORETICAL B ACKDROP
The methodological work we present in this paper draws from the inter-pretivist perspective of design study proposed by Meyer & Dykes [44].1 a r X i v : . [ c s . H C ] A ug his perspective argues for a myriad of opportunities for researchersto make valuable knowledge contributions beyond visualization tech-niques and software. Doing so, however, requires a rethinking of designstudy research practices and the ways we make quality judgments aboutthe inquiry. Six criteria for rigor guide an interpretivist design study ap-proach — INFORMED , REFLEXIVE , ABUNDANT , PLAUSIBLE , RESONANT , and
TRANSPARENT — which are derived from theoretical positions in socialscience [35, 70, 76], information systems [67], and research throughdesign [20, 84]. Achieving all six criteria within a single design studyis unlikely to occur due to pragmatic constraints such as time andresources [44]. In the work presented in this paper, we focus on
ABUN-DANT , REFLEXIVE , and
TRANSPARENT , exploring various ways to achievethese criteria, as well as the kinds of knowledge elucidated by doing so.
A design study with abundance reflects the richness and complexity ofthe situation under study [44]. An abundant design study thus includesa rich and diverse body of evidence, as well as an abundance of otherconsiderations such as participant voices, designs, and time in the field.In our experiments we considered all of these aspects of abundance.The inclusion of a variety of voices and contexts reflects a valuingof pluralism found in critical feminist theory that “insists that the mostcomplete knowledge comes from synthesizing multiple perspectives”[30]. In human computer interaction (HCI), pluralism is argued as amechanism for resisting designs that embed “any single, totalizing, oruniversal point of view” [3]. Arguments for pluralism can be groundedin the idea of situated knowledges [24], which argues an epistemic viewof a singular reality that can only be known only partially, embeddedwithin a specific context. It is by combining these partial perspectives— through “actively and deliberately inviting other perspectives into thedata analysis” [30] — that a researcher achieves a fuller, richer view ofthe situation under study.An emphasis on exploring a design space through many, rapid de-signs similarly helps a designer avoid blind spots and fixation on asingular solution [12, 15]. Design problems are wicked by nature, withan extensive space of possible solutions [11]. By broadly consider-ing a design space, designers are more likely to find good solutions,rather than average or poor ones [66], as well as to develop a betterunderstanding of the problem under study [15]. Dow et al. recommendexploring and refining design ideas in parallel, rather then through a se-quential process, to obtain better and more diverse design artifacts [18].In the same vein, Buxton advocates for rapid sketching with broadideation for developing effective design concepts through iterations of“controlled convergence” [12].Finally, abundance through prolonged engagement with the peopleand context under study is a mainstay of qualitative research [35,68,76].Researchers who establish an early familiarity with a domain build trustwith their participants as well as the ability to understand domain-specific nuances of what they observe: “objects and behaviors take notonly their meaning but their very existence from their contexts” [35].In a visualization study, design by immersion is an approach for engage-ment in which both the visualization researchers and domain experts“participate in the work of another domain such that visualization de-sign, solutions, and knowledge emerge from these transdisciplinaryexperiences and interactions” [23]. This methodology allows visualiza-tion researchers to enrich their understanding of a domain, explore abroader visualization solution space, and build trust and agency withcollaborators. Field studies — in which a researcher spends sustainedtime with participants in their natural environment — is a technique thatcan support visualization researchers in achieving immersion throughprolonged engagement [40].
Being reflexive within a visualization design study is to strive for“explicit and thoughtful self-awareness of a researcher’s own role ina study” [44]. As a cornerstone of interpretivist, qualitative research,reflexivity is an acknowledgement of a researcher’s influence on a study,and vice versa [4]. Researcher bias and perspective are an inherent partof qualitative research, and eliminating them from the research process is arguably impossible [38]. Reflexivity is instead an opportunity togather valuable data [61] that can help researchers understand theirbiases and perspectives as a vector for change and learning [19].Reflexivity is an important consideration in the third wave of HCIresearch [6]. Largely discussed in the critical HCI literature, reflexivityis considered a mechanism for researchers “to be accountable for theways in which HCI construes design(ing) and acknowledge our respon-sibility . . . to challenge the dominant view on design” [2]. Despiteits importance, the HCI community has been slow to broadly adoptreflexive practices in research due to the scrutiny on subjectivity duringthe review process. The visualization research community shares asimilar emphasis and valuing of objectivity [44], and a lack of meth-ods for supporting and exploiting reflexivity. This gap motivated ourexperimentations with reflexivity.Reflexivity is a type of (self) reflection [37]. As a method, reflectiontraces to Sch¨on’s ideas of reflective practice through reflection-in-actionand reflection-on-action [64]. Reflection-in-action is characterized asan intuitive, rapid, reflective response “in the moment” [80]. Reflection-on-action instead happens after an experience, and is characterizedas an “inquiry into the personal theories that lie as the basis of one’sactions” [31]. A commonly employed method for reflection-on-actionin qualitative research is memoing : “Memos can help to clarify think-ing on a research topic, provide a mechanism for the articulation ofassumptions and subjective perspectives about the area of research, andfacilitate the development of the study design” [4]. We used memoingthroughout our design study to facilitate reflexivity and reflection.Pragmatically, reflection-on-action is synonymous with critical re-flection [16], an inquiry process where researchers question their as-sumptions by examining the reasoning and ideology that frame theirpractice and experiences [10, 75]. Work by Kerzner et al. employscritical reflection to construct a general framework for visualizationworkshops from their experiences running 17 of them [29]. Similarly,Satyanarayan et al. create a set of lessons for designing visualization au-thoring toolkits using what they call critical reflections [63]. Althoughnot grounded in the reflection literature, their process is similar to thatof reflection-on-action practices. Other than a handful of exampleslike these, the visualization literature is largely lacking pragmatic guid-ance on how and when to reflect [43]; this work contributes actionablerecommendations for reflecting in a design study.
Transparent reporting of a design study — through scrutinizable docu-mentation of data, methods, analysis, and artifacts — is necessary forsupporting judgments about the quality of the study and its results [44].How to report transparently, however, is an open question. Recent workby Wacharamanotham et al. provides recommendations for sharingHCI research materials based on a survey of researchers [77]. Thiswork, however, considers only software and hardware prototypes fordesign-oriented studies, missing many of the diverse artifacts producedwithin a design study such as sketches, abstractions, reflexive notes,and diagrams. In this work we experimented with recording and report-ing a diverse set of design artifacts, drawing from ideas in qualitativeresearch and research through design.In interpretivist, qualitative research, the audit trail is an establishedmechanism for transparent reporting [1, 13, 17, 35]. An audit trail is adetailed documentation of a research process that is intended for usein an audit process [1]. This process is undertaken by an (external)auditor who reviews the audit trail in order to asses the quality of thestudy, enhancing the trustworthiness of the research [35]. Althoughaudit trails are meant to increase the transparency of a study, they canalso increase the quality through explicit thoughtfulness on the part ofthe researcher on what and how to record [17]. Two recent visualizationdesign studies include audit trails as supplemental materials [29, 40],but neither study performed an audit.Transparently reporting on design decisions and insights is chal-lenging due to the ingrained nature of knowledge within the artifactsthemselves. Design scholars consider the knowledge that a designeracquires to reside in the artifacts they create [14]. This knowledge, how-ever, is implicit and often opaque [71].
Annotated portfolios — textual2 o appear in IEEE Transactions on Visualization and Computer Graphics. annotations of design patterns across a curated collection of designs— is a method used within the research-through-design community toexplicitly communicate knowledge embedded within designs [8, 21].Annotations allow for comparison of designs and highlight relationshipsbetween disparate works, from which designers can develop and com-municate generalized, intermediate knowledge. A different approach toexternalizing design knowledge is that of literate visualization, whichengages the designer in reflective documentation during the creation ofdigital, visualization artifacts [79].
ETHODS
To explore how an interpretivist approach to design study changes whatand how we learn, we set out with the goal of experimenting withthree criteria —
ABUNDANT , REFLEXIVE , and
TRANSPARENT — during anevolutionary biology design study. We positioned this work within theperspective that design studies are wicked, subjective, and diverse [44].Rogers conducted a three-month, immersive field study, followed bya design phase and a reflection phase in collaboration with Lex andMeyer. In this section we provide details about our research site anddomain collaborators, the ways we experimented with the criteria, andthe methods we employed for data collection and analysis. We directlylink to our abundant collection of evidence — for example [T45] — toprovide transparent reporting of our process.
Our study took place at two sites. In the first phase, we undertook athree-month field study in the Harmon Lab at the University of Idaho,which studies ecology and evolution through phylogenetic analysis.During this time, Rogers spent work-hours within the group’s lab,immersed in conditions similar to those in which the evolutionarybiology graduate students worked. The lab environment was open andsocial, with six desks spaced around the edges of the room, a communitycouch often inhabited by other graduate students who stopped by, anda white board filled with scattered drawings and notes. The graduatestudents used this space for their computational work, which was oftenanalysis of the phylogenetic data and field sample measurements takenfrom summer field work. This lab was chosen based on a relationshipestablished through a federally funded research project [45] between theHarmon Lab and the Visualization Design Lab at the University of Utah.The design and reflection phases took place within the VisualizationDesign Lab.During the field study we worked with seven evolutionary biologycollaborators. Two primary collaborators during this phase were Har-mon, the PI of the evolutionary biology lab, and Patton, a graduatestudent at Washington State University who works closely with the Har-mon lab, often on-site. Both primary collaborators are co-authors onthis paper. Five other graduate students in the lab served as secondarycollaborators. All collaborators were involved with the interviewsand informal feedback. The primary collaborators were additionallyinvolved with the design and evaluation of our visualization techniques.
Our decision to focus on the
ABUNDANT , REFLEXIVE , and
TRANSPARENT criteria stemmed from our experiences in previous studies and con-siderations of actionability [T160]. In previous work we attemptedto instill transparency through collecting artifacts and releasing audittrails [29, 40]. These experiences led to numerous conversations withinour research group about how to record and report artifacts in designstudies and other qualitative research studies. We saw this design studyas an opportunity to systematically experiment with abundant recordingand transparent reporting of evidence from the very start of a study. Weincluded reflexivity based on the interests of the research team and theactionability of reflexive memoing. Our approaches to meeting thesecriteria evolved over the course of the study.We attempted to instill abundance in our design study in four ways.First, we meticulously curated a rich collection of artifacts generatedthroughout the design study including field notes and reflective memos[T48], email correspondence [T90], sketchbook scans [T81], photos of collaborator sketches [T55], links to papers [T87], low- and high-fidelity visualization prototypes [T158, T96], and notes reflectivelytranscribed from audio recordings of meetings [T36]. Second, weconducted an immersive field study, in which Rogers situated herselfas a peer in the Harmon Lab for three months. Working in the com-munal space of our domain collaborators, Rogers actively engaged inresearch meetings and reading clubs focused on evolutionary topics ofinterest. She learned how to use the analysis pipelines of her collabora-tors to get a deeper understanding of the domain problem space [T47,T50]. Through time, she gained a deeper understanding of the domainresearch and developed a personal investment in our collaborators’research and social dynamics. These activities encompass the com-munal , personal , and active themes of immersive studies [23]. Third,we contacted domain experts outside the Harmon Lab in an attemptto include multiple voices and datasets. We sent emails to colleaguesof the Harmon Lab, as well as evolutionary biology researchers at theUniversity of Utah, inviting them to participate in the evaluation ofour visualization designs [T109]. Fourth, we relied heavily on sketch-ing to facilitate brainstorming of visualization ideas [T43, T52], tounderstand the domain space [T10, T38], to communicate with domaincollaborators [T55], and to aid in reflective analysis [T138].We implemented reflexivity during the field study through regular,reflective memoing by Rogers. These reflections were reflexive innature and included documenting her feelings as she became moreincorporated into the lab, her insecurities that were potentially limitingthe research [T3, T20], her interpretations on social dynamics andfriendships within the lab, and how those dynamics affected the research[T18]. Memoing was done before and after meetings and during pivot-point moments in the research process.In an attempt to transparently communicate the design study process,we created an auditable website from our collection of research artifacts,which is available at http://vdl.sci.utah.edu/trrrace/ . Thiswebsite which we call a trrrace and discuss in more detail in Section 5.2,traces the project from the field study through the design and reflectionphases, organizing the abundant collection of artifacts we recordedthroughout. The artifacts are organized in an interactive timeline andare discoverable via annotations, descriptive metadata, and directly inthe timeline. We kept a meticulous collection of all recorded artifacts starting fromthe beginning of the field study in an attempt to record an abundanceof evidence from our design study process and support transparency.These artifacts were generated throughout all three phases of research,but the content creation was concentrated during times of immersionin the field study, as well as during times of correspondence withcollaborators in the design phase of the tool. Throughout the fieldstudy, Rogers interviewed members of the lab, taking reflective notesbefore and after every interview. Pre-interview reflections included areview of previous meeting notes and outlining an agenda [T8], andpost-interview reflections summarized the main talking points andspeculated about productive next steps [T20]. Additionally, she audio-recorded these interviews and reflectively transcribed [40] them tocapture the context of what was said when, how things were said, andher interpretation of the conversations [T53]. To capture a rich view theinterviews, Rogers recorded any white-board diagrams [T94], scribbles[T41], or sketches [T55] that were generated during discussions. Inaddition to the pre- and post-interview reflections, Rogers also regularlywrote reflexive memos that included her feelings on her immersionin the lab, her insecurities that were possibly limiting the research,friendships, social dynamics, and how those dynamics affected theresearch [T3, T18, T20].During the second week of the field study, Rogers conducteda creative visualization opportunities workshop [29] with the labmembers to brainstorm about potential visualization directions. Wetook photos of all the materials generated from the workshop exer-cises and audio recorded the workshop [T23,T24,T25,T26,T27,T28,T29,T30,T31,T32,T33,T34 T35,T36].The beginning stages of sketching and prototyping began during3he field study, but the bulk of the design work and tool developmenthappened during the design phase. Our primary collaborators remainedextensively involved in providing feedback on design iterations, withmuch of this feedback happening through video calls, email, and intwo, short, subsequent visits to the Harmon Lab. We recorded feedbackemails [T90, T118], notes from the in-person feedback sessions [T125],and memos capturing personal interpretations of the feedback [T126].Design artifacts generated during this process include sketches [T43,T45, T52], mock-ups [T59], and screen-shots of prototype iterations[T67, T73, T92].
Analysis occurred during the final, reflective stage of the study whenwe started the construction of an audit trail as a website for collectingand annotating our diverse set of research artifacts. The website wasinitially designed to communicate the design study process with ahigh-level of transparency and detail. The organization and curationof artifacts, however, became a powerful catalyst for reflection thatled to significant methodological insights about our design process, aswell as new directions for the design of the visualization tool. Throughcollaborative, critical reflection among the visualization research teammembers, we iteratively developed a set of actionable recommendationsfor conducting interpretivist design study from our insights lookingacross the collection of artifacts.
REVO : A N E VOLUTIONARY B IOLOGY D ESIGN S TUDY
This design study was motivated by the complexity of our collaborators’problem in representing the rich, multivariate, and uncertain data intheir analysis. They work extensively with trees that represent hypothe-sized explanations for how species are related. In this design study wedeveloped a web-based visualization tool Trevo, that allows them toanalyze these trees with multivariate and uncertain attributes.We report on this design study in an abbreviated form as a paper-within-a-paper as part of our larger goal of highlighting the diversecontributions possible from interpretivist design study. This experi-mental format emerged from our dissatisfaction with early paper draftsthat followed a more traditional design study reporting structure [T144,T159]. We felt the traditional structure overly accentuated technicalcontributions while leaving little room for significant methodologicaldiscussions. We developed the paper-within-a-paper style to stress therole of the design study as a method of inquiry [44] that reflects andreports on a more diverse type of knowledge.
The driving question in the field of evolutionary biology is why the liv-ing world evolved the way it did? To answer this question, researchersneed to determine when a given trait evolved, such as a lizard’s longtail, and whether a particular species possesses that trait as a resultof common ancestry or of other forces such as the environment. Toanswer these questions, evolutionary biologists study a group of liv-ing organisms to establish hypotheses about evolutionary forces thatcan generalize to other species. For example, researchers study anolelizards to infer how environment influences evolution. Analysis beginsin the field, where these researchers take samples of living species andmeasure their physical characteristics, such as a lizard’s tail length,snout length, and body mass. They use these measurements of cur-rent species, typically along with DNA sequence data, to reconstructphysical characteristics of the ancestors in a species’ phylogenetic his-tory. These histories are then the basis of studying when and whytraits evolved, and whether the physical characteristics of contemporaryspecies are, or are not, a result of evolution from common ancestors.Evolutionary relationships are commonly represented as a binarytree, referred to as a phylogenetic tree . These trees are usually recon-structed by modeling the evolution of a set of DNA sequences sampledfrom present-day species. The leaf nodes of the tree represent thecontemporary species, whereas inner nodes represent their common an-cestors. All nodes in the tree have associated characteristics describedby a set of traits. Internal nodes (common ancestors) have estimated
Distance from divergenceClosenessDelta a. b.
Fig. 1. Defined preset patterns in the pattern view. (a) Pattern breakdownfor convergence (b) The six predefined patterns. values for these traits. Leaf nodes (species) have measured values fortraits.A common structure evolutionary biologists work with is clades ,which are subtrees of the larger phylogeny in which all species sharea single, unique, common ancestor. For example, for anole lizards,the main clade of study is the genus
Anolis , a group of more than 400species that all evolved from a common ancestral lizard. These subtreesare sometimes predefined, as is the case for well-established cladessuch as anoles, or they can be defined during analysis.Researchers analyze different, possible evolutionary mechanismsby studying patterns of evolution . These patterns can be summarizedin terms of how traits change, or evolve, along the branches of thephylogeny. A common pattern of trait evolution is that of divergence in which species evolve increasingly distinct trait values over time [60].Another pattern, convergence which is shown in Figure 1(a), is char-acterized by traits that diverge early in two species’ histories, but thenconverge later in their evolutionary histories by developing similar oridentical traits [28]. Convergence is an indication of adaption — certaintraits evolve repeatedly because they are beneficial in an environment— and has been studied extensively in the anole lizards. Many of theselizards, having split off from their common ancestors a long time ago,inhabit similar environments on separated islands and have evolvedvery similar characteristics as a consequence. Although other interest-ing patterns besides divergence and convergence exist, such as those inFigure 1(b), they do not have standardized names.Identifying patterns of evolution is a challenging analysis problemthat involves accounting for changes to multiple traits under uncertaintyin the context of the tree topology. We worked with our collaboratorsto explore new ways to enable this complex analysis with interactivevisual analysis tools.
In the datasets our collaborators are analyzing, evolutionary relation-ships are represented as rooted trees. Bifurcations in the tree representspeciation events. Internal nodes encode hypothesized common ances-tors of existing species, which in turn constitute the leaf nodes. The sizeof the trees we focused on here ranged from 20 to 200 species (leaves),each associated with 5 to 25 traits. Traits of a species can be discrete orcontinuous and are uncertain for the reconstructed (inner node) species.Reconstructed discrete traits, such as the geographic location where aspecies is found or whether they lay eggs, are specified as probabilities.Continuous traits, such as tail length, are given as an estimated valueand a 95% confidence interval.To explain why the living world evolved the way it did, our col-laborators’ analysis is focused on understanding when and how traitsevolved in a population, which requires viewing trait values for multipleattributes in the context of the topology of the tree. We break down thislarger analysis goal into three domain tasks:
T1: Understand the uncertainty in multiple reconstructed traits.
Significant uncertainty exists in the reconstructed traits for internalnodes, so adequate visual representations of trait values and their un-certainty are critical. Current methods for visualizing attributes inphylogenetic trees are limited to showing one or two traits at a time,and frequently cannot encode uncertainty [T42, T36, T16]. This task isorthogonal to all other tasks, i.e., uncertainty analysis is a part of everyanalysis task.4 o appear in IEEE Transactions on Visualization and Computer Graphics.
T2: Analyze subtrees.
This task is concerned with creating and analyzing individual subtrees(clades) and comparing between multiple subtrees.
T2.1: Create subtrees.
Our collaborators need the ability to create subtrees by topology andtrait values. For example, an analyst might want to create two subtreesbased on an attribute, such as the island a species is inhabiting [T64,T80]. Definitions of subtrees might also be given as formal clades in adataset.
T2.2: Analyze attribute distributions in subtrees.
Our collaborators need to be able to identify significant changes inmultiple traits at once. For example, understanding whether a shifttoward a longer tail is correlated with a shift toward longer hind-legs cangive hints about the underlying causes of that change [T20]. Viewingmultiple traits at once is particularly difficult for our collaborators,who rely on comparisons of reconstructed traits on separate trees [T36,T72].
T2.3: Identify evolutionary outliers.
It is important for our collaborators to identify individual species, paths,or subtrees that have significantly different trait values compared to therest of the subtree [T17, T91]. For example, they want to identify pathswith species that have a larger body mass than the rest of the subtree.
T2.4: Compare attribute distributions of multiple subtrees.
Comparisons are important in characterizing what makes a subtreeunique. For example, our collaborators want to study whether thespecies in a subtree share common characteristics, such as head andtail length, that set them apart from the rest of the tree. To study howtraits evolved through history, they need to understand how subtree traitdistributions diverge and where this happens in the tree [T4, T20, T66,T72, T88].
T3: Identify and analyze evolutionary patterns
An important task in our collaborators’ analysis is identifying the evolu-tionary patterns that indicate certain mechanisms underlying evolution[T53, T64, T87]. Identifying these patterns requires the comparisonof trait trajectories of multiple species in a tree [T80,T93]. To identifyconvergence, for example, an analyst would search for two paths thatseparated early in the tree with trait values that first diverged, but thenlater converged.
Visualization of phylogenetic data is challenging in three ways: (1) thetrees can be large, requiring sophisticated navigation and/or aggregationstrategies to browse them; (2) the topology of the trees is uncertain,requiring the comparison of multiple alternative trees; and (3) the treesare associated with many (uncertain) attributes, requiring sophisticatedmultivariate tree visualization strategies. Our work addresses the thirdproblem, multivariate trees, but we briefly review all areas.The scale and uncertainty of topology remain challenges in phyloge-netic research and numerous visual solutions have been proposed forboth [5, 7, 33, 34, 36, 51, 62]. Large phylogenetic trees and topologicaluncertainty are not key problems for our collaborators; visualizing treeswith many attributes, however, is. As a generalization, visualizingmany traits in the context of a tree is a type of multivariate networkvisualization problem. Nobre et al. recently described the design spaceof a multivariate network visualization in a survey that included treevisualization [53]. We here focus mostly on approaches for phylogeniesbut refer readers to this survey for a broader overview.Within the evolutionary biology community, visualizations of phylo-genetic data are used for both exploration and presentation in papers.Most figures found in evolutionary biology papers show trees laid outusing node-link diagrams with either linear or circular layouts, andon-node or on-edge encoding to show trait values [58, 60]. Thesefigures are often created with interactive tools such as iTOL [34] orDendroscope [27], or using scripted plotting libraries, such as phytoolsor ggtree for R [57, 81]. Tools such as iTOL can visualize multipleattributes for the leaves, but the inner nodes are usually limited to asingle attribute. Analysts, however, often need to account for multipletraits at once to identify underlying forces influencing trait change. Intheir current workflow, they compare different traits mapped to the nodes of multiple trees side-by-side. Such comparisons are difficultwith just 2 traits, but analysts must often consider up to 10 traits for agiven tree. As expressed by one of our collaborators, “if you have 1continuous trait you can do things. If you have 2 — OK. If you have 3or 4 or 5, there is nothing really sufficient” [T36].In the visualization community, several tools have been designed tovisualize trees with attributes. Lineage [52], for example, visualizesattributes for genealogical trees using a linearization approach, wherethe attributes are shown in a table; Juniper is a generalization of thismethod to networks [54]. Other tools, such as TreeVersity2 [22], vi-sualize attributes using implicit layouts. Researchers currently haveno tool suitable for visualizing many traits for inner nodes and leavesunder uncertainty in the context of phylogenetic trees.
Two technical contributions emerged from this design study. The firstis a technique for visualizing summary distributions of attributes in a(sub)tree — the trait view — designed to address the analysis of sub-trees (T2). The second contribution is a view for querying, ranking, andvisualizing patterns consisting of topological and attribute features —the pattern view — designed to address the identification and analysisof evolutionary patterns (T3). Both views visualize uncertainty (T1)and were implemented in a web-based tool we call Trevo, along withtwo additional views: https://vdl.sci.utah.edu/Trevo/ . A crucial task for our collaborators is analyzing patterns of attributeswithin and between subtrees. When subtrees are defined topologically,this analysis can be supported in the context of a phylogenetic tree.For subtrees defined based on trait values, however, species can bescattered across a phylogentic tree. For example, our collaboratorswant to create two subtrees for anole species that are found on theislands of Hispaniola and Cuba so they can compare the distributionof body mass of the lizards on these islands to study any environmen-tal effects that might appear. The “island” trait does not clearly splitthe phylogenetic tree into disjunct subtrees, as common ancestors col-onized islands multiple times. It instead creates trees with partiallyoverlapping branches. Figure-2(b) shows these disjunct subtrees withthe species color coded by island. Lizards originating from Hispaniolaare colored green, and those originating from Cuba are colored blue.Our collaborators compare the subtrees’ trait values through the evo-lutionary history to determine when and how these groups began todiverge, for example, to determine if there is a difference in body massbetween the two islands and when this divergence in traits began tooccur along the evolutionary history. Identifying differences in valuetrends and when they occur within the phylogenetic tree can be difficultgiven the overlapping topology.Through an iterative design process with our primary collaborators[T68, T74, T108, T114], we tackled this challenge with an aggregationsolution for creating trait-defined subtrees. The key aspect of this newtrait view is that it enables analysts to filter branches of the tree based ontraits of the leaves. Figure 3 shows the steps involved in transforming anode-link tree layout into the trait view. Initially, the tree is filtered toinclude only extant species with a certain attribute such as the greenleaves in Figure 3(a). We then leverage temporal information to binthe other nodes in the subtree by time, shown in Figure 3(b). Theleaves are assigned a separate bin for which the uncertain discrete-and numerical-trait distributions are visualized in columns. Nodes areshown at the top of the bin; their horizontal position is driven by theirtime attribute, allowing analysts to compare multiple uncertain traitdistributions in a temporal context unhindered by the tree’s topology.Next, we use different encodings for leaf nodes with known trait valuesversus inner nodes with uncertain ones, shown in Figure 3(c). Theknown attributes of the leaves are encoded using histograms. Forcontinuous uncertain traits we show the median plus a 95% confidenceinterval for the estimated values and a kernel density estimate plot.Finally, probabilities for uncertain discrete traits are represented in thetrait view as separate one-dimensional dot plots for each state; to reduce5 ig. 2. Trait view showing four continuous and two discrete trait variables for 100
Anolis lizard species. (a) Outliers in the last SVL bin are brushed. Atraditional phylogenetic tree view, shown on the right, can be used to define subtrees. (b) Leaf nodes can be color-coded by trait category. This detailview shows all leaf nodes color-coded by island of origin. These categories can be used to define subgroups by trait category or value, independentof the topology of the tree. A A DB C b. A DB C a. Time Time bin 2
Uncertain internal nodes
A B C D c. Known leaf nodes
Leaf bin
EFG
EFG E F G cont. traitdiscrete traitTime bin 1
Fig. 3. Transforming a phylogenetic tree into the trait view. (a) We selecta subtree by brushing for a trait in the leaves, shown in green. (b) Thesubtree is binned by time intervals and the leaves are assigned a sepa-rate bin. (c) We show continuous uncertain traits using a median plusconfidence interval visualization and a KDE plot. For discrete uncertaintraits we use multiple dot-plots, one for each trait category. Known traitsare visualized using histograms. the risk of overlapping dots, we use transparency and vertical jitter. Theaverage for each state probability is plotted as a line in the plots.
The pattern view allows analysts to query for and find pairs of pathsthat follow a specific pattern of evolution such as convergence and di-vergence. Patterns of evolution are characterized by three key metrics:distance, delta, and closeness. The distance between two species refersto time and topological distance up to the first common ancestor.
Delta is the maximum difference in an estimated continuous trait value afterthe species diverge.
Closeness is the difference in a specific, continuoustrait value between the extant species. We developed a query inter-face, shown in Figure 4(a), that analysts can use to define patterns ofinterest based on these three characterizing parameters. We found thatwhile these simple parameters cannot represent arbitrary patterns, theycovered all the patterns of evolution our collaborators are interestedin. To simplify the pattern definition, we also developed six presetpatterns that an analyst can choose from to score pairs of paths. Thesepatterns, shown in Figure 1(b), emerged from repeated iterations withour collaborators [T94, T96,T129].To create a ranking for paths that match a specified pattern wecalculate scores for all possible pairs of leaves using the selected patternparameters for all traits. We then rank the pairs of paths based on theinitial trait chosen by the domain expert, and visualize the two pathsusing a ranked list of line+area charts, as shown in Figure 4(b). In thischart, the vertical axis corresponds to the trait value. Individual speciesare shown as squares, which are positioned to be centered on their mostlikely trait value. The height of the box shows the confidence interval.The boxes are connected by lines for the most likely value, and areasfor the confidence interval.One limitation of our original design of the pattern view was that it could only show a single trait at a time [T96]. In an early feedbackmeeting, our collaborators asked if it was possible to have an indicationof whether a specific pair of paths was also ranked highly for othertraits [T99, T112, T113]. That is, in some cases the analysts might beinterested in identifying species pairs that have converged in severaltraits, rather than just one. Convergence of sets of traits is of particularinterest because such cases can provide the strongest evidence foradaptation to particular environments. To address this shortcoming,we added a supplementary heat map to the side of the pair plot thatindicates whether the pair is ranked in the top 1% for a given pattern inany other traits in the data set, shown in Figure 4(b) on the right. Here,each square in a heat map represents other traits, where squares withdarker saturation have a higher ranking. To find which pairs are rankedhigh for the pattern in the largest number of other traits, they can besorted by frequency of top rankings from the heat map.
We validate the trait and pattern views instantiated within Trevo bydemonstrating their usefulness in a case study. The case study wasconducted and written by our primary collaborators, who are also co-authors of this paper, and focuses on one of their primary datasets ofthe Anolis lizard genus. We provide a brief summary of findings here.We do not include the more detailed case study in this paper-within-a-paper, instead linking to it as external evidence [T145], as we find thatdomain-specific case studies often do not significantly contribute to abroader understanding of research contributions in design studies, butare rather akin to analysis scripts used in quantitative data analysis: theyare necessary to ensure validity and trust, but do not convey knowledgeon the subject of the research.Using the trait view, our collaborators were able to reduce theiranalysis to a subset of species that exhibit exceptionally large bodyfeatures, and to see how body features evolved differently over time.Traditional visualization approaches would have required coloring dis-junct branches in a phylogenetic tree and making difficult judgmentsabout color variations; the trait view instead provided targeted anal-ysis using spatial encoding of the traits of interest. With the patternview, our collaborators were able to confirm a known convergence anddivergence event, a task not possible with commonly used softwarefor the phylogenetic analysis of trait evolution. Furthermore, they wewere able to identify a new pattern of convergence in a pair of species,leading them to new biological questions about the evolutionary forcesat play. This case study shows that our collaborators not only couldeasily distinguish interesting patterns in their data using Trevo, but alsodocument a previously unknown insight. We offer this case study asevidence of the validity of our proposed designs [50].6 o appear in IEEE Transactions on Visualization and Computer Graphics.
Delta ClosenessDistance from divergence Other top ranked traits b.a.
Fig. 4. Pattern view components. (a) The user interface allows selection of a preset pattern, refined by adjusting the parameters for Distance, Delta,and Closeness. This interface also sorts rank pairs by top score or top rank frequency. (b) The first-ranked pair of paths (the species
Anolis insolitus and
A. angusticeps ) for a convergence pattern for the trait “snout vent length”. The line/area chart shows the most likely values and the associateduncertainty of the trait of consecutive species, with the “delta” between the species in the trait being evident in the middle. Individual species in thetwo extant species’ ancestry are shown as rectangles. The heat map on the right show where other traits rank based on the selected pattern.
We developed a web-based visualization tool we call Trevo in collabo-ration with evolutionary biologists to analyze phylogenetic trees withmultivariate and uncertain attributes. In this paper-within-a-paper wecontribute two novel visualization techniques implemented in the traitview and the pattern view. The two views prioritize visualizing theattributes in multivariate, phylogenetic trees over detailed topologicalinformation. This prioritization is by design. As the tree topology itselfis highly uncertain, visualizing uncertain attributes on uncertain nodesis not helpful. Instead, our approach aggregates relevant subtrees bytime and visualizes possible attribute distributions for temporal bins.The pattern view similarly prioritizes attributes with only rough topo-logical measures, such as the time two species diverged. It is the firstapproach that allows researchers to query for complex evolutionarypatterns based on a trait and topology, and explore these patterns acrossmultiple traits.Trevo is being integrated into the computational workflow of theHarmon Lab. Additionally, the development of Trevo is part of a largersoftware project for creating MultiNet, a web-based tool for visualizingand analyzing multivariate networks [45]. The visualization insightsgenerated from this design study are informing aspects of MultiNet’sdesign. We discuss the methodological insights generated from thisdesign study in the next section.
ETHODOLOGICAL R ECOMMENDATIONS
Our experiments with design study criteria for rigor — specifically
ABUNDANT , REFLEXIVE , and
TRANSPARENT — offered us a wealth of op-portunities to try new things, and to learn along the way. Through acritically reflective process, we distilled our learning into three method-ological recommendations for conducting iterpretivist design studies.
Reflection is a critical aspect of design study [66], yet little is knownon how and when to do this in practice [43]. In our work we reflectedregularly, and reflexively, documenting our reflections as we progressedthrough the study. What we found is that systematic reflection shiftedthe course of our research in productive and demonstrable ways.For example, when Rogers arrived in our collaborators’ lab at thestart of the field study, she initially felt uncomfortable audio-recordingher interactions with them. Because she was not familiar with the groupand the group was not familiar with her, she felt like an intruder in thelab. In a reflective note from one of her first interviews, she noted:
I have not been recording these interviews as I am in thefirst week and I do not want to be intrusive. [T3].She was, however, aware that audio recording would be beneficial toher note taking:
I believe the recording will help me capture more than Ican get from my note taking, and maybe more importantly,be more engaged in the interview process. I was initiallyhesitant to ask people to record them during my initial time here because I was new and unfamiliar and wanted our firstinteractions to be more candid. [T53].The following week she decided to audio-record the participatory work-shop she ran with the lab, and reflected on the experience:
I am glad I recorded the workshop — as I have re-listenedto it and transcribed parts I felt were significant to the goalof the design study. Returning to the audio at a later timeallowed me to notice things that people said when I wasengaged in a conversation with someone else or did nothave the base knowledge on a particular subject to want towrite the moment down initially. [T36]Rogers’ concerns about her intrusive presence in the lab made herinitially hesitant to audio-record interviews, to the detriment of her datagathering. After writing several reflexive memos detailing her feelings,and reflecting on the success of audio-recording the workshop, shechanged her interview method and audio-recorded all interviews withcollaborators. Off-loading the work of note-taking to the recordingallowed her to engage in a more conversational, constructive way whenconducting interviews:
I found [audio-recording] extremely helpful as I was ableto engage in conversation more easily than when I wasattempting to take speed notes....The recording seems toblend into the scene and you forget its running after a coupleof minutes. I will be using a recorder from now on. [T53].By reflecting on her actions, Rogers was able to adjust and improveher research practices. Systematic, reflexive notes such as these areencouraged by qualitative researchers as they offer “a partial means forproviding checks on the researcher’s own biases” [35] and a mechanismto “detect and correct deviations from the design goal early” [59].The start of audio-recording within the design study led to our sec-ond example of productive reflection. After conducting an interview,Rogers would listen to the audio-recording from the interview andreflectively transcribe it within a day or two. Transcription did notinvolve transcribing the audio-recording word for word, but was insteada reflective memo synthesizing the main points taken from the audioalong with concrete quotes as evidence for these findings. When some-thing stood out in the recording, Rogers would memo what time in theaudio this happened, allowing her to easily revisit how something wassaid at a later time [T36].We find that reflectively transcribing an audio-recording — versusrelying on an (automatically or externally generated) word-for-wordtranscription — offers two advantages for analysis. First, listening tothe audio while taking notes slows us down, allowing for a deeper,more thoughtful analysis. Writing down reflections requires us to stop,rewind, and listen to things multiple times, resulting in better notes andinterpretations. Second, we find that listening to a recording allows usto re-experience the interview, but in a more detached and reflectiveway. This allows a new perspective on the discussion, separate fromthe one we experienced in the moment [T126].7ur third example of productive reflection occurred as we con-structed an audit trail of our collected artifacts in order to produce atransparent trace of the design study process. Upon revisiting her oldsketches, Rogers noticed that her design concepts for certain compo-nents were very narrow, particularly for an early version of the traitview [T705, T709, T731]. She reflected on the narrow design conceptsduring a meeting:
I get fixated on one design and I can see that in the sketchesin my sketchbook. [ T111].This reflection prompted Rogers to attempt a redesign of the traitview’s discrete plots, which had last gone through design iterationsthree months prior. Having recently reviewed her notes she took duringthe field study as she added them to the audit trail, she found newmeaning, and new ideas for her redesign:
I still find details that I missed at the time of a meeting or atan initial reflection. [T111].The redesigned trait view, shown in Figure 2, shows relationshipsbetween trait values and their probability distributions that were notshown in earlier designs.We did not anticipate that the act of curating and organizing arti-facts would facilitate productive reflection and play a role in designdevelopment. This redesign would likely not have occurred without thereflective processes of revisiting past notes, a concept emphasized inwork on systematic reflection for design in engineering. By adoptingregular reflection during design, “the chance of overlooking importantaspects is decreased” [59]. Tavory and Timmermans advocate for revis-iting experiential notes to reconsider them with newfound knowledgeor perspective: “We are constantly re-experiencing parts of our worldas we go about the business of living. When we move through oursurroundings, we not only encounter new problem situations but findnew problems in old situations” [74].
RECOMMENDATION
Our work shows that adopting regular, sys-tematic, reflective practices within a design study can improve theresearch methods, domain understanding, and visualization designs.We recommend four opportunities for reflection-on-action. First, takereflective notes before and after interviews with domain collaborators.This activity takes only a few minutes but significantly improves thefocus of an interview as well as captures initial interpretations andideas for next steps. Second, include reflexive considerations in yourfield notes. Reflecting on changing perspectives, biases, methodologi-cal rationale, and feelings can be a valuable source of insight. Third,audio-record interviews and analyze them via reflective transcription.The reflective transcription should occur soon after the interview tosupport experiential recall on the part of the researcher. Fourth, revisitearly notes and sketches. During these revisits look for opportunities toreinterpret experiences through a new lens of deeper understanding.
Providing a transparent, scrutinizable trace of a design study is essentialfor allowing judgments about the quality of the research [44]. As wedeveloped an auditable trace through our collection of research artifacts,we found, however, that revisiting evidence also supported productivereflection that shifted and changed the course of the study. Supportingdifferent ways to trace the design study process was important forencouraging both transparency and reflection in our study.From the start of the field study, we meticulously collected a richset of research artifacts in order to abundantly document our researchprocess. We stored the artifacts in an online repository, and created arecord for each in a spreadsheet that included a descriptive title, thedate it was created, a unique id, and the research artifact type suchas meeting note, sketch, email, etc. Building on this collection ofevidence, we experimented with transparent reporting by creating anaudit trail of the artifacts. Our initial, web-based design of the audittrail was inspired by those created for other experiments on reportingdesign studies [29, 40]. Like previous examples, our website traced thedesign study temporally by visually organizing artifacts on an overview timeline, and providing access to the recorded artifacts themselvesthrough a details-on-demand side panel. Each artifact is represented onthe timeline as a square, color-coded by its type [T161].While building the audit trail, we reflectively engaged with theresearch artifacts, leading to demonstrable changes within our study,as we previously discussed in Section 5.1. This engagement shiftedthe audit trail toward use as an internal, research tool. We found thatwe wanted to trace research concepts across the study, including ourgrowing understanding of domain principles such as convergence anduncertainty, as well as our criteria experiments through reflexivity andsketching. To support concept tracing we extended our metadata foreach research artifact to include tags that pull information embeddedwithin the artifacts. These concept tags allow for a trace of how ourawareness and understanding of various concepts evolved throughoutthe study. We extended the website to include the concept tags for eachartifact in the detail view; clicking on a specific tag highlights otherartifacts with the same tag in the timeline overview [T161].The final iteration of our tool supports an unanticipated range ofresearch tasks: recording diverse research artifacts, reflecting on con-ceptual developments, and reporting on the design study process. Itis a trace of our research process from two perspectives: a temporalperspective for transparent and auditable reporting and a conceptualperspective for reflective research practices. We consider this tool to bea trrrace , as both a speculative nod to material traces [55] and to therecord, reflect, and report tasks it supports.The trrrace has theoretical connections to both audit trails [1, 35]and annotated portfolios [8, 21]. As referential material [35], our re-search artifacts are evidence of the design study process [25, 47, 77, 83],capturing fleeting aspects of the study that led to insight. Organiz-ing these artifacts temporally provides a trace of the study itself [55],providing an auditable mechanism for reviewing the quality of theresearch [48, 73, 77]. Our research artifacts are also manifestationsof design knowledge [21], with the knowledge engrained within theartifact [15]. Each artifact’s concept tag, created from the artifact itself,is an annotation, allowing for a trace that connects seemingly disparateartifacts through more general concepts. These theoretical connectionspoint to an opportunity for further theorizing about, and experimentingwith, design study trrraces.
RECOMMENDATION
Our experiments with abundant evidence andtransparent reporting led us to the concept of a trrrace, which supportsrecording, reflecting, and reporting in design study. We recommendthat design study researchers plan for a trrrace early in a study andconsider three important issues. First, the process of collecting artifactsgreatly benefits from establishing a system for organization early on.We used an online spreadsheet and adopted a regular practice of addingrecords of digitized artifacts as we generated them. Second, developmechanisms to automatically extract concept tags from the artifactsthemselves. We extracted concepts from the artifacts manually forthis project, but in future work we plan to develop an improved, semi-automated approach. Third, the immersive, ethnographic nature ofdesign study requires considerations of how to handle privacy, as wellas anonymization for review. We encourage developing a system foranonymizing artifacts early in the study process. Additionally, we findthat the best method for navigating transparent recording of a studyis to be transparent: tell your collaborators when you are recording,establish what will be on- versus off-record, provide them access toyour notes, and be aware of recording delicate social dynamics.
Employing appropriate and justified research methods within a designstudy is necessary for achieving rigor, but a checklist of methods isnot sufficient for arguing that a study is rigorous. The design studyrigor criteria are meant to provide guidance on what to achieve, not how to do so [44]. Evidence of the criteria within a study is the proof.The type, extent, and depth of evidence that is sufficient for arguingthat a design study meets the criteria for rigor, however, is an openquestion, and likely one without a standardizable answer. As part of ourexperiments we reflected over our research artifacts and experiences,8 o appear in IEEE Transactions on Visualization and Computer Graphics. looking for evidence of the criteria. We found that shifts in the way wecommunicated and interacted with our collaborators suggest that ourstudy was
INFORMED and
ABUNDANT .During the early stages of our field study, work discussions with ourcollaborators centered around semistructured interviews. We organizedinterviews to have 2-3 in a single day and scheduled interview days ev-ery few days. Rogers saved up questions she had until these interviews[T7, T53]. The infrequent discussions were relatively long in duration,lasting from 1-2 hours at a time, were dense with domain information,and had a formal tone. Post-interviews, Rogers would revisit and lookup domain concepts and vocabulary that emerged from these interviewsas she was building her understanding of the domain:
The paper [linked] below was a really good resource forgetting an understanding of the group comparisons thatindicate adaptive events such as convergence.... People Ihave interviewed touched on these concepts, but becausethe concepts are complex and varied, it is really hard to geta good synthesis of the main points. I feel as if I am hearingrecurring words that come up in conversation, but I havebeen missing the connection between them. [T87]As the field study progressed and Rogers felt increasingly comfort-able asking questions outside of scheduled interviews, work communi-cations shifted to shorter, informal discussions and texts. The languageof communications also shifted as Rogers increasingly used domainvocabulary and concepts fluently. For example, Rogers saw some un-expected biological relationships in the data while developing one ofthe visualization views, and messaged a collaborator to confirm herobservations:
Rogers: WOOP. Saving the summary view.AP: BooyahRogers: One thing I noticed the other night is that a lotof the convergent pairs are not both the same ecomorph— but because we are looking at a single trait, wouldit make sense that two ecomorphs would have similarcharacteristics for a single trait? Ex: trunk crown and twighaving similar PCIII Padwitch vs tail?!AP: Hmmmm.... You’re finding that even when using thePC traits? Because those PCs are essentially composites ofmultiple traits [T133]
The texts continued as Rogers also excitedly communicated her findingsof a problem with the pattern ranking system:
Rogers: THINK I FOUND A BUG IN THE DELTA.AP: Oh *** What’s it doing? [T133]
At the time of this text exchange, Rogers had spent significant timeengaged with the domain, and she understood enough about domainconcepts to identify mismatches in what she saw in the data. Further-more, identifying these mismatches excited her.This exchange aligns with indicators for immersion: using domain-specific language the researcher engages in “informal peer-to-peercommunication with domain experts about domain science and visu-alizations”, eventually becoming “concerned with, affected by, andpersonally involved in the other domain” [23]. Design by immersionis an approach that, through long-term, committed engagement, pro-vides visualization researchers an abundant exposure to a domain space,allowing them to develop a deeply informed understanding.Every design study, like other qualitative inquires, is unique incomplex ways and thus requires the construction of careful, thoughtfularguments for its quality: “Excellent research is not achieved solelyby the use of appropriate strategies or techniques. The skillful use ofstrategies only sets the stage for the conduct of inquiry” [49]. Changesin the way we communicated with our collaborators — not the timewe spent in the field or the number of interviews we conducted —suggests that our design study met aspects of both the
INFORMED and
ABUNDANT criteria. We argue for careful argumentation, backed up byrich evidence and grounded in existing literature and theories, as a general model for supporting claims of rigor in design studies. Beingreflexive and noticing not only how we affected the research, but alsohow it affected us, offered us opportunities to more deeply reflect onthe impacts of our criteria experimentations. We speculate that manysuch opportunities may be found in any design study.
RECOMMENDATION
Knowing when a design study has reacheda critical threshold for establishing rigor is difficult, with no single,objective metric. Through critical reflection we positioned our experi-ences and evidence — shifting patterns of communication — withinexisting theoretical concepts — design by immersion [23] — allowingus to build links between what occurred in our research and what itcould mean. We recommend that design study researchers plan for thetime and space to engage critically and reflectively with their researchartifacts and experiences; propose, repropose, and repropose again howwhat they learned engages with the existing literature; and resist theurge to argue that a study is rigorous because of a checklist of meth-ods employed, instead looking for things that changed, shifted, andsurprised.
ONCLUSION
This paper reports on an interpretivist design study and a resulting di-verse set of knowledge outcomes consisting of visualization techniques,methodological insights, and new methods for reporting. We found thatour experiments with establishing rigor through the
ABUNDANT , REFLEX-IVE , and
TRANSPARENT criteria led to a myriad of learning opportunities.Yet those opportunities are messy, overlapping, and difficult to distill.For example, our efforts to provide transparency relied on abundantdata collection, and (reflexively) changed our writing methods as wecrafted this paper. We learned much more than we have reported, butthe challenge of aligning the evidence, our experience, and existing the-ory kept us from fully synthesizing the rich learning this interpretivistdesign study provided.One such example is the trrrace construct we propose for recording,reflecting, and reporting in design study. The idea of the trrrace emergedas we worked to enhance the transparency of this report. The more welinked into our collection of artifacts, the more we noticed how theselinks provided useful traces of our research process. We also becameaware of challenges for a mechanism like a trrrace that is used in boththe research and reporting processes: how do we ensure persistenceof the trrrace and the myriad artifacts it links together? How do weconsider privacy concerns, as well as anonymization constraints? Howdo we develop and maintain a trrrace in a way that does not slow downdesign-oriented research? How do we improve our recording practicesto enhance the traceability of a trrrace? How do we report a trrrace in away that is accessible, understandable, and scrutinizable?This last question offers opportunities to reflect on current practicesfor reporting through traditional supplemental materials that can, atbest, tell a curated story parallel to a paper, but at worst, can be animpenetrable dump of information. What types of visualizations, inter-actions, and interfaces can we design to help a reader navigate a trrrace?How might we tell a data-driven story from an abundant collection ofevidence? If we embrace the concept of material traces, how might thisfundamentally change the way we think about supplemental materials,transparency, and reproducability? Developing theory and pragmaticguidance for design study trrraces is one of the more exciting futuredirections pointed to by this work. We hope this paper is a catalyst forfurther conversations about trrraces, as well as the broader opportunitiesand challenges for interpretivist design study. A CKNOWLEDGMENTS
We thank the members of the Harmon lab, the Visualization Design Lab,and the MultiNet team for their participation, feedback, and support.We also appreciate the thoughtful comments from the anonymous re-viewers that helped to strengthen this paper. We gratefully acknowledgefunding by the National Science Foundation (OAC 1835904).9
EFERENCES [1] S. Akkerman, W. Admiraal, M. Brekelmans, and H. Oost. Auditing qualityof research in social sciences.
Quality & Quantity , 42(2):257–274, 2008.doi: 10.1007/s11135-006-9044-4[2] S. Avle and S. Lindtner. Design(ing) “Here” and “There”: Tech En-trepreneurs, Global Markets, and Reflexivity in Design Processes. In
Proceedings of the 2016 CHI Conference on Human Factors in ComputingSystems , CHI ’16, pp. 2233–2245, 2016. doi: 10.1145/2858036.2858509[3] S. Bardzell. Feminist hci: taking stock and outlining an agenda for design.In
Proceedings of the SIGCHI conference on human factors in computingsystems , pp. 1301–1310, 2010. doi: 10.1145/1753326.1753521[4] M. Birks, Y. Chapman, and K. Francis. Memoing in qualitative research:Probing data and processes.
Journal of Research in Nursing , 13(1):68–75,2008. doi: 10.1177/1744987107081254[5] F. Block, M. S. Horn, B. C. Phillips, J. Diamond, E. M. Evans, and C. Shen.The DeepTree Exhibit: Visualizing the Tree of Life to Facilitate InformalLearning.
IEEE Transactions on Visualization and Computer Graphics ,18(12):2789–2798, 2012. doi: 10.1109/TVCG.2012.272[6] S. Bødker. When second wave hci meets third wave challenges. In
Proceedings of the 4th Nordic conference on Human-computer interaction:changing roles , pp. 1–8, 2006. doi: 10.1145/1182475.1182476[7] R. R. Bouckaert. DensiTree: Making sense of sets of phylogenetic trees.
Bioinformatics , 26(10):1372–1373, 2010. doi: 10.1093/bioinformatics/btq110[8] J. Bowers. The logic of annotated portfolios: Communicating the valueof ’research through design’. In
Proceedings of the Designing InteractiveSystems Conference , DIS ’12, pp. 68–77, 2012. doi: 10.1145/2317956.2317968[9] M. Brehmer, S. Ingram, J. Stray, and T. Munzner. Overview: The design,adoption, and analysis of a visual document mining tool for investigativejournalists.
IEEE transactions on visualization and computer graphics ,20(12):2271–2280, 2014. doi: 10.1109/TVCG.2014.2346431[10] S. Brookfield. Critically reflective practice.
Journal of Continuing Edu-cation in the Health Professions , 18(4):197–205, 1998. doi: 10.1002/chp.1340180402[11] R. Buchanan. Wicked Problems in Design Thinking.
Design Issues ,8(2):5–21, 1992. doi: 10.2307/1511637[12] B. Buxton.
Sketching User Experiences: Getting the Design Right and theRight Design . Morgan kaufmann, 2010.[13] M. Carcary. The research audit trial–enhancing trustworthiness in qual-itative inquiry.
Electronic Journal of Business Research Methods , 7(1),2009.[14] N. Cross. Design research: A disciplined conversation.
Design issues ,15(2):5–10, 1999. doi: 10.2307/1511837[15] N. Cross. Expertise in design: an overview.
Design Studies , 25(5):427–441, 2004. doi: 10.1016/j.destud.2004.06.002[16] A. Daley. Reflections on Reflexivity and Critical Reflection as Crit-ical Research Practices.
Affilia , 25(1):68–82, 2010. doi: 10.1177/0886109909354981[17] R. De Kleijn and A. Van Leeuwen. Reflections and review on the auditprocedure: Guidelines for more transparency.
International Journal ofQualitative Methods , 17(1), 2018. doi: 10.1177/1609406918763214[18] S. P. Dow, A. Glassco, J. Kass, M. Schwarz, D. L. Schwartz, and S. R.Klemmer. Parallel prototyping leads to better design results, more di-vergence, and increased self-efficacy.
ACM Transactions on Computer-Human Interaction , 17(4):18:1–18:24, 2011. doi: 10.1145/1879831.1879836[19] L. Finlay. Negotiating the swamp: the opportunity and challenge ofreflexivity in research practice.
Qualitative research , 2(2):209–230, 2002.doi: 10.1177/146879410200200205[20] C. Frayling. Research in art and design.
Royal College of Art ResearchPapers , 1(1), 1994.[21] B. Gaver and J. Bowers. Annotated portfolios.
Interactions , 2012.[22] J. Guerra-Gomez, M. L. Pack, C. Plaisant, and B. Shneiderman. Visualiz-ing Change Over Time Using Dynamic Hierarchies: TreeVersity2 and theStemView.
IEEE Transactions on Visualization and Computer Graphics(InfoVis ’13) , 19(12):2566–2575, 2013.[23] K. W. Hall, A. J. Bradley, U. Hinrichs, S. Huron, J. Wood, C. Collins, andS. Carpendale. Design by Immersion: A Transdisciplinary Approach toProblem-Driven Visualizations.
IEEE Transactions on Visualization andComputer Graphics , pp. 1–1, 2019. doi: 10.1109/TVCG.2019.2934790[24] D. Haraway. Situated knowledges: The science question in feminism and the privilege of partial perspective.
Feminist studies , 14(3):575–599, 1988.doi: 10.2307/3178066[25] B. Hartmann, M. R. Morris, H. Benko, and A. D. Wilson. Pictionaire:Supporting collaborative design work by integrating physical and digitalartifacts. In
Proceedings of the 2010 ACM Conference on ComputerSupported Cooperative Work , CSCW ’10, pp. 421–424, 2010. doi: 10.1145/1718918.1718989[26] U. Hinrichs, S. Forlini, and B. Moynihan. Speculative practices: Utilizinginfovis to explore untapped literary collections.
IEEE transactions onvisualization and computer graphics , 22(1):429–438, 2015. doi: 10.1109/TVCG.2015.2467452[27] D. H. Huson and C. Scornavacca. Dendroscope 3: An Interactive Tool forRooted Phylogenetic Trees and Networks.
Systematic Biology , 61(6):1061–1067, 2012. doi: 10.1093/sysbio/sys062[28] T. Ingram and D. L. Mahler. SURFACE: Detecting convergent evolutionfrom comparative data by fitting Ornstein-Uhlenbeck models with step-wise Akaike Information Criterion.
Methods in Ecology and Evolution ,4(5):416–425, 2013. doi: 10.1111/2041-210X.12034[29] E. Kerzner, S. Goodwin, J. Dykes, S. Jones, and M. Meyer. A Frameworkfor Creative Visualization-Opportunities Workshops.
IEEE Transactionson Visualization and Computer Graphics , 25(1):748–758, 2019. doi: 10.1109/TVCG.2018.2865241[30] L. F. Klein and C. D’Ignazio.
Data feminism . MIT Press, 2020.[31] F. A. J. Korthagen, J. Kessels, B. Koster, B. Lagerwerf, and T. Wubbels.
Linking practice and theory: The pedagogy of realistic teacher education.
Lawrence Erlbaum Associates Publishers, 2001.[32] P.-M. Law, R. C. Basole, and Y. Wu. Duet: Helping data analysis novicesconduct pairwise comparisons by minimal specification.
IEEE transac-tions on visualization and computer graphics , 25(1):427–437, 2018. doi:10.1109/TVCG.2018.2864526[33] B. Lee, G. G. Robertson, M. Czerwinski, and C. S. Parr. CandidTree: Vi-sualizing Structural Uncertainty in Similar Hierarchies. In C. Baranauskas,P. Palanque, J. Abascal, and S. D. J. Barbosa, eds.,
Human-ComputerInteraction – INTERACT 2007 , Lecture Notes in Computer Science, pp.250–263, 2007. doi: 10.1007/978-3-540-74800-7 20[34] I. Letunic and P. Bork. Interactive Tree Of Life (iTOL) v4: Recent updatesand new developments.
Nucleic Acids Research , 47(W1):W256–W259,2019. doi: 10.1093/nar/gkz239[35] Y. S. Lincoln and E. G. Guba. Establishing Dependability and Confirma-bility in Naturalistic Inquiry Through an Audit. In
Annual Meeting of theAmerican Educational Research Association , 1982.[36] Z. Liu, S. H. Zhan, and T. Munzner. Aggregated Dendrograms for VisualComparison Between Many Phylogenetic Trees.
IEEE Transactions onVisualization and Computer Graphics , pp. 1–1, 2019. doi: 10.1109/TVCG.2019.2898186[37] D. Macbeth. On “Reflexivity” in Qualitative Research: Two Readings, anda Third:.
Qualitative Inquiry , 2016. doi: 10.1177/107780040100700103[38] S. Mantzoukas. The inclusion of bias in reflective and reflexive research:A necessary prerequisite for securing validity.
Journal of Research inNursing , 10(3):279–295, 2005. doi: 10.1177/174498710501000305[39] N. McCurdy, J. Dykes, and M. Meyer. Action Design Research andVisualization Design. In
Proceedings of the Beyond Time and Errors onNovel Evaluation Methods for Visualization - BELIV ’16 , pp. 10–18, 2016.doi: 10.1145/2993901.2993916[40] N. Mccurdy, J. Gerdes, and M. Meyer. A Framework for ExternalizingImplicit Error Using Visualization.
IEEE Transactions on Visualizationand Computer Graphics , 25(1):925–935, 2019. doi: 10.1109/TVCG.2018.2864913[41] N. McCurdy, J. Lein, K. Coles, and M. Meyer. Poemage: Visualizing thesonic topology of a poem.
IEEE transactions on visualization and com-puter graphics , 22(1):439–448, 2015. doi: 10.1109/TVCG.2015.2467811[42] S. McKenna, D. Mazur, J. Agutter, and M. Meyer. Design ActivityFramework for Visualization Design.
IEEE Transactions on Visualizationand Computer Graphics , 20(12):2191–2200, 2014. doi: 10.1109/TVCG.2014.2346331[43] M. Meyer and J. Dykes. Reflection on reflection in applied visualizationresearch.
IEEE Computer Graphics and Applications , 38(6):9–16, 2018.doi: 10.1109/MCG.2018.2874523[44] M. Meyer and J. Dykes. Criteria for Rigor in Visualization Design Study.
IEEE Transactions on Visualization and Computer Graphics , pp. 1–1,2019. doi: 10.1109/TVCG.2019.2934539[45] M. Meyer, A. Lex, J. Baumes, C. Lisle, and L. Harmon. Collaborativeresearch: Framework: Software: Hdr: Reproducible visual analysis of o appear in IEEE Transactions on Visualization and Computer Graphics. multivariate networks with MultiNet, 2019. https://vdl.sci.utah.edu/projects/2019-nsf-multinet/ .[46] M. Meyer, M. Sedlmair, P. S. Quinan, and T. Munzner. The nested blocksand guidelines model. Information Visualization , 14(3):234–249, 2015.doi: 10.1177/1473871613510429[47] B. Middleton, J. Platt, J. Richardson, and B. Blumenfeld. Recommen-dations for Building and Maintaining Trust in Clinical Decision SupportKnowledge Artifacts.
Research Triangle Park, NC: Patient-CenteredClinical Decision Support Learning Network , 2018.[48] A. Moravcsik. Transparency: The Revolution in Qualitative Research.
PS: Political Science & Politics , 47(1):48–53, 2014. doi: 10.1017/S1049096513001789[49] J. Morse. The changing face of qualitative inquiry.
International Journalof Qualitative Methods , 19:1–7, 2020. doi: 10.1177/1609406920909938[50] T. Munzner. A Nested Model for Visualization Design and Validation.
IEEE Transactions on Visualization and Computer Graphics , 15(6):921–928, 2009. doi: 10.1109/TVCG.2009.111[51] T. Munzner, F. Guimbreti`ere, S. Tasiran, L. Zhang, and Y. Zhou. TreeJux-taposer: Scalable Tree Comparison Using Focus+Context with GuaranteedVisibility. In
Proceedings of the ACM Conference on Computer Graphicsand Interactive Techniques (SIGGRAPH ’03) , pp. 453–462, 2003. doi: 10.1145/1201775.882291[52] C. Nobre, N. Gehlenborg, H. Coon, and A. Lex. Lineage: Visualizing Mul-tivariate Clinical Data in Genealogy Graphs.
Transaction on Visualizationand Computer Graphics , 25(3):1543–1558, 2019. doi: 10.1109/TVCG.2018.2811488[53] C. Nobre, M. Meyer, M. Streit, and A. Lex. The State of the Art inVisualizing Multivariate Networks.
Computer Graphics Forum (EuroVis) ,38(3):807–832, 2019. doi: 10.1111/cgf.13728[54] C. Nobre, M. Streit, and A. Lex. Juniper: A Tree+Table Approach to Mul-tivariate Graph Visualization.
Transaction on Visualization and ComputerGraphics (InfoVis ’18) , 25(1):544–554, 2019. doi: 10.1109/TVCG.2018.2865149[55] D. Offenhuber. Data by Proxy – Material Traces as Autographic Visual-izations. arXiv:1907.05454 [cs, math] , 2019.[56] C. Partl, S. Gratzl, M. Streit, A. M. Wassermann, H. Pfister, D. Schmalstieg,and A. Lex. Pathfinder: Visual Analysis of Paths in Graphs.
ComputerGraphics Forum (EuroVis ’16) , 35(3):71–80, 2016. doi: 10.1111/cgf.12883[57] L. J. Revell. Phytools: An R package for phylogenetic comparative biology(and other things).
Methods in Ecology and Evolution , 3(2):217–223, 2012.doi: 10.1111/j.2041-210X.2011.00169.x[58] L. J. Revell. Two new graphical methods for mapping trait evolution onphylogenies.
Methods in Ecology and Evolution , 4(8):754–759, 2013. doi:10.1111/2041-210X.12066[59] I. M. M. J. Reymen and D. K. Hammer. Structured Reflection for Improv-ing Design Processes. In
DS 30: Proceedings of DESIGN 2002, the 7thInternational Design Conference, Dubrovnik , pp. 887–892, 2002.[60] E. L. Rezende and J. A. F. Diniz-Filho. Phylogenetic Analyses: Com-paring Species to Infer Adaptations and Physiological Mechanisms. In
Comprehensive Physiology , pp. 639–674. Wiley Online Library, 2012. doi:10.1002/cphy.c100079[61] J. A. Rode. Reflexivity in digital anthropology. In
Proceedings of theSIGCHI Conference on Human Factors in Computing Systems , CHI ’11,pp. 123–132, 2011. doi: 10.1145/1978942.1978961[62] J. Rosindell and L. J. Harmon. OneZoom: A Fractal Explorer for the Treeof Life.
PLOS Biology , 10(10):e1001406, 2012. doi: 10.1371/journal.pbio.1001406[63] A. Satyanarayan, B. Lee, D. Ren, J. Heer, J. Stasko, J. Thompson,M. Brehmer, and Z. Liu. Critical reflections on visualization author-ing systems.
IEEE transactions on visualization and computer graphics ,26(1):461–471, 2019. doi: 10.1109/TVCG.2019.2934281[64] D. A. Schon.
The Reflective Practitioner: How Professionals Think InAction . Basic books, 1984.[65] M. Sedlmair. Design study contributions come in different guises: Sevenguiding scenarios. In
Proceedings of the Sixth Workshop on Beyond Timeand Errors on Novel Evaluation Methods for Visualization , pp. 152–161,2016. doi: 10.1145/2993901.2993913[66] M. Sedlmair, M. Meyer, and T. Munzner. Design Study Methodology:Reflections from the Trenches and the Stacks.
IEEE Transactions onVisualization and Computer Graphics (InfoVis) , 18(12):2431–2440, 2012.doi: 10.1109/TVCG.2012.213[67] M. K. Sein, O. Henfridsson, S. Purao, M. Rossi, and R. Lindgren. Action design research.
MIS quarterly , pp. 37–56, 2011. doi: 10.2307/23043488[68] A. K. Shenton. Strategies for ensuring trustworthiness in qualitativeresearch projects.
Education for information , 22(2):63–75, 2004. doi: 10.3233/EFI-2004-22201[69] S. Simon, S. Mittelst¨adt, D. A. Keim, and M. Sedlmair. Bridging the gapof domain and visualization experts with a liaison. In
Accepted at theEurographics Conference on Visualization (EuroVis 2015, Short Paper) ,vol. 2015. The Eurographics Association, 2015. doi: 10.2312/eurovisshort.20151137[70] B. Smith and K. R. McGannon. Developing rigor in qualitative research:Problems and opportunities within sport and exercise psychology.
Interna-tional review of sport and exercise psychology , 11(1):101–121, 2018. doi:10.1080/1750984X.2017.1317357[71] P. J. Stappers. Doing design as a part of doing research. In
Design researchnow , pp. 81–91. Springer, 2007. doi: 10.1007/978-3-7643-8472-2 6[72] U. H. Syeda, P. Murali, L. Roe, B. Berkey, and M. A. Borkin. DesignStudy ”Lite” Methodology: Expediting Design Studies and Enabling theSynergy of Visualization Pedagogy and Social Good. In
Proceedings ofthe 2020 CHI Conference on Human Factors in Computing Systems , CHI’20, pp. 1–13, 2020. doi: 10.1145/3313831.3376829[73] P. Talkad Sukumar, I. Avellino, C. Remy, M. A. DeVito, T. R. Dillahunt,J. McGrenere, and M. L. Wilson. Transparency in Qualitative Research:Increasing Fairness in the CHI Review Process. In
Extended Abstracts ofthe 2020 CHI Conference on Human Factors in Computing Systems , CHIEA ’20, pp. 1–6, 2020. doi: 10.1145/3334480.3381066[74] I. Tavory and S. Timmermans.
Abductive Analysis: Theorizing QualitativeResearch . University of Chicago Press, 2014.[75] S. Thompson and N. Thompson.
The critically reflective practitioner .Macmillan International Higher Education, 2018.[76] S. J. Tracy. Qualitative quality: Eight big-tent criteria for excellent qual-itative research.
Qualitative inquiry , 16(10):837–851, 2010. doi: 10.1177/1077800410383121[77] C. Wacharamanotham, L. Eisenring, S. Haroz, and F. Echtler. Trans-parency of CHI Research Artifacts: Results of a Self-Reported Survey. In
Proceedings of the 2020 CHI Conference on Human Factors in ComputingSystems , CHI ’20, pp. 1–14, 2020. doi: 10.1145/3313831.3376448[78] Y. L. Wong, K. Madhavan, and N. Elmqvist. Towards characterizingdomain experts as a user group. In , pp. 1–10. IEEE,2018. doi: 10.1109/BELIV.2018.8634026[79] J. Wood, A. Kachkaev, and J. Dykes. Design exposition with literatevisualization.
IEEE transactions on visualization and computer graphics ,25(1):759–768, 2018. doi: 10.1109/TVCG.2018.2864836[80] D. Yanow and H. Tsoukas. What is Reflection-In-Action? A Phenomeno-logical Account.
Journal of Management Studies , 46(8):1339–1364, 2009.doi: 10.1111/j.1467-6486.2009.00859.x[81] G. Yu. Using ggtree to Visualize Data on Tree-Like Structures.
CurrentProtocols in Bioinformatics , 69(1):e96, 2020. doi: 10.1002/cpbi.96[82] Y. Zhang, K. Chanana, and C. Dunne. Idmvis: Temporal event sequencevisualization for type 1 diabetes treatment decision support.
IEEE transac-tions on visualization and computer graphics , 25(1):512–522, 2018. doi:10.1109/TVCG.2018.2865076[83] J. Zimmerman and J. Forlizzi. The Role of Design Artifacts in DesignTheory Construction.
Artifact , 2(1):41–45, 2008. doi: info:doi/10.1080/17493460802276893[84] J. Zimmerman, J. Forlizzi, and S. Evenson. Research through design asa method for interaction design research in hci. In
Proceedings of theSIGCHI conference on Human factors in computing systems , pp. 493–502,2007. doi: 10.1145/1240624.1240704, pp. 493–502,2007. doi: 10.1145/1240624.1240704