Designing for Ambiguity: Visual Analytics in Avalanche Forecasting
©© 2020 IEEE. This is the author’s version of the article that has been published in the proceedings of IEEEVisualization conference. The final version of this record is available at: xx.xxxx/TVCG.201x.xxxxxxx/
Designing for Ambiguity: Visual Analytics in Avalanche Forecasting
Stan Nowak * Simon Fraser University
Lyn Bartram † Simon Fraser University
Pascal Haegeli ‡ Simon Fraser University A BSTRACT
Ambiguity, an information state where multiple interpretations areplausible, is a common challenge in visual analytics (VA) systems.We discuss lessons learned from a case study designing VA tools forCanadian avalanche forecasters. Avalanche forecasting is a complexand collaborative risk-based decision-making and analysis domain,demanding experience and knowledge-based interpretation of hu-man reported and uncertain data. Differences in reporting practices,organizational contexts, and the particularities of individual reportsresult in a variety of potential interpretations that have to be negoti-ated as part of the forecaster’s sensemaking processes. We describeour preliminary research using glyphs to support sensemaking underambiguity. Ambiguity is not unique to public avalanche forecasting.There are many other domains where the way data are measuredand reported vary in ways not accounted explicitly in the data andrequire analysts to negotiate multiple potential meanings. We arguethat ambiguity is under-served by visualization research and wouldbenefit from more explicit VA support.
Index Terms:
Human-centered computing—Visualization—Visu-alization application domains—Visual analytics;
NTRODUCTION
Uncertainty is an important issue in visualization, but to date, vi-sualization research has primarily focused on uncertainties that areexplicit in data and how to represent them [5,11,13,16–18,35]. How-ever, uncertainty concerns much more than just data, and researchershave highlighted the need to explicitly consider uncertainties that re-sult from reasoning processes in analysis [22, 37]. One such criticaltype of uncertainty that arises in complex and collaborative analysesis ambiguity , which we define as an informational state in whichmultiple interpretations may be equally plausible. It pertains to amultiplicity of potential ways to interpret, rather than simply a lackof or erroneous data. While uncertainty in data itself may lead toambiguity, it is only one aspect in many that are relevant to consider.While VA researchers have recently focused on more explicit sup-port for knowledge-based and interpretative processes in VA [1,7,8],ambiguity, to our best knowledge, has not been considered.We partnered with Avalanche Canada (AC), a public avalancheforecasting organization, to develop VA tools for assessing avalanchehazard. Our collaboration revealed that ambiguity is an importantchallenge in avalanche forecasting and meaningfully accounting forit is critical to the design of helpful VA. This paper presents theresults and lessons learned from our case study. We discuss howchallenges in data and analysis give rise to ambiguity, our designstrategies to facilitate sensemaking in the face of ambiguity, and feed-back from forecasters who have used our tools. The contributionsof this paper include an applied discussion of the role and natureof ambiguity in complex analysis and decision-making applicationssuch as avalanche forecasting and a preliminary design trajectory for * e-mail: [email protected] † e-mail: [email protected] ‡ e-mail: pascal [email protected]. improved VA support for sensemaking under ambiguity. A key find-ing from our initial research is that ambiguity cannot—and shouldnot—be ”designed away”, but rather acknowledged and sometimesencouraged for richer sensemaking. VALANCHE F ORECASTING
Avalanches are natural disaster phenomena where an instability inthe snowpack stratigraphy releases a mass of snow that slides down-hill with destructive force. They pose a significant risk to thoseworking and recreating in mountainous terrain. Avalanche forecast-ing is concerned with predicting current and future snow instabilitiesthat may result in avalanches through human or natural triggers [23].Forecasters produce daily avalanche hazard assessments that com-municate avalanche hazard conditions and risks to those travelingthrough avalanche-prone mountainous terrain. As with the fore-casting of floods, wildfires, hurricanes, and other similar extremeweather events, avalanche forecasting involves risk prediction andcommunication where the audience varies in role and expertise, fromthe general public to those experienced with extreme natural hazards.Avalanche forecasting is viewed as a largely inductive orBayesian-like process where new information is used to continu-ously update mental models of avalanche conditions formed over thecourse of an entire season [24]. Forecasters assess and characterizeavalanche hazards by answering a series of questions: (1) what typesof avalanches exist? (2)
Where are they located? (3)
How likely areavalanches to occur? (4)
How large will they be? [33]. To answerthese questions, Forecasters utilize a variety of observations and datasources as part of the avalanche hazard assessment process [24].While many forecasters have the benefit of physically workingin the field within small-scale regions producing assessments foran expert audience, AC public avalanche forecasters assess hazardsand communicate them to the public from an office-based setting,relying heavily on second-hand reports, and applying them to largeregions that often experience significant variability in avalancheconditions. Such reports are provided by the Canadian AvalancheAssociation’s Industry Information Exchange (InfoEx) [14]. It isa web-based platform where professional avalanche safety opera-tions such as ski resorts, helicopter skiing operations, or operationsdealing with transportation corridors such as railways or highwaysshare local observations and assessments. Reported data includefield observations of weather, snow conditions, avalanche activity,descriptions of locations traveled to and hazard assessments formal-ized by industry standards. Data are viewed in large text tables withminimal visualization support.The complexity of avalanche phenomena and the variability ofoperational contexts from which these data are sourced leave fore-casters to rely heavily on subjective judgement and knowledge todiscern context and fill gaps in understanding. This process leavesroom for multiple potential interpretations and ambiguities. Thechallenges of interpretation under complexity and uncertainty aresimilar to those faced in the forecasting of many other types of natu-ral disasters [3] as well as risk management applications that utilizehuman-produced data [9].
EFINING A MBIGUITY
We distinguish ambiguity as a separate issue from that of data uncer-tainty. Ambiguity has been described as dealing with the reliability,1 a r X i v : . [ c s . H C ] S e p credibility, or adequacy of risk information [10], as well as a mul-tiplicity of states [32] or outcomes [2] among many others. Wedefine it as a multiplicity of plausible interpretations. This definitionis more closely related to the philosophical or semiotic notions ofambiguity [30].Ambiguity emerges out of sensemaking under complexity: com-prehensive understanding of a complex system is intractable givenlimitations of human perception and observation [12]. These lim-itations mean that prediction involves more subjective judgement,speculation, and imagination than it does precise deductions (this isoften referred to as mental simulation [19]). Sensemaking involvesdrawing on prior knowledge to negotiate alternative interpretationsof the problem at hand [20]. In this way, sensemaking is moreabout resolving multiple meanings, rather than simply accountingfor missing information [20]. Recent research has highlighted howdata scientists intervene and transform data based on their intuitiveand knowledge-based understanding to better support more mean-ingful sensemaking processes [26]. Choices made in the process ofanalysis, however defensible, can lead to widely different analyticresults, even when following high standards of scientific rigor [31].Ambiguity is ubiquitous in complex analysis because sensemakinginvolves much more than data itself.When sensemaking is shared, ambiguity arises not only from thevarying perspectives of the collaborators, but also the complexitiesof communication [15]. In large organizations or settings whereanalysis is shared across a variety of operational contexts, ambi-guity is common because as soon as analysis is shared it loses itscontext [28]. The relevance of any piece of information is context-sensitive as information itself is defined by the relations betweendata, the world it represents, and observers’ goals, expectations,and interests [36]. Further, sensemaking processes involve priorknowledge that may not be explicitly visible in shared data or in-formation [27], leaving collaborators to make inferences based ontheir own knowledge and experiences. Hence, domains that relyheavily on analyzing data and assessments produced by others suchas public health [25], intelligence analysis [28], search and rescueoperations, crisis management, avalanche forecasting, and manyothers are likely to experience challenges of ambiguity. ASE S TUDY
In collaboration with the AC development team we designed a set ofVA tools to support daily operational avalanche hazard assessmentand address it’s accompanying challenges of sensemaking underambiguity. The forecasters prioritized two types of observations thatare essential in avalanche hazard assessment: weather observationsfrom weather stations which provide a source of ground truth formeteorological forecast validation, and structured field reports ofavalanche observations which are considered to be key indicatorscharacterizing the nature of avalanche hazards [24].
To meaningfully inform our design process, we initially conducteda series of studies using a suite of qualitative interview and obser-vational methods improving our understanding of the applicationdomain. Through consultations with the avalanche forecasters, weiteratively developed a set of prototypes. Think-aloud studies andunstructured interviews were used to solicit feedback for improvingthe designs. We used historical and synthetic data to evaluate theavalanche observation tool, while the weather tool used real-timedata and was used operationally during the 2019/2020 winter season.At the end of the winter season, we conducted semi-structured inter-views to allow forecasters to reflect on how the prototypes addressedthe challenges of avalanche forecasting and how they changed thenature of work. Eight avalanche forecasters participated in the inter-views. Seven were actively involved throughout the design process.One forecaster simply provided feedback based on their experience using the weather tool. Forecasters were interviewed using a videoconferencing tool. These interviews were recorded, transcribed, andqualitatively coded for common themes that emerged.
Public avalanche forecasters use telemetry readings from remoteweather stations as a form of ground truth to validate previous fore-casts and to track weather systems throughout the workday.
The spatial distribution of these weather stations is sparse relativeto the spatial heterogeneity of weather patterns that these weatherstations are expected to measure, challenging the reliability of suchmeteorological observations [21]. To account for such variability,forecasters focus on familiar weather stations and use their experi-ences and knowledge of how local terrain at weather station sitesinteracts with weather systems to inform interpretation of such data.
Traditionally, forecasters have had to view station data individuallythrough a variety of web portals. Our visualizations provide ag-gregate descriptive statistics summarizing recent spatio-temporalpatterns in precipitation, wind, and temperature. They also showindividual station data through tooltips.
Forecasters found it challenging to adjust their traditionally bottom-up approach to fit with the top-down overview our tool provided,yet nevertheless found the tool to enrich their sensemaking process.While our design approach was not especially novel, forecastersfeedback revealed how forecasters used the tool to support theirinductive and Bayesian-like reasoning processes. They noted thatnumerical aggregations do provide a useful overview and startingpoint, but that the impressions that such an overview leaves haveto be refined using an iterative process that involves imagining howweather station telemetry at individual stations translate from itslocal geographic context into broader regional patterns. This processis as much speculative, involving the formation and evaluation ofvarious plausible expectations, as it is observational, grounding theseexpectations in data that captures traces of the material world. Thisinformed our subsequent designs.
The visualization tool shown in Figure 1 was developed to sup-port the investigation of field reported data characterizing observedavalanches.
Observations of avalanches are the most highly valued informationforecasters have at their disposal because they are direct evidenceof the existence of an avalanche problem. In the InfoEx, Avalancheobservations are described using unstructured text fields and stan-dardized structured fields following the Observation Guidelines andRecording Standards (OGRS) set by the Canadian Avalanche Associ-ation [6]. They are used to answer (in part) the essential questions inavalanche hazard assessment. These data explicitly include the types of avalanches observed, wherein the terrain they were observed (e.g.geo-position, elevation and, ”aspect”: the compass direction thata mountain slope faces), and what the respective sizes of observedavalanche were. However, to understand how likely avalanches arerequires a more nuanced reading.
Likelihood is determined througha combination of sensitivity to triggering and spatial distribution which are inferred from the integration of a wide variety of dataattributes such as trigger types, the number of avalanches triggered,geographic position, descriptions of terrain as well as other data andknowledge not included in our visualization tool. This is a deeply2
Figure 1: Multiple coordinated views featuring unit visualizations and glyphs. The spatial data used in this visualization are synthesized in part toprotect the identity of the avalanche safety operations whose data we acquired to develop this tool. Three days-worth of avalanche observationreports are shown. A) Map displaying glyphs positioned at centroids of polygons related to particular avalanche safety operations. B) Chartshowing elevation ranges (y-axis) associated with avalanche reports (x-axis). C) Arc diagram showing ranges of cardinal directions (polar axis)associated with avalanche reports (radial axis). D) Grid matrix showing avalanche observations across types of avalanche problems (rows) as wellas avalanche trigger types (columns). E) Timeline showing avalanche observations by day. interpretative process that leaves forecasters with a holistic under-standing of avalanche conditions which is continuously refined withnew information.In spite of standardization, even structured data may be interpretedin a variety of ways due to differences in operational needs, goals,and constraints of the reporting organizations. ...the InfoEx system and the standards they kind of define the boxthat we all work in. But what bits and pieces reside inside that boxand which ones you use and how you use them... context drives thatyou might... use a certain approach data that are obviously withinthat... general framework or box that we’ve created, but you mightnot use them exactly the same way. P8 One example of this is an attribute describing the number ofavalanches that were observed. This single attribute can be reportedusing numerical data or ordinal bins that describe specific ranges forthe number of observed avalanches (e.g. ”several” is defined as 2-9avalanches) [6]. Transforming such data into a single and unifieddata type to enable numerical aggregations and more parsimoniousvisualizations may appear like a desirable choice given the technicaldefinition. Doing so, however, would hide critical contextual infor-mation that forecasters use in their sensemaking process. The choiceof whether to use a number or an ordinal bin can indicate muchmore than the sum of observed avalanches when considering theoperational context in which the report was produced. For instance,the use of an ordinal bin can be interpreted as conveying the rate atwhich avalanches are occurring relative to the amount of terrain thathas been observed, as expressing uncertainty around the number ofavalanches truly observed, or simply reflect a time-saving groupingof avalanche observations that are characteristically different andwould otherwise be reported separately.This is but one example illustrating the challenges of ambiguityembodied in these data. There are many other fields where forecast-ers similarly rely on contextual information and the particularitiesof reports to support meaningful interpretation. As a result, our vi-sualizations needed to improve on sensemaking support of existingtools (text tables) without occluding contextual information. Oursuccess with the weather visualization suggested that an overviewwould be beneficial, but we could not rely on numerical aggregations.This is not least because the ambiguity of avalanche observationdata involves a re-conceptualization of the dimensions or measure- ments in question, whereas weather station telemetry remain largelyconceptually self-consistent.
To address the ambiguity related challenges of this data we tooka glyph-based approach. Glyphs can operate at multiple scalesof resolution and can provide an overview without having to relyon numerical aggregations. Instead, they allow visual aggregationoperations such as summarizing data, detecting outliers, detectingtrends, or segmenting data into clusters [34], while at the same timeshowcasing granular data to reveal its particularities.We used a packed bubble chart glyph (shown in Figure 1A, 1D,1E). Each circle represents an individual report. The size of re-ported avalanches is encoded as the size of each circle. Additionally,colour hue is used to distinguish whether a numerical (blue) or or-dinal (green) data type was used to report the number of observedavalanches. Colour saturation/luminance is used to encode value inthe number of avalanches observed with darker colours encodinghigher values. Circles are organized using a packed layout. The re-sulting glyph supports various forms of visual aggregation providinga holistic multidimensional overview of the data. For instance, whencomparing any two glyphs, the overall size of the glyph, the numberof circles, and their sizes and colours, all combine to provide a nu-anced and multidimensional view that allows forecasters to evaluatemultiple perspectives of the data.The glyphs are positioned within a timeline presenting the dayavalanches were estimated to have occurred (Figure 1E) and a ma-trix presenting information about the types of avalanches observedand what triggered them (figure 1D). Glyphs are also positioned atcentroids of polygons representing the tenures of the reporting oper-ation (figure 1A). Polygons are shaded according to the estimatedpercentage of the tenure that was observed.Elevations and aspects of avalanches are reported using non-uniform intervals. To maintain visibility of each individual reportand prevent overplotting, we developed several charts that encodereports as line or arc segments with their lengths representing thesenon-uniform intervals. Figure 1B shows elevation ranges of reportedavalanches with elevation on the y-axis and an index for each in-dividual report on the x-axis. Similarly, Figure 1C shows aspect3 ranges of reported avalanches with the polar axis encoding aspectand the radial axis serving as an index for each individual report.Reports are presented across multiple coordinated displays thatsupport standard brushing and highlighting interactions. Selectingreports highlights them in all corresponding visualizations providinga multidimensional perspective into the data. Additionally, tooltipsprovide access to unstructured data associated with individual re-ports.It is important to note that the visual overview provided by suchglyphs deliberately uses perceptually weak visual encodings to pro-vide forecasters with a holistic and multidimensional perspective asa starting point for their assessments. Our design is aimed to facili-tate the Bayesian-like reasoning processes of avalanche forecasting.As forecasters explore the data, access contextual information, andnegotiate meaning in these ambiguous data, they update their mentalmodel of avalanche conditions without being hampered by overlyprecise visualizations that could prescribe a certain perspective.Forecasters use this interface to answer the essential questionsin avalanche hazard assessment pertaining to avalanche type, like-lihood, size, and location in terrain. While this interface supportsseveral low-level visual tasks such as comparison and trend detec-tion to help answer these questions, its primary purpose is to allowforecasters to view and access the particularities of individual re-ports and mentally tune their understanding of any identified visualpatterns. This functionality is non-trivial. Being able to discern whatis meant by a particular datum such as the choice to use an ordinalbin for the number of avalanches observed can mean the differencebetween being able to or not being able to detect central characteris-tics of avalanche hazards. In this example, the operational contextin which the number of avalanches was reported in can influencehow to interpret the spatial density of avalanche occurrences in turninfluencing the forecaster’s perception of likelihood and the entirenature of the avalanche hazard.We found that this approach is well suited to the task of negoti-ating the potential meanings of such ambiguous data and analyses.This may not be the only or even the most optimal solution, however,it serves to highlight how sensemaking around ambiguous humanreported data can be supported through careful and ambiguity-awaredesign choices.
Forecasters informed us that our prototype was a good representationof the mental operations they perform using conventional tools. Theyreflected that our approach is more methodologically sound thanpurely quantitative approaches.
I like seeing the individual events more than that aggregates itseems full of flaws and limitations to kind of summarize all theactivity with one number. P7 Our prototype was used as a point of reference in internal discus-sions that surfaced discrepancies in how forecasters interpret certaindata, illustrating the potential for visualizations such as these toenrich organizational knowledge and practices.
ISCUSSION
Field reported data used by AC and the Canadian avalanche industryembody the central challenges of ambiguity in distributed collabora-tive analysis and decision-making. Standards can provide a commonlanguage, but complexity and differences in context create chal-lenges in interpretation that cannot simply be designed away. Thesedata may be reported along seemingly uniform and coherent dimen-sions, but in context of analysis, they have to be re-conceptualized toconsider multiple alternative interpretative lenses at the same time.Our glyph-based visualizations were designed to capture this mul-tidimensional correspondence with meaning, while simultaneouslydisplaying the particularities of each report to allow forecasters to glean enough context to decide what is relevant and how to interpretthe visual aggregates that the glyphs provide.A key take-away from this case study is that identifying andcharacterizing sources of ambiguity is critical for designing visu-alizations that are intended to meaningfully support sensemaking.Another is that while there is a seemingly widespread belief thatambiguity is something to always be prevented or reduced, it oftenserves a critically functional role in sensemaking and is thereforebetter embraced and specifically designed for . We recognize thatambiguity is much broader and arises from many more sources thanjust human reported data. Our design approach was more aboutenabling ambiguities to be recognized and reasoned through, ratherthan explicitly encoded. Our intent is to provide a starting point forfuture research in this area and encourage researchers to explore chal-lenges of ambiguity in similar complex and shared decision-makingapplications.A key challenge that remains is how to address ambiguity moreexplicitly. When analysis is shared and the same data are revisitedas part of subsequent analyses, any identified ambiguities, relevantmaterials, or knowledge are often not explictly captured in the data.Researchers studying how data workers cope with uncertainty foundthat when faced with ambiguity, common coping approaches includeannotations and references to clarifying materials [4]. Others haveused annotations with templated structured fields to aid public healthexperts in externalizing their knowledge of measurement errors im-plicit in international infectious disease reporting [25]. Supportingawareness of uncertainty is critical in analysis and visualization [29].We argue this is also the case for ambiguity and that conventionalvisualization techniques such as annotations can be extended andspecifically tailored to address the challenges of collaborative sense-making under ambiguity. However, we recognize that externalizingknowledge requires effort and can be disruptive [15]. This is es-pecially the case in constrained and risk-based decision-makingapplications such as avalanche forecasting. Hence, tailoring existingvisualization techniques to provide low-cost mechanisms to addressambiguity introduces a rich research and design space to explore.
ONCLUSION
Through a case study with public avalanche forecasters, we dis-covered that ambiguity, a state in which multiple interpretationsare plausible, bears direct relevance to the design of VA systems.In this domain, complexity, data sparsity, and the challenges ofcommunication in distributed collaborations produce ambiguity andleave forecasters to use their knowledge and experience to negotiateinterpretation. Our visualization tools addressed ambiguity usinga glyph-based approach that enables forecasters to navigate mul-tiple potential interpretation for meaningful sensemaking. Whilethis study highlights the challenges of ambiguity, we argue that re-search into a more targeted and explicit approach for capturing andrepresenting ambiguities is warranted. A CKNOWLEDGMENTS
Thanks to Avalanche Canada, the Vancouver Institute for VisualAnalytics (VIVA), the Big Data Initiative at Simon Fraser University(SFU), the SFU Avalanche Research Program, and our reviewers fortheir thoughtful feedback. This work was supported in part by theNatural Sciences and Engineering Research Council Industry Re-search Chair in Avalanche Risk Management (grant no. IRC/515532-2016), which receives industry support from Canadian Pacific Rail-way, HeliCat Canada, Canadian Avalanche Association, and MikeWiegele Helicopter Skiing. R EFERENCES [1] N. Andrienko, T. Lammarsch, G. Andrienko, G. Fuchs, D. Keim,S. Miksch, and A. Rind. Viewing Visual Analytics as Model Building.
Computer Graphics Forum , 37(6):275–299, 2018. [2] B. M. Ayyub. Elicitation of expert opinions for uncertainty and risks .CRC Press, Boca Raton, Fla, 2001.[3] K. J. Beven, S. Almeida, W. P. Aspinall, P. D. Bates, S. Blazkova,E. Borgomeo, K. Goda, J. W. Hall, J. C. Phillips, M. Simpson, P. J.Smith, D. B. Stephenson, T. Wagener, M. Watson, and K. L. Wilkins.Epistemic uncertainties and natural hazard risk assessment. 1. A reviewof different natural hazard areas.
Natural Hazards and Earth SystemSciences , 2018.[4] N. Boukhelifa, M.-E. Perrin, S. Huron, and J. Eagan. How DataWorkers Cope with Uncertainty: A Task Characterisation Study. In
Proceedings of the 2017 CHI Conference on Human Factors in Com-puting Systems , CHI ’17, pages 3645–3656, New York, NY, USA,2017. ACM.[5] K. Brodlie, R. AllendesOsorio, and A. Lopes. A Review of Uncertaintyin Data Visualization. In J. Dill, R. Earnshaw, D. Kasik, J. Vince, andP. C. Wong, editors,
Expanding the Frontiers of Visual Analytics andVisualization , pages 81–109. Springer London, London, 2012.[6] Canadian Avalanche Association.
Observation guidelines and record-ing standards for weather, snowpack and avalanches . CanadianAvalanche Assoc., 1995.[7] I. K. Choi, T. Childers, N. K. Raveendranath, S. Mishra, K. Harris,and K. Reda. Concept-Driven Visual Analytics: An Exploratory Studyof Model- and Hypothesis-Based Reasoning with Visualizations. In
Proceedings of the 2019 CHI Conference on Human Factors in Com-puting Systems , CHI ’19, pages 68:1–68:14, New York, NY, USA,2019. ACM.[8] I. K. Choi, N. K. Raveendranath, J. Westerfield, and K. Reda. Vi-sual (dis)Confirmation: Validating Models and Hypotheses with Vi-sualizations. In , pages 116–121, July 2019.[9] A. Diehl. User-Uncertainty: A Human-Centred Uncertainty Taxonomyfor VGI through the Visual Analytics Workflow. 2018.[10] D. Ellsberg. Risk, Ambiguity, and the Savage Axioms.
The QuarterlyJournal of Economics , 75(4):643–669, 1961.[11] M. Fernandes, L. Walls, S. Munson, J. Hullman, and M. Kay. Un-certainty Displays Using Quantile Dotplots or CDFs Improve TransitDecision-Making. In
Proceedings of the 2018 CHI Conference onHuman Factors in Computing Systems , CHI ’18, pages 144:1–144:12,New York, NY, USA, 2018. ACM.[12] F. Grabowski and D. Strzalka. Simple, complicated and complexsystems the brief introduction. In , pages 570–573, May 2008.[13] M. Greis, A. Joshi, K. Singer, A. Schmidt, and T. Machulla. Uncer-tainty Visualization Influences how Humans Aggregate DiscrepantInformation. In
Proceedings of the 2018 CHI Conference on HumanFactors in Computing Systems - CHI ’18 , pages 1–12, Montreal QC,Canada, 2018. ACM Press.[14] P. Haegeli, J. Obad, B. Harrison, B. Murray, J. Engblom, and J. Neufeld.InfoEx 3.0Advancing The Data Analysis Capabilities Of Canada’sDiverse Avalanche Community. In
Proceedings of the InternationalSnow Science Workshop , page 8, 2014.[15] J. Heer and M. Agrawala. Design Considerations for CollaborativeVisual Analytics.
Information Visualization , page 8, 2008.[16] J. Hullman. Why Evaluating Uncertainty Visualization is Error Prone.In
Proceedings of the Beyond Time and Errors on Novel EvaluationMethods for Visualization - BELIV ’16 , pages 143–151, Baltimore,MD, USA, 2016. ACM Press.[17] J. Hullman, X. Qiao, M. Correll, A. Kale, and M. Kay. In Pursuitof Error: A Survey of Uncertainty Visualization Evaluation.
IEEETransactions on Visualization and Computer Graphics , 25(1):903–913,Jan. 2019.[18] A. Kale, F. Nguyen, M. Kay, and J. Hullman. Hypothetical Out-come Plots Help Untrained Observers Judge Trends in AmbiguousData.
IEEE Transactions on Visualization and Computer Graphics ,25(1):892–902, Jan. 2019.[19] G. Klein. The Role of Mental Simulation in Problem Solving andDecision Making. In
Local Applications of the Ecological Approach toHuman-Machine Systems , pages 324–358. CRC Press, 1 edition, May2018.[20] G. Klein, J. Phillips, E. Rall, and D. Peluso. A data-frame theory of sensemaking.
Expertise out of Context: Proceedings of the SixthInternational Conference on Naturalistic Decision Making , pages 113–155, Jan. 2007.[21] J. Lundquist, M. Hughes, E. Gutmann, and S. Kapnick. Our Skill inModeling Mountain Rain and Snow is Bypassing the Skill of Our Ob-servational Networks.
Bulletin of the American Meteorological Society ,100(12):2473–2490, Dec. 2019. Publisher: American MeteorologicalSociety.[22] A. M. MacEachren. Visual Analytics and Uncertainty: Its Not Aboutthe Data. In
EuroVA@EuroVis . The Eurographics Association, 2015.[23] D. M. McClung. The Elements of Applied Avalanche Forecasting, PartI: The Human Issues.
Natural Hazards , 26(2):111–129, 2002.[24] D. M. McClung. The Elements of Applied Avalanche Forecasting, PartII: The Physical Issues and the Rules of Applied Avalanche Forecasting.
Natural Hazards , 26(2):131–146, 2002.[25] N. Mccurdy, J. Gerdes, and M. Meyer. A Framework for ExternalizingImplicit Error Using Visualization.
IEEE Transactions on Visualizationand Computer Graphics , 25(1):925–935, Jan. 2019.[26] M. Muller, I. Lange, D. Wang, D. Piorkowski, J. Tsay, Q. V. Liao,C. Dugan, and T. Erickson. How Data Science Workers Work withData: Discovery, Capture, Curation, Design, Creation. In
Proceedingsof the 2019 CHI Conference on Human Factors in Computing Systems ,CHI ’19, pages 1–15, Glasgow, Scotland Uk, May 2019. Associationfor Computing Machinery.[27] P. Pirolli and S. Card. The Sensemaking Process and Leverage Pointsfor Analyst Technology as Identified Through Cognitive Task Analysis.In
Proceedings of International Conference on Intelligence Analysis ,volume 5, pages 2–4, May 2005.[28] B. Prue, M. Jenkins, L. D. Stern, and J. Pfautz. Overcoming LimitedCollaboration Channels in Distributed Intelligence Analysis: Visual-ization Tools and Design Seeds. In , pages 17–24, Sept. 2014.[29] D. Sacha, H. Senaratne, B. C. Kwon, G. Ellis, and D. A. Keim. TheRole of Uncertainty, Awareness, and Trust in Visual Analytics.
IEEETransactions on Visualization and Computer Graphics , 22(1):240–249,Jan. 2016.[30] A. Sennet. Ambiguity. In E. N. Zalta, editor,
The Stanford Encyclopediaof Philosophy . Metaphysics Research Lab, Stanford University, spring2016 edition, 2016.[31] R. Silberzahn et al. Many Analysts, One Data Set: Making TransparentHow Variations in Analytic Choices Affect Results:.
Advances inMethods and Practices in Psychological Science , Aug. 2018.[32] M. Smithson. Ignorance and Science: Dilemmas, Perspectives, andProspects.
Knowledge , 15(2):133–156, Dec. 1993. Publisher: SAGEPublications.[33] G. Statham, P. Haegeli, E. Greene, K. Birkeland, C. Israelson, B. Trem-per, C. Stethem, B. McMahon, B. White, and J. Kelly. A conceptualmodel of avalanche hazard.
Natural Hazards , 90(2):663–691, Jan.2018.[34] D. A. Szafir, S. Haroz, M. Gleicher, and S. Franconeri. Four types ofensemble coding in data visualizations.
Journal of Vision , 16(5):11–11,Mar. 2016. Publisher: The Association for Research in Vision andOphthalmology.[35] J. Thomson, E. Hetzler, A. MacEachren, M. Gahegan, and M. Pavel.A typology for visualizing uncertainty. In
Visualization and DataAnalysis 2005 , volume 5669, pages 146–158. International Society forOptics and Photonics, Mar. 2005.[36] D. D. Woods. Diagnosis of Data Overload. In
Escape from DataOverload . Institute for Ergonomics. Ohio State University, 2002.[37] T. Zuk and S. Carpendale. Visualization of Uncertainty and Reasoning.In A. Butz, B. Fisher, A. Krger, P. Olivier, and S. Owada, editors,
Smart Graphics , Lecture Notes in Computer Science, pages 164–177.Springer Berlin Heidelberg, 2007., Lecture Notes in Computer Science, pages 164–177.Springer Berlin Heidelberg, 2007.