A Psychology of Visualization or (External) Representation?
AA Psychology of Visualization or (External) Representation?
Amy Rae Fox * Department of Cognitive ScienceUniversity of California, San Diego A BSTRACT
What is a visualization? There is limited utility in trifling with def-initions, except insofar as one serves as a tool for communicatingand conceptualizing our subject matter; a statement of identity fora community. To establish Visualization Psychology as a viableinterdisciplinary research programme, we must first define the ob-ject(s) of our collective inquiry. I propose that while we might referto the study of “visualization” for the terms colloquial accessibilityand pragmatic alignment with other fields, we should consider forexploration a class of artifacts and corresponding processes more ex-pansive and profound: external representations. What follows is anargument for the study of external representation as the foundationfor a new interdisciplinary endeavor, and approach to mapping thecorresponding problem space. Index Terms: information visualization, external representation
NTRODUCTION
The language of representation is slippery and self-referencing. IfI show you a collection of artifacts containing marks on surfaces,you might label some as pictures or art, others as diagrams, maps,schematics, some graphs, charts or plots, and others also graphsbut you might use air quotes and call them “graph-theory graphs”.Some you’ll identify as writing, and others, like writing but notwriting—some peculiar or particular systems of notation. The labelsyou apply to each marking likely depend on your disciplinary back-ground, and are neither exhaustive, nor mutually exclusive. Whichof these, are visualizations? N V ISUALIZATION
Let us start with definitions put forth in popular Visualization texts.Stephen Few offers a functional definition, characterizing data visu-alization as “an umbrella term to cover all types of visual representa-tions that support the exploration, examination, and communicationof data. Whatever the representation, as long as its visual, andwhatever it represents, as long as its information, this constitutesdata visualization” [11, pg.12]. This is a delightfully inclusivespecification, according to which the words on this page wouldconstitute a visualization—but would scarcely be considered so bymost visualization practitioners. Why? Because visualizations aresomehow graphic in nature, more depictive than descriptive. FromWare [32, pg.2], “a graphical representation of data or concepts.”
Infact, one might find it easier to depict the set of markings one consid-ers visualizations than to describe them in words. Similarly, in theirtimeless text, Card, Mackinlay and Shneiderman define informationvisualization as, “The use of computer-supported, interactive, visualrepresentations of abstract data to amplify cognition” [6, pg.7]. Inboth cases, we see the (appropriate) characterization of informa-tion visualization as artifact and process. But we are left with an * e-mail: [email protected] under-specification of what constitutes such an artifact. Need the rep-resentations be interactive, computer-generated marks on surfaces ?Is the nature of the information constrained? What makes a markinggraphic? I draw on these examples not in critique of their notablecontributions, but rather to call attention to a gap in the foundation ofthe field, and subsequent opportunity for psychologists (and perhapsphilosophers) of visualization to make an impactful contribution.Definitions, as terminology, serve as tools for communicating andconceptualizing one’s subject matter [5]. I argue that to establishVisualization Psychology as a viable interdisciplinary research pro-gramme, we must first define the object(s) of our collective inquiry.To this I propose that while we might prefer the term visualization for its colloquial accessibility and pragmatic alignment with theSCI/VIS/VAST communities, there is an alternative characterizationof artifacts and processes more pertinent to the way “visualizations”are encountered in everyday life: as external representations. N E XTERNAL R EPRESENTATION
The term external representation stems from early cognitive scienceand information-processing psychology; inquiries into the existenceand nature of mental representations (see [17, 20, 22]). Palmer ar-gued that as cognitive representations are, “exceedingly complex anddifficult to study,” one might start with the examination of “noncog-nitive” (ie. external) representations, as they are “simple, and easyto study” [21, pg.262]. His subsequent elaboration of representa-tional systems demonstrates there is much to explore with respect tothe nature and function of such “noncognitive” structures, withoutreliance on the form of any internal counterparts. Zhang & Normandescribed external representations as knowledge and structure, “inthe world, as physical symbols (e.g., written symbols, beads of aba-cuses, etc.) or as external rules, constraints, or relations embeddedin physical configurations (e.g., spatial relations of written digits,visual and spatial layouts of diagrams, physical constraints in aba-cuses, etc.) ” [34, pg.3]. Zhang later writes that the information in external representations, “can be picked up, analyzed, and processedby perceptual systems alone, although the top-down participation ofconceptual knowledge from internal representations can sometimesfacilitate or inhibit the perceptual processes.” [33, pg.180]. What iswisely made explicit in these characterizations is the assertion thatexternal representations do not exist in isolation, rather, they workin concert with internal representations, whatever their form.In the proceeding decades, cognitive scientists took up the chal-lenge of discovering how various forms of external representationinfluence various forms of thinking; with particular attention tothe fashion in which representations support computation, suchas in problem solving [16], and scientific discovery [8]. Dis-tinctions were drawn between the sentential/propositional, andgraphic/diagrammatic, where the latter class was taken up by itsown interdisciplinary community in the early 2000s. Educationalpsychologists and learning scientists turned their attention to mul-timodal representations (where modality refers both to the sensorymodality (eg. visual, auditory) as well as the encoding media [19,24].By the late 2000s, sufficient interest across allied disciplines war-ranted a special issue of the journal TopiCS in Cognitive Science More ”accessible” being perhaps the more accurate characterization. a r X i v : . [ c s . H C ] S e p edicated to visual-spatial representations, with milestone contri-butions on visual analytics [12], graph comprehension [27], anddiagrams [7], with comprehensive reviews of visual-spatial repre-sentations as tools for thinking [31] and corresponding implicationsfor design [14]. It is worth noting that while the articles discussingvisual analytics, diagrams and statistical graphics might have founda home at a VIS conference (after the 2010 cognitive turn) the latterreview articles would be more out of place, but are indicative of workthat should be at the theoretical core of a Visualization Psychology. RAGMATIC P ROPOSAL
As psychologists, we are concerned not only with the tools, design,and efficacy of such representations, but with their mechanics: howthey function (or not). This function is enacted between the arti-fact(s) and person(s), embodied, and situated in their environmentsand complex social structures. What follows is a first proposalfor what minimal conception of external representation might betaken into the “hard core” of any Lakatosian research programme inVisualization Psychology.When we study a visualization, we are studying the func-tion of an external representation: the construction ofmeaning in a distributed cognitive system. The con-struction of meaning is oft followed by intelligent actionwith that meaning, be it by learning, making a decision,solving a problem, forming a judgement, or any of amultitude of complex cognitive activities which form the communicative context of the representation.Here we admit visualization as a subset of external representation,an active construction (rather than transmission) of meaning, andthat meaning serves some purpose in the context of interaction withthe representation. What is crucial is that we orient ourselves equallytoward the artifact and the procedure; representation as thing , andrepresentation as process . ROBLEM S PACE
We have moved from the study of computer-generated, interactive,graphics, to any externalization of thought. What we are left with,it seems, is a Goldilocks problem. The idiomatic conception ofvisualization is too narrow, and a faithful conception of externalrepresentation, too broad. Fortunately there are dimensions alongwhich this metaphorical problem space can be surveyed. We mightthink of these dimensions as ranges along which we can tune ourattention, progressively expanding or narrowing our scope of inquirydepending on the state of theoretical and technological advancement.
External representations can be constructed for any sensory modality,though by far the most attention has been paid to the visual. De-servedly so, as visuals are the most pervasive information artifacts,and the sensory modality about which we have the most understand-ing. Though we are surely far from exhausting the wellspring ofquestions to be asked of visual representations, and contributionsfor vision scientists to make, I suggest that we accept within ourscope multi-sensory representations. From a theoretical stance, thisrequires broader inclusion of expertise across perceptual psychology,though the applications are consequential. Accessibility demandsinformationally-equivalent representations for those without visualperception, in an increasingly visualization-driven world.
Though I’ve noted the lack of precision in defining the scope ofvisualizations, there has been no lack of effort in cataloging [13] andtaxonomizing them, from general descriptive frameworks [3,9,23,28–30], to those concerned with specific domains of data [1, 2, 4]. Twoparticularly useful (and under-appreciated) are those of Engelhardt [10] who offers an atomic, generative framework deserving of itscharacterization as a language of graphics, and Massironi [18] whooffers both a taxonomy and evolutionary timeline.While most taxonomies deal with some intersection of graphicalstructure and data type (eg. geographic-maps, relational-networks),the more common distinction in the cognitive and learning scienceliterature is the continuum from descriptive to depictive , roughlyanalogous with symbolic to analog, or propositional to graphic.These terms refer to a semiotic modality (also: medium), which indi-cates the degree of convention (how arbitrary) the relation betweena representation and thing to which it refers. While the poles of adepictive–descriptive continuum can be easily identified, there laysbetwixt a murky medium. At what point of abstraction does an iconbecome a symbol? When it is no longer identifiable as its referentwithout convention? In whose judgment? We are more accuratein describing our scope of inquiry as multimedia, than ‘primarilygraphic’. I propose that while origins of visualization as a field lie inthe distinction of graphics from text, fundamental questions aboutframing, persuasion, and even comprehension rely on understandingthe function of text alongside graphics. It is rarely the case thatexternal representations of the visual graphic variety are not accom-panied by some form of linguistic propositions or notation. Indeed,a visualization without a title and labels may be worth no words atall. Visualization texts describe the purpose of visualization as being to‘amplify’ cognition [6,11,32], though as psychologists we appreciatethe story is more nuanced [15]. What kind of cognition, to what end?The most generic case is that of communicating to simply inform :the boxplot in my manuscript or barchart in newspaper, where I aimto inform the reader of some aspects of the underlying information,in as clear a manner as possible. But I might design that artifactdifferently if I want you to explore the data, to undertake an analysis,make a decision, a plan, or a forecast. I’ll certainly change mystrategy if I want to strongly persuade you, or alternatively, wantyou to use the representation to learn. There are entire systems ofdiagrams designed for solving particular kinds of problems, and thedesign of representations to support conceptual change is the focusof entire subdiscplines in STEM education. I use the term commu-nicative context , to refer to the “cognitive activity” the designer ofa representation intends the user to perform. The structure of theseactivities has not been taxonomized, though a compelling frameworkfor the hierarchical, emergent structure of such activities is detailedby Sedig & Parsons [25, 26]. The relevant insight is that certainparameters of a representation, such as the computational efficiency,or relative explicitness of certain aspects of the data, will be tunedin accordance with the task the reader is expected to perform.
ONCLUSION
We began with the question, ‘What is a visualization?’ and end withthe contention it is a class too narrow to characterize the scope ofour endeavor. When we study visualizations in real-world activity,we are actually studying multimedia representations. Similarly, wemust work at the development and understanding of multimodalmultimedia representations in order to realize a world where accessto information is not the exclusive privilege of those with visualperception. While we might focus primarily on graphic visuals tothe extent that they are the most common tools for representing in-formation, these porous boundaries are both pragmatic and realistic.By adopting a broader scope, our nascent programme positions itselfat the boundaries of established visualization scholarship, alongsideresearchers exploring data visceralization, accessibility and multi-modal communication, multimedia learning and a wider communityof scholars who share interest in diverse communicative artifacts.
EFERENCES [1] W. Aigner, S. Miksch, W. Muller, H. Schumann, and C. Tominski. Visu-alizing time-oriented data. a systematic view.
Computers & Graphics ,31(3):401–409, June 2007. doi: 10.1016/j.cag.2007.01.030[2] F. Beck, M. Burch, S. Diehl, and D. Weiskopf. A Taxonomy andSurvey of Dynamic Graph Visualization.
Computer Graphics Forum ,00(00):1–27, 2016. doi: 10.1111/cgf.12791[3] A. Blackwell and Y. Engelhardt. A Meta-Taxonomy for DiagramResearch. In
Diagrammatic Representation and Inference , pp. 47–64,2002. doi: 10.1007/978-1-4471-0109-3 3[4] T. Blascheck, K. Kurzhals, M. Raschke, M. Burch, D. Weiskopf, andT. Ertl. Visualization of Eye Tracking Data: A Taxonomy and Survey.
Computer Graphics Forum , 36(8):260–284, 2017. doi: 10.1111/cgf.13079[5] M. Cabr´e Castellv´ı.
Terminology: Theory, Methods, and Applications ,vol. 1. John Benjamins Publishing, 1999.[6] S. Card, J. Mackinlay, and B. Shneiderman.
Readings in InformationVisualization. Using Vision to Think.
Morgan Kaufmann Publishers,San Francisco, CA, 1999.[7] P. C. Cheng. Probably good diagrams for learning: Representationalepistemic recodification of probability theory.
Topics in CognitiveScience , 3(3):475–498, 2011. doi: 10.1111/j.1756-8765.2009.01065.x[8] P. C. Cheng and H. A. Simon. Scientific discovery and creative reason-ing with diagrams. In S. M. Smith, T. T. Ward, and R. A. Finke, eds.,
The Creative Cognition Approach , pp. 205–228. The MIT Press, 1995.[9] E. Chi. A taxonomy of visualization techniques using the data statereference model. In
IEEE Symposium on Information Visualization2000. INFOVIS 2000. Proceedings , pp. 69–75, 2000. doi: 10.1109/INFVIS.2000.885092[10] Y. Engelhardt.
The Language of Graphics . PhD thesis, University ofAmsterdam, 2002.[11] S. Few.
Now You See It: Simple Visualization Techniques for Quantita-tive Analysis . Analytics Press, 2009.[12] B. Fisher, T. M. Green, and R. Arias-Hernndez. Visual Analyticsas a Translational Cognitive Science.
Topics in Cognitive Science ,3(3):609–625, 2011. doi: 10.1111/j.1756-8765.2011.01148.x[13] R. L. Harris.
Information Graphics: A Comprehensive IllustratedReference . Oxford University Press, 1999.[14] M. Hegarty. The Cognitive Science of Visual-Spatial Displays: Impli-cations for Design.
Topics in Cognitive Science , 3(3):446–474, 2011.doi: 10.1111/j.1756-8765.2011.01150.x[15] D. Kirsh. Thinking with external representations.
AI and Society ,25(4):441–454, 2010. doi: 10.1007/s00146-010-0272-8[16] J. Larkin and H. Simon. Why a diagram is (sometimes) worth tenthousand words.
Cognitive Science , 99:65–99, 1987.[17] P. H. Lindsay and D. Norman.
Human information processing: anintroduction to psychology . Academic Press, New York, 1972.[18] M. Massironi.
The Psychology of Graphic Images . Psychology Press,2001.[19] R. Mayer. Cognitive Theory of Multimedia Learning. In
The Cam-bridge Handbook of Multimedia Learning , pp. 31–48. 2005.[20] U. Neisser.
Cognitive Psychology . The Century psychology series.Appleton-Century-Crofts, New York, 1967.[21] S. E. Palmer. Fundamental Aspects of Cognitive Representation. InE. Rosch and B. B. Lloyd, eds.,
Cognition and Categorization , pp.259–302. Erlbaum, Hillsdale, NJ, 1978.[22] S. E. Palmer and R. Kimchi. The Information Processing Approach toCognition. In
Approaches to Cognition: Contrasts and Controversies ,p. 38. Lawrence Erlbaum Associates, 1986.[23] P. Parsons and K. Sedig. Common Visualizations: Their CognitiveUtility. In
Handbook of Human Centric Visualization , pp. 671–691.2014.[24] W. Schnotz. An Integrated Model of Text and Picture Comprehension.In
The Cambridge Handbook of Multimedia Learning , pp. 49–69. 2005.[25] K. Sedig, P. Parsons, and A. Babanski. Towards a Characterization ofInteractivity in Visual Analytics.
Journal of Multimedia Processingand Technologies, Special Issue on Theory and Application of VisualAnalytics , 3(1):12–28, 2012. doi: 10.1145/0000000.0000000[26] K. Sedig, P. Parsons, M. Dittmer, and R. Haworth. Human-Centered Interactivity of Visualization Tools: Micro and Macro-level Consid-erations. In
Handbook of Human Centric Visualization , pp. 671–691.2014.[27] P. Shah and E. G. Freedman. Bar and line graph comprehension: Aninteraction of top-down and bottom-up processes.
Topics in CognitiveScience , 3(3):560–578, 2011. doi: 10.1111/j.1756-8765.2009.01066.x[28] B. Shneiderman. The Eyes Have It: A Task By Data Type Taxonomyfor Information Visualization.
Proceedings 1996 IEEE Symposium onVisual Languages , pp. 336–343, 1996. doi: 10.1109/VL.1996.545307[29] J. K. H. Tan and I. Benbasat. Processing of Graphical Information: ADecomposition Taxonomy to Match Data Extraction Tasks and Graph-ical Representations.
Information Systems Research , 1(4):416–439,1990.[30] M. Tory and T. Moller. Rethinking Visualization: A High-Level Taxon-omy. In
IEEE Symposium on Information Visualization , pp. 151–158,Oct. 2004. doi: 10.1109/INFVIS.2004.59[31] B. Tversky. Visualizing Thought.
Topics in Cognitive Science , 3(3):499–535, 2011. doi: 10.1111/j.1756-8765.2010.01113.x[32] C. Ware.
Information Visualization: Perception for Design . ElsevierInc., San Francisco, CA, 2004.[33] J. Zhang. The Nature of External Representations in ProblemSolving.
Cognitive Science , 21(2):179–217, 1997. doi: 10.1207/s15516709cog2102 3[34] J. Zhang and D. Norman. Representations in Distributed CognitiveTasks.