Torsten Möller

A Systematic Review on the Practice of Evaluating Visualization

We present an assessment of the state and historic development of evaluation practices as reported in papers published at the IEEE Visualization conference. Our goal is to reflect on a meta-level about evaluation in our community through a systematic understanding of the characteristics and goals of presented evaluations. For this purpose we conducted a systematic review of ten years of evaluations in the published papers using and extending a coding scheme previously established by Lam et al. [2012]. The results of our review include an overview of the most common evaluation goals in the community, how they evolved over time, and how they contrast or align to those of the IEEE Information Visualization conference. In particular, we found that evaluations specific to assessing resulting images and algorithm performance are the most prevalent (with consistently 80-90% of all papers since 1997). However, especially over the last six years there is a steady increase in evaluation methods that include participants, either by evaluating their performances and subjective feedback or by evaluating their work practices and their improved analysis and reasoning capabilities using visual tools. Up to 2010, this trend in the IEEE Visualization conference was much more pronounced than in the IEEE Information Visualization conference which only showed an increasing percentage of evaluation through user performance and experience testing. Since 2011, however, also papers in IEEE Information Visualization show such an increase of evaluations of work practices and analysis as well as reasoning using visual tools. Further, we found that generally the studies reporting requirements analyses and domain-specific work practices are too informally reported which hinders cross-comparison and lowers external validity.

Visual Parameter Space Analysis: A Conceptual Framework

Various case studies in different application domains have shown the great potential of visual parameter space analysis to support validating and using simulation models. In order to guide and systematize research endeavors in this area, we provide a conceptual framework for visual parameter space analysis problems. The framework is based on our own experience and a structured analysis of the visualization literature. It contains three major components: (1) a data flow model that helps to abstractly describe visual parameter space analysis problems independent of their application domain; (2) a set of four navigation strategies of how parameter space analysis can be supported by visualization tools; and (3) a characterization of six analysis tasks. Based on our framework, we analyze and classify the current body of literature, and identify three open research gaps in visual parameter space analysis. The framework and its discussion are meant to support visualization designers and researchers in characterizing parameter space analysis problems and to guide their design and evaluation processes.

Visualization as Seen through its Research Paper Keywords

We present the results of a comprehensive multi-pass analysis of visualization paper keywords supplied by authors for their papers published in the IEEE Visualization conference series (now called IEEE VIS) between 1990-2015. From this analysis we derived a set of visualization topics that we discuss in the context of the current taxonomy that is used to categorize papers and assign reviewers in the IEEE VIS reviewing process. We point out missing and overemphasized topics in the current taxonomy and start a discussion on the importance of establishing common visualization terminology. Our analysis of research topics in visualization can, thus, serve as a starting point to (a) help create a common vocabulary to improve communication among different visualization sub-groups, (b) facilitate the process of understanding differences and commonalities of the various research sub-fields in visualization, (c) provide an understanding of emerging new research trends, (d) facilitate the crucial step of finding the right reviewers for research submissions, and (e) it can eventually lead to a comprehensive taxonomy of visualization research. One additional tangible outcome of our work is an online query tool (http://keyvis.org/) that allows visualization researchers to easily browse the 3952 keywords used for IEEE VIS papers since 1990 to find related work or make informed keyword choices.We present the results of a comprehensive analysis of visualization paper keywords supplied for 4366 papers submitted to five main visualization conferences. We describe main keywords, topic areas, and 10-year historic trends from two datasets: (1) the standardized PCS taxonomy keywords in use for paper submissions for IEEE InfoVis, IEEE Vis-SciVis, IEEE VAST, EuroVis, and IEEE PacificVis since 2009 and (2) the author-chosen keywords for papers published in the IEEE Visualization conference series (now called IEEE VIS) since 2004. Our analysis of research topics in visualization can serve as a starting point to (a) help create a common vocabulary to improve communication among different visualization subgroups, (b) facilitate the process of understanding differences and commonalities of the various research sub-fields in visualization, (c) provide an understanding of emerging new research trends, (d) facilitate the crucial step of finding the right reviewers for research submissions, and (e) it can eventually lead to a comprehensive taxonomy of visualization research. One additional tangible outcome of our work is an application that allows visualization researchers to easily browse the 2600+ keywords used for IEEE VIS papers during the past 10 years, aiming at more informed and, hence, more effective keyword selections for future visualization publications. Key-words: Keyword analysis, research themes, research topics, taxonomy, visualization history, theory. ∗ Petra Isenberg is with Inria, France. E-mail: petra.isenberg@inria.fr . † Tobias Isenberg is with Inria, France. E-mail: tobias.isenberg@inria.fr . ‡ Michael Sedlmair is with the University of Vienna, Austria. E-mail: michael.sedlmair@univie.ac.at . § Jian Chen is with the University of Maryland, Baltimore County, USA. E-mail: jichen@umbc.edu . ¶ Torsten Möller is with the University of Vienna, Austria. E-mail: torsten.moeller@univie.ac.at . Vers une meilleure compréhension de la visualisation à travers l’analyse de mots-clés Résumé : Nous présentons les résultats d’une analyse exhaustive de mots-clés pour 4366 articles de visualisation soumis à cinq principales conférences de visualisation. Nous décrivons les mots clés principaux, domaines thématiques, et les tendances historiques sur 10 ans pour deux jeux de données: (1) les mots-clés standardisés de la taxonomie PCS actuellement utilisés pour la soumission d’articles à IEEE InfoVis, IEEE Vis-SciVis, IEEE VAST, EUROVIS, et IEEE PacificVis depuis 2009 et (2) les mots-clés choisis par les auteurs pour les articles publiés dans la série de conférences de visualisation IEEE (appelée IEEE VIS depuis 2004). Notre analyse des sujets de recherche en matière de visualisation peut servir de point de départ pour (a) aider à créer un vocabulaire commun pour améliorer la communication entre les différents sous-groupes du domaine de la visualisation, (b) faciliter la compréhension des différences et points communs entre ces différents sous-groupes, (c) mieux comprendre les nouvelles tendances de recherche émergentes, (d) faciliter l’étape cruciale consistant à trouver les bons experts pour la relecture de soumissions, et (e), à terme, conduire à une taxonomie complète de la recherche en visualisation. Un résultat supplémentaire et tangible de notre travail est une application qui permet aux chercheurs en visualisation de parcourir facilement les mots-clés utilisés dans plus de 2600 articles de la conference IEEE VIS au cours des 10 dernières années. Cette application vise à faciliter la sélection mieux informée et par conséquent plus efficace de mots clés pour les futures publications en visualisation. Mots-clés : analyse de mots-clés, thèmes de recherche, taxonomie, l’histoire de la visualisation, la théorie. Toward a deeper understanding of Visualization through keyword analysis 3 1 Motivation One of the main reasons why the field of visualization is such a fascinating field of research is due to its diversity. We not only refer to the diversity of applications, but the diversity of research methods being employed, the diversity of research contributions being made, as well as the diversity of its roots. Diversity of roots: The term visualization can be understood very broadly, expressing a long history of its use in common language. Therefore, it is not surprising that concepts of visual thinking have penetrated many areas of science, engineering, and philosophy. The field of modern (computer-based) visualization has been greatly influenced by research methods from the fields of numerics and computer graphics, which have given it its birth in 1990. The impact of human-computer interaction affected the birth of the InfoVis community in 1995 and the influence of applied statistics (such as data mining) and cognition has led to the establishment of VAST in 2006. Diversity of research methods: Given its diverse roots, visualizations remains a highly inter-disciplinary field that borrows and extends research methods from other fields. Methods come from fields as diverse as the broader computer science, mathematics, statistics, machine learning, psychology, cognitive science, semiotics, design, or art. Diversity of contributions and applications: Based on these diverse influences, the results of visualization research can be manifold: from engineering solutions to dealing with large data sources (such as real-time rendering solutions, distributed and parallel computing technologies, novel display devices, and visualization toolkits) to understanding design processes (as in perceptual guidelines for proper visual encodings and interaction or facilitating collaboration between different users through visual tools) to scientific inquiries (such as improved understanding of perceptual and cognitive processes). While all these diverse influences make the field of visualization research an exciting field to be a part of, they also create enormous challenges. There are different levels of appreciation for all aspects of visualization research, communication challenges between visualization researchers, and the challenge of communicating visualization as a research science to the outside. These issues lead, in particular, to the frequently asked question “what is visualization?”—among funding agencies or even between colleagues. Given our field’s broad nature, we need to ask how we can comprehensively describe and summarize all on-going visualization research. These are not just theoretical and philosophical questions, but the answer to these question has many real-world impacts—from such simple (but career-deciding) questions as finding the right reviewers during peer-review to administrative strategic decisions on conference and journal structures and foci. So while “what is visualization?” is a fundamental question, it is little discussed within our community. In fact, thus far the approaches have mostly focused on understanding some sub-field of visualization (e. g., [16, 29, 34]) but the question for the broader community has rarely been tackled beyond general textbook definitions (e. g., [7]). Those who have approached the problem, did so in a top-down approach. For example, several taxonomies were suggested by experts based on tasks, techniques, or data models (e. g., [8, 29, 35]). Another way of splitting visualization into more focused areas has been through specific application foci (e. g., VisSec, BioVis, SoftVis, etc.). What is missing in this picture is a bottom-up analysis: What types of visualization research are actually happening as expressed by single research contributions in the visualization conferences and journals. Our paper is one of the first steps in this direction. We analyze author-assigned keywords from the three IEEE VisWeek/VIS conferences of the past ten years as well as author-selected taxonomy entries in the submission system for three IEEE VisWeek/VIS conferences, EuroVis, and PacificVis of the past six years. Based on this analysis, we make the following contributions: Mapping visualization research: In Sect. 4, through the vehicle of keyword analysis, we build a conceptual map of all visualization work as indexed by individual authors. Our main assumption here is that, while each single keyword might be understood in a slightly different way by different researchers, their co-occurrence with other keywords clarifies their meaning, especially when aggregated over many

Sparse PDF Volumes for Consistent Multi-Resolution Volume Rendering.

This paper presents a new multi-resolution volume representation called sparse pdf volumes, which enables consistent multi-resolution volume rendering based on probability density functions (pdfs) of voxel neighborhoods. These pdfs are defined in the 4D domain jointly comprising the 3D volume and its 1D intensity range. Crucially, the computation of sparse pdf volumes exploits data coherence in 4D, resulting in a sparse representation with surprisingly low storage requirements. At run time, we dynamically apply transfer functions to the pdfs using simple and fast convolutions. Whereas standard low-pass filtering and down-sampling incur visible differences between resolution levels, the use of pdfs facilitates consistent results independent of the resolution level used. We describe the efficient out-of-core computation of large-scale sparse pdf volumes, using a novel iterative simplification procedure of a mixture of 4D Gaussians. Finally, our data structure is optimized to facilitate interactive multi-resolution volume rendering on GPUs.

TagFlip: Active Mobile Music Discovery with Social Tags

We report on the design and evaluation of TagFlip, a novel interface for active music discovery based on social tags of music. The tool, which was built for phone-sized screens, couples high user control on the recommended music with minimal interaction effort. Contrary to conventional recommenders, which only allow the specification of seed attributes and the subsequent like/dislike of songs, we put the users in the centre of the recommendation process. With a library of 100,000 songs, TagFlip describes each played song to the user through its most popular tags on Last.fm and allows the user to easily specify which of the tags should be considered for the next song, or the next stream of songs. In a lab user study where we compared it to Spotifys mobile application, TagFlip came out on top in both subjective user experience (control, transparency, and trust) and our objective measure of number of interactions per liked song. Our users found TagFlip to be an important complementary experience to that of Spotify, enabling more active and directed discovery sessions as opposed to the mostly passive experience that traditional recommenders offer.

GEMSe: Visualization-Guided Exploration of Multi-channel Segmentation Algorithms

We present GEMSe, an interactive tool for exploring and analyzing the parameter space of multi‐channel segmentation algorithms. Our targeted user group are domain experts who are not necessarily segmentation specialists. GEMSe allows the exploration of the space of possible parameter combinations for a segmentation framework and its ensemble of results. Users start with sampling the parameter space and computing the corresponding segmentations. A hierarchically clustered image tree provides an overview of variations in the resulting space of label images. Details are provided through exemplary images from the selected cluster and histograms visualizing the parameters and the derived output in the selected cluster. The correlation between parameters and derived output as well as the effect of parameter changes can be explored through interactive filtering and scatter plots. We evaluate the usefulness of GEMSe through expert reviews and case studies based on three different kinds of datasets: A synthetic dataset emulating the combination of 3D X‐ray computed tomography with data from K‐Edge spectroscopy, a three‐channel scan of a rock crystal acquired by a Talbot‐Lau grating interferometer X‐ray computed tomography device, as well as a hyperspectral image.

A team-approach to putting learner-centered principles to practice in a large course on Human-Computer Interaction

We present a case study on how a team of instructors put learner-centered principles into practice in a large undergraduate course on Human-Computer Interaction (HCI) that was run in 4 parallel groups of about 50 students. The course stands on the crossroads between software engineering, business, and research in so far as student-teams apply human-centered design techniques to develop mobile apps, test them with real end-users, read research papers and regularly reflect upon their experience. As a proof of the course-concept, selected results from formative and summative assessments are presented. The summative results show that students rated the course as one of the best of the 87 computer science courses run in the summer term of 2015 at the University of Vienna. The primary goal of this paper is to provide instructors intrigued by learner-centered approaches with ideas for their own practice. In particular, this paper is of interest to those who teach Human-Computer Interaction and to those who seek inspiration on mapping their course to the 14 learner-centered principles.

Listen or interact? A Large-scale survey on music listening and management behaviours

The results of an online survey on music listening and management are presented and analysed. With 590 participants, we especially focused on understanding how precise of a control the respondents desired on their music listening, how much interaction with their music source they were willing to have to exert such control, and how these preferences were affected by the type of activity accompanying music listening. A need was observed for novel interfaces that require minimal effort and let users steer the listening experience by controlling key attributes of songs, within the confines of a mobile device. Examples of such attributes were found to be mood, genre, tempo, familiarity, and how distracting the songs are. Along with type of accompanying activity, factors such as age, gender, size of collection, and music listening hours were found to influence the listeners’ control and interaction preferences. Some other notable findings were that our participants had a median of 4650 songs in their music collections, that portable devices were their most popular music source, and that commuting and work were the top activities accompanying music.

Sliceplorer: 1D slices for multi-dimensional continuous functions

Multi‐dimensional continuous functions are commonly visualized with 2D slices or topological views. Here, we explore 1D slices as an alternative approach to show such functions. Our goal with 1D slices is to combine the benefits of topological views, that is, screen space efficiency, with those of slices, that is a close resemblance of the underlying function. We compare 1D slices to 2D slices and topological views, first, by looking at their performance with respect to common function analysis tasks. We also demonstrate 3 usage scenarios: the 2D sinc function, neural network regression, and optimization traces. Based on this evaluation, we characterize the advantages and drawbacks of each of these approaches, and show how interaction can be used to overcome some of the shortcomings.

Hypersliceplorer: Interactive visualization of shapes in multiple dimensions

In this paper we present Hypersliceplorer, an algorithm for generating 2D slices of multi‐dimensional shapes defined by a simplical mesh. Often, slices are generated by using a parametric form and then constraining parameters to view the slice. In our case, we developed an algorithm to slice a simplical mesh of any number of dimensions with a two‐dimensional slice. In order to get a global appreciation of the multi‐dimensional object, we show multiple slices by sampling a number of different slicing points and projecting the slices into a single view per dimension pair. These slices are shown in an interactive viewer which can switch between a global view (all slices) and a local view (single slice). We show how this method can be used to study regular polytopes, differences between spaces of polynomials, and multi‐objective optimization surfaces.