Competing Models: Inferring Exploration Patterns and Information Relevance via Bayesian Model Selection
CCompeting Models: Inferring Exploration Patterns and InformationRelevance via Bayesian Model Selection
Shayan Monadjemi, Roman Garnett, and Alvitta Ottley
Abstract — Analyzing interaction data provides an opportunity to learn about users, uncover their underlying goals, and createintelligent visualization systems. The first step for intelligent response in visualizations is to enable computers to infer user goalsand strategies through observing their interactions with a system. Researchers have proposed multiple techniques to model users,however, their frameworks often depend on the visualization design, interaction space, and dataset. Due to these dependencies, manytechniques do not provide a general algorithmic solution to user exploration modeling. In this paper, we construct a series of modelsbased on the dataset and pose user exploration modeling as a Bayesian model selection problem where we maintain a belief overnumerous competing models that could explain user interactions. Each of these competing models represent an exploration strategythe user could adopt during a session. The goal of our technique is to make high-level and in-depth inferences about the user byobserving their low-level interactions. Although our proposed idea is applicable to various probabilistic model spaces, we demonstratea specific instance of encoding exploration patterns as competing models to infer information relevance. We validate our technique’sability to infer exploration bias , predict future interactions , and summarize an analytic session using user study datasets. Our resultsindicate that depending on the application, our method outperforms established baselines for bias detection and future interactionprediction. Finally, we discuss future research directions based on our proposed modeling paradigm and suggest how practitioners canuse this method to build intelligent visualization systems that understand users’ goals and adapt to improve the exploration process. Index Terms —User Interaction Modeling, Bayesian Machine Learning
NTRODUCTION
Visual analytics has transformed the process of reasoning with data byviewing humans and machines as teammates with unique strengths. Dueto the growing complexity and volume of data, there is a need for moreintelligent visual analytic systems to assist in data exploration, decision-making, and communication of findings. Studies suggest that even withinteractive visualizations, there are human-related factors that hinderdata exploration and informed decision-making. For instance, analystsoften search through a large amount of information where the irrelevantportions of the data can be distracting and exceed the limits of humancognition [19]. An intelligent visual analytic system can respond byonly showing portions of the data relevant to the analyst. An everydayuser might interact with only a biased subset of the data which impedesdecision-making [10]. An intelligent visual analytic system can respondby informing the user about their biases through notifications or mitigatetheir biases through modifying the visualization. Creating an effectiveintelligent visual analytic system requires careful design considerationsincluding the level of intrusiveness and the means of intervention [42].Regardless of what form this intelligent visual analytic system takes on,it must be able to offer well-informed assistance to the user.One promising approach to informing useful machine response isthrough capturing and analyzing users’ interaction data. Research inthe area of
Analytic Provenance has shown that interaction data canreveal valuable information about the user and their analysis strategiesand how the machine teammates can better assist the user.For example, Dabek and Caban proposed a grammar-based approachto uncover common patterns among a group of users [7], resulting insuggestions to assist users in data exploration. Battle et al. [1] used aMarkov chain that utilized navigation behavior to determine portionsof a satellite image that a given user will likely explore in the future,resulting in improving latency by 430%. Similarly, Ottley et al. [33]and Wall et al. [41] used hidden Markov models to infer users’ attention • Shayan Monadjemi is with Washington University. [email protected].• Roman Garnett is with Washington University. [email protected].• Alvitta Ottley is with Washington University. [email protected] received xx xxx. 201x; accepted xx xxx. 201x. Date of Publicationxx xxx. 201x; date of current version 15 Oct. 2020. For information onobtaining reprints of this article, please send e-mail to: [email protected] Object Identifier: 10.1109/TVCG.2020.3030430 and cognitive bias respectively, resulting in anticipating future clicksand analyzing user exploration bias. For a more comprehensive surveyon user modeling with provenance data, see Ragan et al. [35] andXu et al. [43]. Unfortunately, much of the existing solutions dependon the visualization encoding or the interaction design, and do noteasily provide a general algorithmic solution to uncovering explorationpatterns and quantifying information relevance.In this paper, we take a more comprehensive view of user modelingthat encompasses what the user has done, is doing, and will do in thefuture. Specifically, we use observed interactions as model evidence,and we leverage Bayesian model selection to reason about the subsetof a multi-dimensional dataset and the collection of dimensions thatmost likely gave rise to the observations. This approach provides avaluable framework to build, maintain, and select models that we derivefrom the dimensions in the dataset, and it offers two key advantages.First, it utilizes straightforward probability theory that is easy to un-derstand and implement. Second, a unique benefit of this approachis that we can make high-level inferences about a user’s explorationpattern by passively observing low-level interactions with the datapoints. Once we identify the data dimensions that are most relevant,we demonstrate that we can use our models to achieve a variety ofuser modeling objectives: infer exploration bias , predict future inter-actions , and summarize an analytic session . We validate our proposedframework using three crowdsourced test datasets and we compare ouralgorithm’s performance to existing baseline models. We show that ourmodeling technique outperforms established baselines in most cases.Furthermore, we discuss the possibilities this modeling paradigm opensfor future investigations. We summarize our contributions as follows:• We introduce a straightforward approach to inferring multi-dimensional data exploration strategy in real-time.
We encodedifferent exploration strategies as a set of models and utilizesBayesian model selection to reason about which strategy is likelyto give rise to past interactions with a point-based visualization.•
We demonstrate our method’s flexibility to address multiple as-pects of user explorations modeling when applied to informationrelevance.
Specifically, we show that we can apply this frame-work to infer exploration bias, to predict future interaction, and tosummarize an analysis session.•
We validate the proposed framework using three crowdsourceduser interaction datasets.
Our results indicate that, depending onthe application, our method outperforms established baselines. a r X i v : . [ c s . H C ] O c t B ACKGROUND
Learning and modeling user behavior is a common goal in the Vi-sual Analytics community [33, 35, 36, 41].
Analytic Provenance , whichbroadly refers to the research goal of tracking and modeling the analysisprocess, provides valuable information for inferring certain characteris-tics of exploration. Ragan et al. [35] categorize provenance into fivecategories: provenance of data, visualization, interactions, insights, andrationale. Furthermore, they categorize the primary purpose of trackingprovenance data as recall [12, 28, 39], replication [8, 24, 29, 32], actionrecovery [9, 11, 18, 26, 38], collaborative communication [5, 13, 14, 30], presentation [20, 21], and meta-analysis [22, 34]. This paper aims toleverage interaction provenance data to learn about the users and theirgoals in order to improve the visualization system.
Recent work has begun to investigate how learning about the users’ an-alytic process can help us improve visualization systems. For example,Battle et al. [1] demonstrated this concept by using user explorationmodeling to intelligently pre-fetch data for a map visualization. Theyobserved user actions such as { hover, click, pan-left, pan-right, ... } ,maintained a Markov chain model, and ranked the data for pre-fetchingaccording to the likelihood of the users taking actions corresponding tosaid data. Their approach improved the overall latency by 430%.Brown et al. [3] have proposed Dis-function as a method to learnexpert knowledge through interaction data as a distance function rep-resenting the underlying similarities between data points. In theirframework, the expert user interacts with the visualization by posi-tioning similar points close to one another. By observing the user’sorganization of points, the algorithm infers a distance function thatmimics the experts’ knowledge. In similar work, Iwata et al. [23] took astep further to minimize the number of user interactions needed for find-ing the desired visualization representing high dimensional data. Usingan active learning approach, they queried the user to relocate pointsstrategically in order to reduce the uncertainty of the visualization.Dabek and Caban [7] have proposed a grammar-based approachto model the sequence of low-level interaction in a session. Theirwork addresses the overwhelming number of controls and interactionchannels available on modern visual analytics systems by using finitestate machines to make suggestions for future interactions during asession. While their technique is task-dependent and relies on trainingdata from past users performing the same tasks, their findings showthat appropriate recommendations help users in data exploration.This set of investigations provide examples on how user explorationmodeling can improve user experience and decision making outcomesin visual analytic systems. However, many of them suffer from de-pendence on visualization design, tasks, or datasets. In line with ourmotivation of building intelligent visual analytic system who observeuser interactions and infer what the user has done, is doing, and will doin the future, we propose an algorithmic and generalizable approach tobuilding competing models on how users may interact with data. More-over, we use Bayesian iterative updating to maintain our beliefs duringsessions. We specifically design our models to take a comprehensiveview on interactive sessions by inferring exploration bias, predictingfuture interactions, and summarizing sessions. In the remainder of thissection, we discuss the most closely related work to this paper.
In the context of visual analytics, bias may arise from different sources.Related literature has studied the phenomena of bias from the lensof data, models, and users . Gotz et al. [16] proposed a toolkit tocombat selection bias, where the source of bias is in the data . Forinstance, filtering a real estate data set to include houses less than$300K will automatically introduce a bias towards the location ofselected houses since price and neighborhoods are highly correlated.Cabrera et al. [4] proposed a visual analytic tool, FairVis, to combatbias in machine learning models . For example, a model designed forrecommending employment candidates for interviews may unfairlydisregard candidates solely due to their historic under-representation ina career field. In this work, we are interested in bias from the lens of users . As themost relevant work to this paper, Wall et al. [41] introduced metrics tomeasure cognitive bias through user interaction with visualized data.Cognitive bias is measured in terms of how much of the dataset hasbeen explored, which attributes of the data points were important tothe user, and what value of those attributes have influenced the users’decisions. As an example, imagine a parent who is interacting witha map of crimes to determine if a specific neighborhood is safe fortheir family. They may begin by looking for sex crimes in that specificneighborhood. The algorithm soon determines that the location and type of the crime are the two attributes driving the user’s exploration,while the time attribute has been uniformly explored. Presenting thisbias to the user during the session may result in them confirming theirinterest in a subset of data, or it may encourage more explorationand mitigate unintentional bias. While mitigating information biasrequires careful design considerations which are beyond the scope ofthis paper (e.g. degree of guidance, level of intrusiveness, and meansof interference [42]), we demonstrate our technique’s ability to inferintersectional bias in exploration which provides critical informationfor bias mitigation.
The term interaction can take on numerous meanings in the contextof interactive visualizations. For example, interaction may refer tolow level events such as clicks, hovers, and drag/drops, or it mayrefer to higher level tasks such as filtering and sorting. As a result,Gotz et al. [16] have proposed a taxonomy where analytic behavioris categorized in to four groups: tasks, sub-tasks, actions, and events .Their taxonomy created a spectrum based on semantics, where tasks onone end of the spectrum refer to interactions with high semantics (e.g.identify market insights for promising investments) and events on theother end of the spectrum refer to interactions with poor semantics (e.g.an individual click).In the current literature, predicting future interactions can take ondifferent meanings as well. For example, Dabek and Caban [7] buildmodels to predict future low-level events , whereas Ottley et al. [33] andBattle et al. [1] build models to predict which data point the user islikely to interact with next. Our work is related to the latter category,as we will infer the relavance of data points to the user in light of pastvisited data points and compare our technique with Ottley et al. [33] asa baseline. Ottley et al. [33] proposed a hidden Markov model (HMM)approach to maintain a belief over users’ evolving attention and actionsin a visualization system. They encode user clicks as a sequence ofvisual attributes with an added bias metric to determine what attributeis influencing the exploration. As user clicks arrive, they update themodel and use particle filtering to infer a set of top- k point candidatesfor the next click. In contrast to their work, our models are built onthe data rather than the visual elements . Since not all attributes of amulti-dimensional dataset can have a visual representation, we believethat building the model on the dataset itself opens up an opportunityfor learning about users’ interest in any of the attributes, not only theones visualized. The body of work in this area ranges from primitive undo/redo function-alities to communicating provenance data from collaborative sessions.Heer et al. [20] explore tools for visualizing provenance data, anddiscuss how such tools can be used to improve user interface design.Xu et al. [44] propose
Charts Constellations , a visualization tool toaggregate provenance data from multiple analysts in order to commu-nicate the depth of exploration in different parts of a dataset. Chen etal. [6] observe low-level interactions and create annotations to assistusers with insight externalization. Gratzl et al. [17] propose
CLUE , amodeling framework that joins exploration and communication of dis-coveries by automatically storing provenance data during explorationand generating a session report.In another related work, Sarvghad et al. [37] designed an experimentwhere users explore a multi-dimensional dataset to gain insight intobusiness performance of an online retailer. Their study suggests that able 1. Table of notations used in this paper and their descriptions.
Notation Description D The set of all data points visualized C The set of data points interacted with X The space of all possible data points M The set of all models M i An individual model from MA c , i The set of continuous attributes relevant to M i A d , i The set of discrete attributes relevant to M i those who were provided with a summary on which attributes they havecovered in their exploration were able to formulate more questionsand explore more broadly without sacrificing the depth of insights.Our proposed model in this paper allows us to accomplish a similarpremise as prior work: to summarize a session. This capability cansimply communicate our model’s understanding of the user or it can bepresented to the user to promote more exploration in a similar fashionas Sarvghad et al. [37]. RELIMINARY D EFINITIONS
We begin with some definitions to frame the reader’s understanding ofthe problem space. We assume the user interacts with a set of point-based visual data and each point of interest (cid:126) x can be described by avector of d continuous or discrete attributes. We define a corresponding d -dimensional data space of possible realizations of these attributes, X = { X × X × ... × X d } , where X i is the domain of the i th attribute.We will consider a visualization of a dataset – that is, a collection ofpoints in the data space – D = { (cid:126) x ,(cid:126) x ,...,(cid:126) x m } , (cid:126) x i ∈ X . We assumethat a user generates a stream of interaction data , a sequence of itemsin the dataset the user has interacted with, which we will notate with C = { (cid:126) c ,(cid:126) c ,... } , (cid:126) c j ∈ D . While our definition of interaction data isindependent of the type of interaction, this paper considers exampleswhere the interactions occur by clicking or hovering . Our technique,however, may be expanded to include datapoints from other means ofinteraction subject to limitations discussed in Section 8. We approachuser interaction modeling via the framework of Bayesian model selec-tion. We construct competing high-level models of a user’s explorationpattern based on different attributes of D , and use a given set of inter-action data to determine which of these models is the most plausible.Here, a model is a parametric family of probability distributions toexplain a given set of observations, and we will construct models thatallow for a variety of different subspaces/subsets of the data space tobe relevant (or not) for a user, spanning a large space of hypotheses.In the Bayesian approach, there are three major steps to probabilisticinference:• The prior represents a belief distribution over possible values ofthe parameter before data is observed.• The likelihood specifies the probability of observing a given setof observations (such as a click stream) assuming given valuesfor the parameters.• The posterior is the updated distribution over possible values ofthe parameter in light of the prior beliefs and the information inthe observations.Going forward, we initiate the prior belief and maintain the posterior belief over a set of competing models in Section 4.1. Moreover, wediscuss the likelihood , or the relevance of data points in light of pastobservations, in section 4.2. Finally, we use the competing models todetect exploration bias, predict future interactions, and summarize asession (Section 5). OMPETING M ODELS F RAMEWORK
The novelty of this work is that we present different exploration patternsas models, and use the Bayesian model selection framework to maintaina belief over all possible exploration patterns. Our approach to user in-teraction modeling assumes that users interact with visualizations withsome (possibly subconscious) pattern in mind. The ultimate goal of ourtechnique is to detect and model these exploration patterns. We define model space to be a finite set of models representing various explo-ration patterns. The model space is denoted by M = { M , M ,..., M n } where M i is an individual model representing one possible explorationpattern. Depending on the choice model space, these individual modelscould inform us about user’s bias in certain attributes of data, a specifictask a user may be performing, or something about a user’s personal-ity. In this work, we encode exploration patterns in terms of whichsubset of data dimensions drive user exploration. For a dataset with d dimensions, we create a model space of 2 d models, where each modelrepresents exploration based on a subset of d dimensions. As a userinteracts with a visualization, we maintain a belief over viability ofeach model given the observed interactions. The belief over our modelsis incrementally updated using the Bayes’ rule: p ( M i | interactions ) ∝ p ( interactions | M i ) p ( M i ) , (1)where each model is evaluated in light of observed interactions andprior beliefs. Next, we discuss the details of initiating a prior beliefover the model space and updating the belief as the user interacts withdata points. Before observing any interactions, our prior assumes a uniform distri-bution over the set of all possible models. That is, for every model, M i , p ( M i ) = d (2)This choice of prior reflects the idea that we are uniformly uncertainat the beginning about what combination of attributes best describeexploration patterns. In other words, we consider every explorationpattern to be equally likely before observing any interactions. In Section7.2, we will demonstrate an alternative prior that penalizes modelswhich encompass exploration bias towards a large number of attributes.As new interactions occur, we update these models throughout asession via the process of iterative Bayesian updating. That is, giventhe set of interactions, C , the posterior belief over the set of models is: p ( M i | C ) ∝ p ( C | M i ) p ( M i ) (3)where p ( C | M i ) is the likelihood representing how well a model M i can explain the observations in C . We can expand the likelihoodfunction via chain rule to get: p ( C | M i ) = ∏ (cid:126) c j ∈ C p ( (cid:126) c j | M i , C j − ) (4)In Eq. 4, (cid:126) c j is the j th click and C j − is the set of first j − p ( (cid:126) c j | M i , C j − ) is a measure of how relevant (cid:126) c j is given model M i in light of all past observations.This framework of competing models thus far is flexible to operateregardless of visualization design or dataset. Picking a likelihood func-tion is the key part to fitting this framework into specific applications.For example, if our application involves a collection of text documentsas the underlying dataset, we need to choose a likelihood function thatdescribes user interactions with support over the set of text documents.If our application involves multidimensional data with continuous anddiscrete attributes, we need to choose a likelihood function that de-scribes user interactions with support over all data points. In Section4.2, we narrow the scope of possibilities and consider a specific likeli-hood function that models information relevance in multidimensionaldatasets with continuous and discrete attributes. The focus of this section is on modeling how relevant a data point isto the user given their past interactions. This measure of relevance isdenoted in Eq. 4 as p ( (cid:126) c j | M i , C j − ) . In more formal terms, we wantto use the evolving interaction data to infer the user’s data objective, which we define to be some user-determined probability distribution p ( (cid:126) x ) over the data space with support over the items of interest. Pointswith higher values of p ( (cid:126) x ) are considered more relevant to the session.ur framework requires the models to be probabilistic (i.e. includeinformation about uncertainty in outputs). There are multiple tech-niques for inferring the relevance of a data point to the user in light ofpast observations. Some examples include probabilistic classificationtechniques such as logistic regression and k -nearest neighbors model.This freedom in model choices makes our approach design-agnostic.Depending on the scenario and the dataset, practitioners may encodeexploration patterns into appropriate models. In this work, we modelthe distribution of data points with which the user has interacted. Thisproof-of-concept approach provides us with a simple and interpretablemethod to explore the idea of competing models. Since the multi-dimensional datasets in this paper involve discrete and continuousdimensions, we utilize two kinds of parametric distributions: Gaussianand categorical. While the parametric nature of these distributionsimpose some restrictions (e.g. unimodality of the Gassian), their pa-rameters make the distributions easier to interpret by simply observingthe value of parameters. Continuous Dimensions
The Gaussian distribution is commonlyused for continuous variables. We utilize this family of distributionsto model user’s interest in each continuous dimension of the data. Forevery model, M i , we construct a | A c , i | + time ) of interactions, where | A c , i | is thenumber of continuous dimensions involved in a given model M i andan extra dimension is added to model the order at which observationsarrive. Multivariate Gaussian distribution is parameterized by a meanvector (cid:126) µ and a covariance matrix Σ , representing center and spread ofthe distribution respectively. Since the value of these parameters ( (cid:126) µ , Σ ) for a given set of observations C is unknown, we infer them thoroughthe Bayesian inference process which results in a closed form posteriorpredictive distribution function, f c ( (cid:126) x | C ) . This probability distributionfunction, f c is known as Student’s t-distribution [2, 31]. Moreover, theBayesian inference for this distribution involves four hyper-parameters.We set these hyper-parameters so that the prior belief is uninformative (i.e. contains no specific information about user exploration beforeinteractions are observed). The derivation details of this posterior pre-dictive distribution as well as the choice of hyper-parameters are furtherdiscussed in the supplementary material . Finally, we normalize theprobability density function f c across all data points in D to assign aprobability value to the continuous dimensions of every point (cid:126) x ∈ D : p c ( (cid:126) x | M i , C t , time = t + ) = f c ( (cid:126) x [ A c , i ] | C t , time = t + ) ∑ (cid:126) x (cid:48) ∈ D f c ( (cid:126) x (cid:48) [ A c , i ] | C t , time = t + ) (5)In the equation above, (cid:126) x [ A c , i ] denotes the continuous dimensions of (cid:126) x that are relevant in model M i . Discrete Dimensions
The categorical model is used to explain theprobability of discrete events occurring. For an attribute domain with K possible categories, the categorical model has a K -dimensional vector (cid:126) µ which describes the probability of observing each of the K choices.Since the value of (cid:126) µ is unknown, we infer it via the Bayesian inferenceprocess. The Bayesian inference process for the categorical distributionrelies on one hyper-parameter ( (cid:126) α ) representing the pseudocount for theprior. We set this hyper-parameter so that the prior is uninformative asdemonstrated in Fig. 5 (i.e. contains no specific information about userexploration before interactions are observed). Let d (cid:48) be a categoricalattribute with K possible values. The posterior predictive of observingcategory k given a set of interactions C is [2, 40]: f d (cid:48) ( k | C ,(cid:126) α ) = α k + m k ∑ Ki = ( α i + m i ) (6)where α i is the pseudocount for category i (from hyper-parameter (cid:126) α )and m i is the observation count for category i . We offer more detailson this hyper-parameter and the derivation of closed-form posterior https://github.com/smonadjemi/competing_models predictive in the supplemental material. We normalize the posteriorpredictive of each categorical distribution across all data points in D to assign a probability value to each discrete dimension of every point (cid:126) x ∈ D . For a discrete dimension d (cid:48) we have: p d (cid:48) ( (cid:126) x | C t ) = f d (cid:48) ( (cid:126) x [ d (cid:48) ] | C t ) ∑ (cid:126) x (cid:48) ∈ D s.t. (cid:126) x [ d (cid:48) ]= (cid:126) x (cid:48) [ d (cid:48) ] (cid:126) x [ d (cid:48) ] is the value of attribute d (cid:48) for data point (cid:126) x . Combined Point Likelihood
Equations 5 and 7 provide us withprobability distributions over continuous and discrete dimensions ofthe dataset respectively. The overall probability of any point (cid:126) x ∈ D within a model M i in light of past t interactions C t is: p ( (cid:126) x | M i , C t ) = p c ( (cid:126) x | M i , C t , time = t + ) ∏ d ∈ A d , i p d ( (cid:126) x | C t ) (8)This section provided a detailed explanation for initiating and main-taining a belief over a set of competing models. Specifically, section 4.1discussed the prior and posterior portions of equation 3, and section 4.2discussed the likelihood portion of equation 3. Going forward, we ap-ply these concepts to detect exploration bias, predict future clicks, andsummarize an analytic session. PPLICATIONS OF C OMPETING M ODELS
The ultimate goal of our technique is to make high-level and in-depthinferences by observing their low-level user interactions. More specifi-cally, our technique takes a more comprehensive view of user modelingby enabling us to infer exploration bias , predict future interactions , and summarize an analysis session . In this section, we use our competingmodels framework (Section 4) in order to gain insight into users. Bias often has a negative connotation; however, in reality bias can bedesirable. In very simple cases, bias can indicate the criteria in a dataspace on which people make decisions [41]. Therefore, we categorizebias into two categories: intentional and unintentional. Intentional biasholds information about the users’ search criteria and can acceleratethe analysis process. Unintentional bias, on the other hand, arisesfrom the user subconsciously avoiding some information and exploringothers based on some personal factor. Unintentional bias leads tomissing information. Regardless of the category, detecting biases invisualization systems can improve the quality of analysis and decision.Throughout a session, our framework maintains a posterior beliefover the set of models { M , M ,..., M d } each of which representbiased exploration towards a subset of attributes. The normalizedposterior belief, { p ( M | C ) , p ( M | C ) ,..., p ( M d | C ) } , informs usabout which subset of attributes the user may be biased towards. Highervalues of p ( M i | C ) indicate higher chances of the user being biasedtowards attributes represented in M i .Using the normalized posterior belief over models, { p ( M | C ) , p ( M | C ) ,..., p ( M d | C ) } , we can use the lawof total probability to calculate the chance of bias towards eachindividual attribute: p ( bias towards attribute a | C ) = ∑ M i ∈ M a p ( M i | C ) (9)where M a denotes the subset of models in which bias towards attribute a is assumed. Predicting which data points the user may interact with next opensopportunities to add features such as target assistance and target gravityto help users find the next most interesting data points in their explo-ration session [33]. Anticipating next clicks given past clicks involvesa time component. Adding time-steps as an additional dimension toour continuous model enables us to correlate clicks not only by theirttributes, but also by the time-step at which they were clicked. Foreach of the data points, x ∈ D , we calculate the probability of the userinteracting with x at the next time-step via: p ( (cid:126) x | C t , time = t + ) = ∑ M i ∈ M p ( (cid:126) x | M i , C t , time = t + ) p ( M i | C t ) (10)The Bayesian approach for not losing any information by selectingone single model is Bayesian model averaging , which we use in theequation above.
Using provenance data to summarize analytic sessions has been of inter-est in the visualization community [17, 20, 44]. Due to our parametricchoice of models in Section 4.2, we show that those parameters can beused to represent a summary distribution from which the clicks weregenerated. For example, let a be a continuous attribute. The updatedparameters of the Gaussian distribution ( µ a , σ a ) after observing inter-actions can display a summary of what distribution of the values inattribute a the user has explored. In the discrete case, the parameter ofcategorical distribution serves as a summary of observed interactions ina certain categorical domain. Section 7 demonstrates this effect throughfigures. ND - TO -E ND E XAMPLE
In this section, we consider an end-to-end example with a small datasetto demonstrate our framework from Sections 4 and 5. Let D be a set ofseven fictitious restaurants with location and food type being continuousand discrete attributes respectively. We encode each restaurant into a3D vector of form ( latitude , longitude , type ) to get: D = { ( . , . , Italian ) , ( . , . , Mexican ) , ( . , . , Persian ) , ( . , . , Italian ) , ( . , . , Mexican ) , ( . , . , Persian ) , ( . , . , Mexican ) } We assume that exploration based on latitude or longitude alone islikely unnatural, and combine the two to be a single 2D continuousattribute, location . Hence, the set of continuous attributes is A c = { location } and the set of discrete attributes is A d = { type } . Sincethe user may explore this dataset based on any combination of theseattributes, we construct 2 | A c | + | A d | = C = { ( . , . , Persian ) , ( . , . , Mexican ) } The interaction type here is irrelevant, but for the sake of examplewe could assume it occurs through clicking, hovering, or adding tofavorites.
Table 2. Example model space M , where each model represents ex-ploration based on a subset of attributes. ¸ indicates an attribute isconsidered important to the exploration session and Ø indicates anattribute is not considered important to the exploration session. Model loc. type p ( M i ) p ( M i | C ) p ( M i | C ) M Ø Ø M ¸ Ø M Ø ¸ M ¸ ¸ p ( M i ) col-umn in Table 2). As more clicks arrive, our posterior belief gainsconfidence that location and type attributes explain the interactions ( p ( M | C ) = .
51 and p ( M | C ) in Table 2). The posteriors(columns p ( M i | C ) and p ( M i | C ) ) are computed using a normal-ized probability mass function (Figure 1) as the likelihood function.In order to detect exploration bias towards a certain attribute, weuse Eq. 9 to compute the marginal probability of models involving thatattribute. For example, the probability of bias towards location afterobserving two clicks is 0.80, since p ( M | C ) + p ( M | C ) = . { M , M } both involve the location attribute.For next click prediction , we compute the likelihood of every everypoint averaged over all models. For every point x ∈ D , we compute: ∑ M i ∈{ M , M , M , M } p ( x | M i , C , time = ) p ( M i | C ) which results to ( Mexican , . , . ) being the predicted next inter-action (assuming no repeated clicks). More specifically, the order ofpoints for being the next interaction is as follows: ( Mexican , . , . ) > ( Persian , . , . ) > ( Mexican , . , . ) > ( Mexican , . , . ) > ( Persian , . , . ) > ( Italian , . , . ) > ( Italian , . , . ) Notice that location and type driving exploration (as inferred in Table 2)is reflected in the ordering above where the top choices for a third click(assuming no repeated clicks) are Mexican and Persian restaurants inorder of their proximity to past interactions. Moreover, notice that thedistributions in Figure 1 start with uninformative priors when t =
0, butupdate to summarize interactions as they arrive ( t = t = Fig. 1. The inferred Gaussian and categorical distributions summarizingthe first two interactions . As clicks (denoted by (cid:155) ) arrive, our models getupdated to represent user interactions.
ALIDATION WITH U SER S TUDY D ATASETS
We validate our proposed technique on three user study datasets. Eachdataset seeks to highlight a unique aspect of our technique. First, wevalidate our technique using a user study dataset collected by Ottley etal. [33] where users are asked to perform specific tasks using a mapvisualization of crimes in St. Louis. Since this user study dataset in-cludes ground truths associated with each session, we can compare ouroutcomes of exploration bias detection and next interaction prediction with established baselines by Wall et al. [41] and Ottley et al. [33]respectively.Second, we demonstrate our technique’s independence from visual-ization design by validating it on a user study dataset collected by Fenget al. [15] where users freely interact with a visualization of S&P 500companies. While this dataset still involves a point-based visualization,it differs from the first dataset in that it is not a map-based visualizationand the interaction technique is hovers instead of clicks . Consideringthat the ground-truth bias in these open-ended sessions are unknown,we are only able to compare the outcome of next interaction prediction with the established baseline by Ottley et al. [33].Lastly, we demonstrating our technique’s independence from tasksby validating it on an open-ended user interaction session with the map ig. 2. The map of St. Louis crimes used by Ottley et al. [33]. Theposition of the dots indicate the location of the crime, and the color of thedots indicate the type of crime. Users explored the data by clicking onthe dots and observing more information in a tooltip.Table 3. Number of participants and ground truth exploration biases foreach category of tasks in the Ottley et al. experiment [33]
Task Category Ground-Truth Bias latitude, longitude type latitude, longitude, type et al. experimental setup
Ottley et al. [33] describe a user study in which they capture mouseclick data as participants interact with a map visualization of crimes inSaint Louis. In this experiment, participants were presented a set of1,951 reported crimes from March 2017 visualized on an interactivemap. Each instance was displayed as a dot with a position and a colorindicating the location and type of the crime respectively. The mapresponded to user clicks by triggering a tooltip containing more detailson the crime. The questions presented to the participants required themto search the map to find an answer based on the data:1. Out of all the cases of Homicide, one case differs from the othercases with regard to time. What is the time of that case?2. How many cases of arson occur during PM?3. There are four types Theft-Related crime in the red shaded region:Larceny, Burglary, Robbery and Motor Vehicle Theft. Count thenumber of cases of Robbery in the red shaded region.4. There are two types of Assault: Aggravated and Non-Aggravatedassault. Count the number of Non-Aggravated Assault in the redshaded region5. Count the number of crimes that occur during 7:00 AM - 12:30PM in the red shaded region.6. Count the number of crimes during AM in the red shaded region.The six questions above were categorized into three groups of tasks: geo-based (search specific location, questions 5-6), type-based (searchfor specific type of crime, questions 1-2), and mixed (search for aparticular type of crime in a specific region of the map, questions 3-4).Table 3 summarizes the number of participants for each group of tasks.
In our first set of analysis, we consider exploration bias detection basedon the ground-truths presented in Table 3, and we compare the output of our technique from Eq. 9 with the equivalent baseline metric. Asdiscussed in Section 2, Wall et al. [41] propose an attribute distribution metric to measure how biased an exploration session is towards anyparticular attribute. They define this metric to be the complement ofobserving the set of interactions assuming the distribution of attributeswithin the full dataset and interaction dataset are the same. Theyspecifically suggest the non-parametric KS test for continuous attributesand Chi-Square test for discrete attributes. For each attribute, these teststake two sets of 1-dimensional sets as inputs (full data and interactiondata), and decide if the two sets are from the same distribution. Let p a be the output of a KS or Chi-Square test for attribute a ; then, the attribute distribution metric is defined as b Ad ( a ) = − p a . Note thathigher values of b Ad ( a ) correspond to more exploration bias towardsattribute a . For tasks with bias towards more than one attribute, wecompute the product of this attribute distribution metric to representthe conjunction of biases. A comparison between our algorithm’s biasdetection and the baseline is shown in Figure 3. After 12 clicks, ourmethod significantly outperformed Wall et al. on the geo-based andmixed tasks according to a paired t-test with p-value 1 . × − and0 . next clickprediction and compare the outcome of our technique with the baselinefrom Ottley et al. [33]. This baseline is based on the idea that usersclick on items that are within a close proximity from each other. Theimplementation of this baseline involves iteratively sampling particlesfrom a current belief, using the observation model to re-weight theparticles based on observed clicks, and ranking the datapoints accordingto the weight and proximity of particles at each timestep. Then, theset of top- k candidates are selected to represent a prediction set forthe next datapoint with which the user may interact. In order to beconsistent with the existing work, we follow a similar approach inour technique: after averaging our competing models to compute thepredictive posterior of each data point being the next observation asoutlined in Eq. 10, we sort them based on ranking and pick the top- k data points as the set of candidates for next interaction. At each time-step, we determine if our prediction set included the next click andcompute the rate of success in this prediction process. A comparisonbetween our algorithm’s next click prediction and the baseline is shownin Figure 4. Across all three tasks, our technique outperforms thisbaseline for the majority of k values and performs within the marginof error for the rest. Anticipating future data points with which theuser may interact with has implications in designing more intelligentvisualization systems which understand user goals and assist them infinding the information they desire. We discuss more of this topic inSection 9.The final goal of our technique is to summarize analytic sessions .Since we chose parametric models to represent user interactions, sum-marizing a session is just a matter of observing the value of parametersat each time-step. Figure 5 shows the progression of our algorithmfrom as new interactions arrive. By only looking at the summary ofinteractions through time, we can easily map the session to question4, where the user is exploring Assault cases in a particular region ofSt. Louis. Thus far in our validation, we have relied on a map visualization withwhich users have to interact through clicks in order to accomplish pre-defined tasks. In this part of our validation, we consider a multi-sectionvisualization of S&P 500 board of directors published by Wall Street ig. 3. Results of exploration bias detection on user study dataset from the Ottley et al. experiment (Section 7.1). The participants were askedto interact with a map visualization of crimes in St. Louis in order to answer geo-based questions (based on latitude and longitude) , type-basedquestions (based on type) , and mixed questions (based on latitude, longitude, and type) . After 12 clicks, our method significantly outperformedWall et al. on the geo-based and mixed tasks according to a paired t-test with p-value . × − and . , respectively. Wall et al. outperformedour method on the type tasks with p-value 0.1. The left graph shows that our method detects exploration bias towards location quicker and moreconfidently than the baseline. The middle graph shows that our technique detects exploration bias towards the type , however, it takes longer than thebaseline to gain confidence in the level of bias (discussed in Section 7.1.2). The right graph shows that our method detects bias towards locationand type quicker and more confidently than the baseline. The shaded region in all three graphs represent the standard error among 28 geo-basedsessions, 23 type-based sessions, and 27 mixed sessions.Fig. 4. Results of next click prediction on user study dataset from the Ottley et al. experiment (Section 7.1). Our method performs within themargin of error of the previously studied hidden Markov model in all categories, while outperforming it in most cases. The error bars in all threegraphs represent the standard error among 28 geo-based sessions, 23 type-based sessions, and 27 mixed sessions.Fig. 5. An instance of summarizing analytic sessions , where the user is performing mixed interactions (based on location and type) in order toanswer question 4 from Section 7.1. The left-most graph shows the distribution of the full crime dataset presented to participants. At t = before anyclicks have been observed, our models reflect our prior. As clicks (denoted by (cid:155) ) arrive, our models get updated to represent user interactions. Noticethe significant difference in type model from t = to t = . This is because our pseudo-count for the prior is small, making the prior insignificant incomparison to observed user clicks.ig. 6. “Inside America’s Boardrooms” a multi-section visualization pub-lished in the Wall Street Journal presenting data on leaders of S&P 500companies [27]. Users from the Feng et al. experiment [15] interactedwith this visualization by hovering over different companies and observingmore information about the leaders of the companies in a tooltip. Journal [27]. The underlying data for this visualization has one discreteand six continuous attributes. A tooltip containing these attributesappears on hover. A view of this visualization is shown in Figure 6. et al. experimental setup
Feng et al. [15] collected user interaction sessions, where users freelyinteract with point-based elements in a visualization and search forkeywords. Specifically, they use the visualization called ”Inside Amer-ica’s Boardroom” published in the Wall Street Journal (Fig. 6). Thisunderlying data for this visualization contains 7 dimensions: marketcapitalization, ratio of unrelated board members, ratio of female boardmembers, average age of board members, average tenure, median pay,and industry group . As opposed to the Ottley et al. experiment, partici-pants in this study were asked to perform an open-ended explorationon the dataset. The recorded sessions contain a sequence hovers ondifferent data points and the duration of each hover. While this userstudy was designed to study the impact of having search capabilitiesin visualizations, we utilize the hover data to validate our techniquefor predicting future hovers. Since hovers can inherently be noisierthan clicks due to unintentional hovers while going from one data pointto another, we filtered the sessions to only include hovers that lastedfor over one second. This will eliminate most unintentional hoversthat occurred in transition of going from one data point to another.Moreover, we filtered the data further to include only those with morethan 3 hovers. Before filtering, there were 41 sessions with avg. 391.19hovers per sessions (SD 237.75). After filtering, we had 39 sessionswith avg. 26.83 hovers (SD 18.95) and at least 3 hovers per session.
In our analysis of this user study, we consider next hover prediction .We modify the Ottley et al. baseline from Section 7.1 to include allseven attributes of this dataset. Our technique creates a space of 2 models (corresponding to all possible subsets of the 7 dimensions),where each model represents biased exploration towards a certain subsetof attributes. Furthermore, we make an adjustment to the prior beliefover the model space in order to penalize models that encompass biastowards a large subset of attributes. In Section 4.1, we suggested theuniform prior belief p ( M i ) = / d , however, here we use p ( M i ) ∝ / ( d + ) , where d (cid:48) i is the number of attributes represented in model M i . Figure 7 shows that our technique outperforms the establishedbaseline for higher values of k and performs within the margin of errorfor lower k . In our final phase of validation, we reconsider the map of crimes inSt. Louis. The experiments in Section 7.1 involved narrowly definedtasks. In this case-study, we have selected one analytic session wherethe user freely interacted with the visualization and self-reported aninsight they had. The main purpose of this final case study is to highlighthow the self-reported insight relates to our model’s summary of session.
Fig. 7. Average accuracy of predicting the next hover in the set oftop- k candidates for boardrooms dataset. There are 39 sessions, andprediction starts after the third click for consistency with the baseline fromOttley et al. [33]. The error bars indicate the standard error of accuracyacross all sessions. There are 500 data points in the visualization,making k = to be 10% of the dataset. Our technique outperforms orperforms within margin of error in comparison to the baseline.Fig. 8. The experiment interface from Kern et al. [25]. Crime cases werevisualized on an interactive map, where the colors indicate the type ofcrimes. Users were able to zoom, pan, hover, and click. While hoveringon dots, a tooltip with more details opens. When a dot is clicked, thecrime case is added to the sidebar as shown above. et al. experimental setup In this crowdsourced experiment, Kern et al. [25] propose an alternativeexperimental design in which participants freely interact with a mapvisualization of crimes in St. Louis and report their insights. Figure 8shows the visual interface for this experiment, hovers trigger a tooltipwith more details and clicks add the data point to the side bar. In thispaper, we used clicks from one session that had ≥
10 click events. Be-fore the participants start the experiment, they are asked to familiarizethemselves with the interface and functionalities. We use recordedclicks from this experiment to study how our model’s understandingtowards the user compares with the self-reported insight.
After observing six clicks in this session, our model posterior inferredthe user is exploring based on only location (geo-based) with probabil-ity > .
8. By the time eight clicks were observed, our model was morecertain on this task being geo-based with probability > .
95. Noticethat this finding is trivially justified in Figure 9, where the distributionof type attribute in the interaction set is very similar to the full data dis-tribution (hence no bias towards type), but the distribution of location attribute in the interaction set is narrow compared to the full dataset.Our model’s understanding of the user reflects their self-reported in-sight: “Two cars were stolen in the same week from the 3900 block ofMiami St.”
We hypothesize that the absence of bias towards type is dueto the user looking for an insight among different type s of crimes in thesession. ig. 9. The summary of a session in which the user is freely inter-acting with a map visualization to report their insights. The left-mostgraph shows the distribution of the full crime dataset presented to theparticipant. As clicks (denoted by (cid:155) ) arrive, our models get updatedto represent user interactions. Notice that type model at time t = does not significantly differ from the type distribution in the full data set,resulting in no detection of exploration bias towards type . ISCUSSION
The novelty of our technique is that we encode different explorationpatterns as models and use the Bayesian model selection to maintaina belief over all possible exploration patterns. To demonstrate howa set of competing models can uncover user exploration patterns, wedesigned our model space so that each model represents informationrelevance based on a subset of dimensions in data.One of the immediate insights from our results is that posing userexploration modeling as a Bayesian model selection problem results inbetter performance in exploration bias detection and next interactionprediction in most cases. In particular, our choice of likelihood functionfrom Section 4.2 resulted in improved performance for sessions inwhich the users were asked to explore a specific subset of the data(Figures 3, 4, and 5). In the case of bias detection for type-based tasks(Fig. 3, middle), the baseline outperformed our technique. After furtherinvestigation, we concluded that this behavior was due to the Wall et al.[41] baseline taking into account the overall data distribution whereasour likelihood function does not take into account the overall datadistribution. An alternative choice of likelihood function that takes theunderlying data distributions into account may result in improvements.For open-ended tasks, we believe there could be more expressivechoices of models to learn multi-modal distributions or more complexdecision boundaries. Although our technique outperformed the existingbaseline for open-ended tasks (Figure 7), we believe a more expressivechoice of models and a more specific user study to prevent users fromunintentional hovers would be beneficial. Another limitation of ourwork is that interactions that manipulate the underlying dataset arenot supported as we assume our dataset D is constant throughoutthe session. In other words, we could consider interactions such asdrag/drop to learn which subset of the data one may interact with,however, further investigation is needed to build models that learnabout the context of re-positioning and make more complex predictions(such as where should a given point move given past movements).The technique presented in this paper takes the visualization com-munity a step closer to modeling users in scenarios where there aremultiple possible exploration strategies, analysis goals, or personalpreferences. By representing each possibility as a model and updatingthe belief as evidence is observed (through interactions), we can enableintelligent machine response in visualization systems. However, furtherwork is required to address some of the shortcomings of our study. We have identified three main areas for further exploration, which wediscuss in the next section. UTURE D IRECTIONS
Framing user exploration modeling as a Bayesian model selectionproblem opens a new set of research opportunities. In particular, wecan utilize recent advancements in Bayesian machine learning (i.e.active learning, active search, etc.) to further improve the collaborationbetween human and computer in visual analytics systems. In thissection, we highlight four specific areas for future investigations.
Different Model Spaces
Visual analytics research is constantlyevolving to study new human-related factors that impact how we useinteractive visualization systems. This evolution in user modelingalongside our competing models framework provide an opportunity toinfer which human-related factor is impacting an individual session themost and move towards more individualized visual analytics systems.
Sampling Methods for More Efficiency
Throughout this work,we compute exact values of model posteriors. While this was notprohibitive for our largest model space (2 models in Section 7.2, wherethere were 7 attributes), we recognize computing exact posteriors is notalways feasible. This could be due to the unavailability of posteriorpredictive distributions in closed form or the need for low latency inreal-time systems. In such scenarios, sampling methods may be usedto approximate the posterior distribution. Active Learning to Mitigate Information Bubble
Active learn-ing is the idea behind an algorithm that queries an oracle in order tolearn. In the scope of our work, this concept can be utilized to confirmdetected biases with the user in order to avoid unintentional biases(through exploration ) and mitigate information overload by focusingon relevant information (through exploitation ). The mechanics of ef-fectively querying users and updating the visualization view are alsoleft for future studies. This line of research can lead to building au-tomatic filtering features to mitigate information bubbles created byunintentional bias.
10 C
ONCLUSION
We began this paper by claiming that interactive visualizations alonecannot fully support humans in analyzing large datasets and makinginformed decisions. There are numerous human factors such as limitedcognitive capacity and personal biases that have roles in hinderingeffective analysis of data. As the visual analytics research commu-nity investigates more about human factors related to data analysis,we need to build intelligent visual analytic systems who understandusers through the lens of their low-level interactions and ultimatelyimprove the visual analytic experience. In order to enable machineteammates to make high-level inferences by observing low-level inter-actions, we proposed competing models : a technique which enumeratesa set of human-related possibilities as models and then utilizes iterativeBayesian updating to learn about users as interactions are observed.To narrow our focus, we outlined the process of creating and main-taining a set of competing models, and described how visual analyticsresearchers can use this idea to gain a deeper insight into user explo-ration. In the process of validating our technique, we demonstratedthat by building models on the underlying dataset in visualizationsand then updating said models as interactions are observed, we canuncover exploration biases and predict future interaction. In particular,we reached high rates of accuracy for next interaction prediction whenthe participants were asked to perform specific tasks. In open ended sce-narios, we saw a lower overall performance while still outperformingthe baseline. This observation calls for more investigation on modelingopen-ended sessions, which is inherently a more difficult objective. A CKNOWLEDGMENTS
The authors wish to thank Sunwoo Ha for assisting in data preparation.Lane Harrison, Mi Feng, and Adam Kern for sharing their data. EmilyWall for her conversation on the bias detection metric. This material isbased upon work supported by the National Science Foundation undergrant numbers 1755734, 1845434, and 1940224.
EFERENCES [1] L. Battle, R. Chang, and M. Stonebraker. Dynamic prefetching of datatiles for interactive visualization. In
Proceedings of the 2016 InternationalConference on Management of Data , pages 1363–1375. ACM, 2016.[2] C. M. Bishop.
Pattern Recognition and Machine Learning . Springer,2006.[3] E. T. Brown, J. Liu, C. E. Brodley, and R. Chang. Dis-function: Learningdistance functions interactively. In , pages 83–92. IEEE, 2012.[4] ´A. A. Cabrera, W. Epperson, F. Hohman, M. Kahng, J. Morgenstern, andD. H. Chau. Fairvis: Visual analytics for discovering intersectional bias inmachine learning. In , pages 46–56. IEEE, 2019.[5] S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, andH. T. Vo. Vistrails: visualization meets data management. In
Proceedingsof the 2006 ACM SIGMOD international conference on Management ofdata , pages 745–747. ACM, 2006.[6] Y. Chen, S. Barlowe, and J. Yang. Click2annotate: Automated insightexternalization with rich semantics. In , pages 155–162. IEEE, 2010.[7] F. Dabek and J. J. Caban. A grammar-based approach for modeling userinteractions and generating suggestions during the data exploration process.
IEEE transactions on visualization and computer graphics , 23(1):41–50,2016.[8] N. Del Rio and P. P. Da Silva. Probe-it! visualization support for prove-nance. In
International Symposium on Visual Computing , pages 732–741.Springer, 2007.[9] M. Derthick and S. F. Roth. Enhancing data exploration with a branchinghistory of user operations.
Knowledge-Based Systems , 14(1-2):65–74,2001.[10] E. Dimara, A. Bezerianos, and P. Dragicevic. The attraction effect in in-formation visualization.
IEEE transactions on visualization and computergraphics , 23(1):471–480, 2016.[11] J. Doboˇs, N. J. Mitra, and A. Steed. 3d timeline: Reverse engineering of apart-based provenance from consecutive 3d models.
Computer GraphicsForum , 33(2):135–144, 2014.[12] W. Dou, D. H. Jeong, F. Stukes, W. Ribarsky, H. R. Lipford, and R. Chang.Recovering reasoning processes from user interactions.
IEEE ComputerGraphics and Applications , 29(3):52–61, 2009.[13] C. Dunne, N. Henry Riche, B. Lee, R. Metoyer, and G. Robertson. Graph-trail: Analyzing large multivariate, heterogeneous networks while sup-porting exploration history. In
Proceedings of the SIGCHI conference onhuman factors in computing systems , pages 1663–1672. ACM, 2012.[14] T. Ellkvist, D. Koop, E. W. Anderson, J. Freire, and C. Silva. Using prove-nance to support real-time collaborative design of workflows. In
Interna-tional Provenance and Annotation Workshop , pages 266–279. Springer,2008.[15] M. Feng, E. Peck, and L. Harrison. Patterns and pace: Quantifying diverseexploration behavior with visualizations on the web.
IEEE transactionson visualization and computer graphics , 25(1):501–511, 2018.[16] D. Gotz and M. X. Zhou. Characterizing users’ visual analytic activity forinsight provenance.
Information Visualization , 8(1):42–55, 2009.[17] S. Gratzl, A. Lex, N. Gehlenborg, N. Cosgrove, and M. Streit. From visualexploration to storytelling and back again.
Computer Graphics Forum ,35(3):491–500, 2016.[18] D. P. Groth and K. Streefkerk. Provenance and annotation for visualexploration systems.
IEEE transactions on visualization and computergraphics , 12(6):1500–1510, 2006.[19] G. S. Halford, R. Baker, J. E. McCredden, and J. D. Bain. How manyvariables can humans process?
Psychological science , 16(1):70–76, 2005.[20] J. Heer, J. Mackinlay, C. Stolte, and M. Agrawala. Graphical histories forvisualization: Supporting analysis, communication, and evaluation.
IEEEtransactions on visualization and computer graphics , 14(6):1189–1196,2008.[21] J. Heer and B. Shneiderman. Interactive dynamics for visual analysis.
Queue , 10(2):30, 2012.[22] Z. Hensley, J. Sanyal, and J. New. Provenance in sensor data management:A cohesive, independent solution.
Communications of the ACM , 57(2),2014.[23] T. Iwata, N. Houlsby, and Z. Ghahramani. Active learning for interactivevisualization. In
Artificial Intelligence and Statistics , pages 342–350,2013. [24] N. Kadivar, V. Chen, D. Dunsmuir, E. Lee, C. Qian, J. Dill, C. Shaw,and R. Woodbury. Capturing and supporting the analysis process. In , pages131–138. Ieee, 2009.[25] A. Kern. The effects of mixed-initiative visualization systems onexploratory data analysis. https://openscholarship.wustl.edu/eng_etds/523/ , 2020.[26] D. Kurlander and S. Feiner. Editable graphical histories. In VL , pages127–134. Citeseer, 1988.[27] R. Lightner and T. Francis. Inside america’s boardrooms.http://graphics.wsj.com/boards-of-directors-at-SP-500-companies/, 2016.[28] H. R. Lipford, F. Stukes, W. Dou, M. E. Hawkins, and R. Chang. Helpingusers recall their reasoning process. In , pages 187–194. IEEE, 2010.[29] E. Maguire, P. Rocca-Serra, S.-A. Sansone, J. Davies, and M. Chen.Taxonomy-based glyph design—with a case study on visualizing work-flows of biological experiments. IEEE Transactions on Visualization andComputer Graphics , 18(12):2603–2612, 2012.[30] N. Mahyar and M. Tory. Supporting communication and coordinationin collaborative sensemaking.
IEEE transactions on visualization andcomputer graphics , 20(12):1633–1642, 2014.[31] K. P. Murphy. Conjugate Bayesian analysis of the Gaussian distribution.
Technical Report , 2007.[32] C. North, R. Chang, A. Endert, W. Dou, R. May, B. Pike, and G. Fink.Analytic provenance: process+ interaction+ insight. In
CHI’11 ExtendedAbstracts on Human Factors in Computing Systems , pages 33–36. ACM,2011.[33] A. Ottley, R. Garnett, and R. Wan. Follow the clicks: Learning and antic-ipating mouse interactions during exploratory data analysis.
ComputerGraphics Forum , 38(3):41–52, 2019.[34] S. G. Parker and C. R. Johnson. Scirun: a scientific programming environ-ment for computational steering. In
Supercomputing’95: Proceedings ofthe 1995 ACM/IEEE Conference on Supercomputing , pages 52–52. IEEE,1995.[35] E. D. Ragan, A. Endert, J. Sanyal, and J. Chen. Characterizing prove-nance in visualization and data analysis: an organizational framework ofprovenance types and purposes.
IEEE transactions on visualization andcomputer graphics , 22(1):31–40, 2015.[36] W. Ribarsky and B. Fisher. The human-computer system: Towards anoperational model for problem solving. In , pages 1446–1455. IEEE, 2016.[37] A. Sarvghad, M. Tory, and N. Mahyar. Visualizing dimension coverageto support exploratory analysis.
IEEE transactions on visualization andcomputer graphics , 23(1):21–30, 2016.[38] Y. B. Shrinivasan and J. J. van Wijk. Supporting the analytical reasoningprocess in information visualization. In
Proceedings of the SIGCHI con-ference on human factors in computing systems , pages 1237–1246. ACM,2008.[39] Y. L. Simmhan, B. Plale, D. Gannon, and S. Marru. Performance evalua-tion of the karma provenance framework for scientific workflows. In
Inter-national Provenance and Annotation Workshop , pages 222–236. Springer,2006.[40] S. Tu. The Dirichlet-Multinomial and Dirichlet-Categorical models forBayesian inference.
Computer Science Division, UC Berkeley , 2014.[41] E. Wall, L. M. Blaha, L. Franklin, and A. Endert. Warning, bias mayoccur: A proposed approach to detecting cognitive bias in interactivevisual analytics. In , pages 104–115. IEEE, 2017.[42] E. Wall, J. Stasko, and A. Endert. Toward a design space for mitigatingcognitive bias in vis. In , pages111–115. IEEE, 2019.[43] K. Xu, A. Ottley, C. Walchshofer, M. Streit, R. Chang, and J. Wenskovitch.Survey on the analysis of user interactions and visualization provenance. , 2020.[44] S. Xu, C. Bryan, J. K. Li, J. Zhao, and K.-L. Ma. Chart constellations:Effective chart summarization for collaborative and multi-user analyses.