A Visual Analytics Framework for Contrastive Network Analysis
TTo appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020
A Visual Analytics Framework for Contrastive Network Analysis
Takanori Fujiwara * , Jian Zhao † , Francine Chen ‡ Kwan-Liu Ma § , University of California, Davis, University of Waterloo A BSTRACT
A common network analysis task is comparison of two networks toidentify unique characteristics in one network with respect to theother. For example, when comparing protein interaction networks de-rived from normal and cancer tissues, one essential task is to discoverprotein-protein interactions unique to cancer tissues. However, thistask is challenging when the networks contain complex structural(and semantic) relations. To address this problem, we design Con-traNA, a visual analytics framework leveraging both the power ofmachine learning for uncovering unique characteristics in networksand also the effectiveness of visualization for understanding suchuniqueness. The basis of ContraNA is cNRL, which integrates twomachine learning schemes, network representation learning (NRL)and contrastive learning (CL), to generate a low-dimensional embed-ding that reveals the uniqueness of one network when compared toanother. ContraNA provides an interactive visualization interface tohelp analyze the uniqueness by relating embedding results and net-work structures as well as explaining the learned features by cNRL.We demonstrate the usefulness of ContraNA with two case studiesusing real-world datasets. We also evaluate ContraNA through acontrolled user study with 12 participants on network comparisontasks. The results show that participants were able to both effectivelyidentify unique characteristics from complex networks and interpretthe results obtained from cNRL.
Keywords:
Contrastive learning, network representation learning,interpretability, network comparison, visual analytics.
NTRODUCTION
A network is a common form for modeling various types of relation-ships in real-world applications, such as social connections [14, 22],biological interactions [18, 39], and supercomputer communica-tions [15]. In practice, comparative analysis of two networks is vi-tal [28, 85], especially for the identification of the uniqueness of onenetwork compared to another. We call this task contrastive networkanalysis . For example, when studying the effect of Alzheimer’s dis-ease on a human brain [37], neuroscientists want to find unique func-tional connections in the brain network of a patient with Alzheimer’sdisease by comparing to that of a healthy subject. Also, for re-searcher collaborations in different disciplines [62], analysts in afunding agency may want to reveal unique ways of collaboration inthe disciplines for decision making.Despite the demands for network comparison, there is littleadequate visual analytics support. Most of the existing methods(e.g., [4, 55, 81]) presuppose the existence of node-correspondence(i.e., pairwise correspondence between nodes in two different net-works) [85]. This is a critical limitation since we usually do notknow such information in advance when the networks are collectedfrom different resources. One potential solution is identifying thenode-correspondence by using network alignment (or graph match-ing) [28, 85]. However, these algorithms notoriously have highcomputational costs [28, 85], and thus are only suitable for treatingsmall networks (e.g., 100 nodes). Also, there may not exist a clearcorrespondence between nodes. * e-mail: [email protected] † e-mail: [email protected] ‡ e-mail: [email protected] § e-mail: [email protected] Another approach for visual comparison of networks is basedon statistical measures (e.g., network density) [31], centralities(e.g., degree centrality) [92], graphlets [61], or a combination ofthese [89]. For example, with graphlets [74] (small, connected, andnon-isomorphic subgraph patterns in a network), the similaritiesof two networks can be measured by comparing the frequency ofappearance of each graphlet in each network [61]. While these ap-proaches can provide a (dis)similarity between different networks,they compare networks only based on simple measures, which areoften insufficient. Also, they only provide network-level similarities,and thus cannot compare networks at more detailed levels (e.g., anode-level). Without a detailed-level comparison, it is difficult tofind which part of a network relates to its uniqueness.To address the above problems, we introduce a novel visual analyt-ics framework,
ContraNA , for comparative network analysis, whichintegrates contrastive network representation learning (cNRL) [36]into interactive visualization. Empowered by cNRL, our frameworkallows for discovering unique characteristics of one network bycontrasting with another in a comprehensive (i.e., using multiple ad-vanced measures) and detailed (i.e., analyzing a node or subnetworklevel) manner without node-correspondence information. Specifi-cally, we employ an interpretable version of cNRL (i-cNRL) [36]to provide human-understandable explanations of discovered char-acteristics that are further revealed by novel visual representations.We enhance i-cNRL by designing an interactive visual interfacethat allows analysts to integrate their domain knowledge into theautomated analysis. Particularly, we introduce a method to visuallyidentify the uniqueness in one network based on the i-cNRL result,a visual summary to intuitively inform network features that highlycontribute to the result, and interactive linkings with the existingnetwork visualizations to explain and refine the result.In summary, our main contributions include:• A cNRL-based visual analytics framework, ContraNA, whichaims to support a new network analysis approach, named con-trastive network analysis, to effectively reveal unique charac-teristics in one network relative to another.• Enhancements of i-cNRL with a visual interface that providesfour major abilities—
DIIF : (1) D iscovery of uniqueness innetworks, (2) I nterpretability of features generated by i-cNRL,(3) I ntuitive analysis with common visualizations, and (4) F lexibility of adjusting i-cNRL based on analysts’ interests.• Two case studies and a controlled user study with multiple real-world datasets, which assess the effectiveness and usefulnessof ContraNA for contrastive network analysis. ACKGROUND AND R ELATED W ORK
In this section, we first describe network representation learningand contrastive learning—two machine learning schemes used inContraNA. Then, we review the relevant works for visual networkcomparison.
Network representation learning (NRL) [17, 95], also known asgraph embedding, aims to learn low-dimensional latent vectors thatrepresent a network while maximally preserving certain network in-formation, such as the structural and semantic characteristics. Oncea low-dimensional representation obtained, we can easily and ef-ficiently conduct network analysis tasks (e.g., node classification1 a r X i v : . [ c s . S I] A ug o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 and link prediction). Typical NRL methods include node2vec [44],DeepGL [77], and some other deep neural networks [96]. Morecomprehensive descriptions of NRL methods have been included inseveral recent surveys [17, 95]. Contrastive learning (CL) [99] focuses on finding patterns thatare more salient in one dataset relative to another [2]. This is unlikediscriminant analysis (e.g., linear discriminant analysis [52]), whichaims to discriminate data points based on their classes. Several CLmethods have been developed in the machine learning community,such as contrastive versions of latent Dirichlet allocation [99], hid-den Markov models [99], and regressions [38]. CL methods forrepresentation learning have been also introduced [2, 3, 27, 38, 80],such as contrastive PCA (cPCA) [2, 38] and contrastive variationalautoencoder (cVAE) [3, 80].Recently, contrastive network representation learning(cNRL) [36] integrates the above two machine learning schemesto achieve comparative network analysis. It uses NRL to generatetwo sets of latent vectors for two networks and then employsCL to perform comparative analysis based on the vectors. Thisapproach embeds network nodes into a low-dimensional space thatreveals the uniqueness of one network compared to another. Tooffer interpretability, ContraNA uses a specific cNRL method (seeSect. 3) and further provides novel visual analysis capabilities toenable effective network comparison.
There exist three general approaches in visual comparison: jux-taposition, superposition, and explicit encoding [41]. Through acomprehensive survey, Gleicher [40] provided a framework of con-siderations for visual comparison, such as tasks, challenges, strate-gies, and designs. Here we review the relevant works in visualnetwork comparison.
Comparing multiple static networks has been a classic problem invisualization research. Alper et al. [4] presented several superpo-sition designs for node-link and adjacency matrix visualizations tosupport weighted network comparison. TileMatrix [68] uses juxta-position to place the triangular adjacency matrices of two networksonto upper and lower areas of a square matrix. On the other hand,John et al. [53] juxtaposed each pair of weighted links in a matrixcell. MatrixWave further extended this approach to support thecomparison of multi-layer networks [98]Researchers have focused on developing techniques for compar-ing brain networks due to its special characteristics (e.g., very dense)and importance. Shi et al. [81] opted to visualize links that are signif-icantly different between two brain networks. Yang et al. [91] useda clustering algorithm with NodeTrix [50], a hybrid of node-linkand adjacency matrix representations. Fujiwara et al. [32] enabledthe comparison of a larger number of brain networks by providingan overview with dimensionality reduction. Some other domainshave been addressed as well, such as genome interaction [55] andegocentric networks [65].All the above methods require the information of exact node-correspondence [28], unlike ContraNA. While a few works [6,59,67]applied network alignment [28] to find node-correspondence beforevisualization, they do not scale well due to the computation cost.
Dynamic networks contain nodes and/or links changing over time.A comprehensive survey is provided by Beck et al. [11]. Here, wefocus on the comparison of networks at different timestamps.One approach is based on the juxtaposition of networks at differ-ent timestamps. Federico et al. [29] applied a 2D network layoutthat produces stable node positions across time and then juxtaposednetworks at multiple time points in a 2.5D view. On the other hand, TimeArcs [25] lays out a network at each time point in 1D, and usesan arc diagram to display links. A wall-size display was used tojuxtapose an array of networks [63]. Moreover, animated transitionshave been employed, which can be viewed as juxtaposition in thetemporal domain, e.g., GraphDiaries [8] and DiffAni [78].Moreover, several works summarize a dynamic network based onthe similarity of the network at each timestamp. For example, SmallMultiPiles [7] groups similar weighted adjacency matrices acrossconsecutive time points and then shows a representative matrix foreach group. EgoLines [97] effectively visualizes a k -hop dynamicegocentric network with a “subway map” metaphor. van den Elzenet al. [88] utilized dimensionality reduction to overview the sim-ilarities of networks across time. A similar approach was usedto visualize dynamic brain networks [9] and compare dominancevariation in animal groups [19].Lately, researchers have started to utilize time-series or topo-logical analysis to summarize or identify important trends in a dy-namic network. Examples include using graph wavelet transform toclassify nodes [24] and persistent homology to capture topologicalchanges [45]. Fujiwara et al. [35] applied change point detection [5]to segment a dynamic network and generate summaries. Severalworks extended this approach in other cases, such as visualizingstreaming networks [56, 72].Again, the above methods still require the information of node-correspondence. To overcome this limitation, we utilize NRL tocapture the network’s topological and semantic features. Several systems were developed to support network compar-ison without the limitation of knowing node-correspondence.ManyNets [31] uses a tabular interface to list several basic networkstatistics (e.g., degree centrality) for each network. von Landesbergeret al. [89] used graphlet frequencies and other network-statistics mea-sures and to generate a self-organization map for arranging networkson a 2D grid. A similar approach was used by Harrigan et al. [47]to visualize egocentric networks, and by Kwon et al. [61] to showsimilar networks given an input network. In addition to node-levelfeatures, Gove [43] suggested network-level features (e.g., density)that are easier to interpret and faster to compute. Along this line,Graph Thumbnails [92] uses the k -core number in a nested circlepacking representation of networks.While the above methods can be used for comparing networkswithout node-correspondence, they lack the ability to compare net-works from multiple levels. ContraNA addresses this by employingthe state-of-the-art NRL method, allowing for comparison at bothnode and subgraph levels. Further, by leveraging CL, we focus onrevealing the uniqueness in one network relative to another, which isdifferent from the purpose of the above works (i.e., identifying simi-larities of networks). To the best of our knowledge, the only workusing CL for visual analytics [34] focuses on high-dimensional data.However, ContraNA focuses on comparative analysis of networks.In sum, the existing methods have limited flexibility in use due tothe requirement of node-correspondence or to insufficient analysisability due to the absence of multiple level comparison. ContraNAaddresses these issues by utilizing cNRL, which we describe inSect. 3. Then, with interactive visualizations, ContraNA furthersupplements cNRL’s limitations that are identified in Sect. 4. ONTRASTIVE N ETWORK R EPRESENTATION L EARNING
Here, we provide a brief introduction to the core analysis methodused in ContraNA: contrastive network representation learning(cNRL) [36].
Fig. 1 shows an overview of the cNRL architecture. Given twonetworks, a target network G T and a background network G B , the2 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 Figure 1: General architecture for cNRL. objective of cNRL is learning a contrastive representation Y T thatreveals unique characteristics in G T relative to G B . To achieve this,cNRL employs a two-step embedding process: (1) NRL, whichobtains network features of G T and G B , and (2) CL, which generatesa contrastive representation from the network features.The input networks G T and G B can be any combination of undi-rected or directed, unweighted or weighted, and non-attributed orattributed networks. In their adjacency matrices A T and A B , thenumbers of G T and G B nodes, n T and n B , do not have to be the same.Similarly, when G T and G B are attributed, the numbers of attributes m T and m B in matrices of node attributes P T and P B may be differ-ent. The first embedding with NRL produces target and backgroundnetworks’ feature matrices X T and X B , where both X T and X B needto have the same d features. Using NRL, cNRL preserves the targetand background network information in X T and X B with explicitand comprehensive network features. Based on X T and X B , thesecond embedding using CL generates a projection matrix W of d rows and d (cid:48) columns ( d (cid:48) ≤ d ) and then G T and G B ’s contrastiverepresentations Y T and Y B can be produced by multiplying X T and X B with W , respectively. Through CL, the contrastive representation Y T captures relationships (e.g., the network structural differencesamong network nodes) that appear in G T but do not appear in G B .The cNRL architecture provides flexibility in selection of meth-ods for both NRL and CL. For NRL, we can choose any algorithmthat can produce the same features across networks, such as in-ductive NRL methods (e.g., GraphSAGE [46] and DeepGL [77]),which learn transferable knowledge from a training network to othernetworks. For CL, we can choose any method designed for repre-sentation learning. See the work about cNRL [36] for further detailsabout available algorithm options for NRL and CL.
In ContraNA, to interactively examine the identified unique charac-teristics with human-understandable explanations, we specificallyemploy an interpretable version of cNRL, called i-cNRL , whereDeepGL [77] and cPCA [2] are used as the NRL and CL meth-ods, respectively. Providing interpretability is the core feature ofContraNA as it helps the analysts understand the meaning of uniquecharacteristics found in one network and the reason why such unique-ness can be seen in only that network.
NRL with DeepGL.
Using DeepGL [77] as NRL, i-cNRL gener-ates feature matrices X T and X B with interpretable network features.The features consist of the base feature x and relational function f .A base feature x is a measure we can obtain for each node, such as in-, out-degree, degeneracy ( k -core numbers ) [73], PageRank [73],or an attribute (e.g., gender of a node in a social network).A relational function f is a combination of relational featureoperators (RFOs), each of which summarizes base feature valuesof one-hop neighbors of a node. For example, the operator canbe a computation of the mean, sum, maximum base feature val-ues of one-hop neighbors of a node. Also, the neighbors can beeither in-, out-, total-neighbors . Together with the summary mea-sure S , the operators are denoted Φ − S , Φ + S , and Φ S , respectively.For example, Φ − mean ( x ) computes the mean x of the in-neighborsof a node. Moreover, the RFO can be applied repeatedly. For ex- ample, f = ( Φ + mean ◦ Φ − max )( x ) first computes the maximum x ofin-neighbors for each out-neighbor of a node and then produces themean of these maximum values.During the learning process, from the user-input base features,RFOs, and the maximum number of hops to be considered, DeepGLevaluates combinations of these inputs and parameters and automati-cally selects important network features for preserving the topologi-cal (and semantic) information. For example, when using in-degree and out-degree as base features, Φ mean and Φ sum as RFOs, and 2 asthe maximum number of hops, DeepGL may generate the networkfeatures { in-degree, out-degree, Φ mean ( in-degree ), Φ sum ( in-degree ), Φ mean ◦ Φ mean ( in-degree ) } . CL with cPCA.
From the target and background feature matrices X T and X B , cPCA [2] produces contrastive principal components(cPCs) , which are analogous to principal components (PCs) in ordi-nary PCA [54]. cPCs are low-dimensional representative directionswhere X T has high variance but X B has low variance. That is, Y T ,an embedding of X T with cPCs, depicts unique characteristics (withthe consideration of variance) of a target network G T relative to abackground network G B .cPCA requires one hyperparameter α ( ≤ α ≤ ∞ ) , called a contrast parameter . The contrast parameter α controls the trade-offbetween having high target variance and low background variancein cPCs. When α =
0, cPCs only maximize the variance of X T , thesame as those in classical PCA. As α increases, cPCs place greateremphasis on directions that reduce the variance of X B . Because α has a strong impact on the result, researchers have developedsemi-automatic [2] and automatic [36] selection of α .Similar to PCs in PCA, cPCs are represented as linear transformsof d features of X T and X B . Analogous to PC loadings, cPCAprovides cPC loadings [34], which indicate how strongly each of the d input features contributes to the corresponding cPC. By examininga list of d learned features via NRL and cPC loadings, we canunderstand the relationships between the d features and cPCs; andwe can also interpret the contrastive representation Y T . ESIGN C ONSIDERATIONS
The aforementioned i-cNRL can generate a contrastive representa-tion which highlights the uniqueness of a target network. However,to thoroughly understand the uniqueness, we opt to empower theautomated analysis with interactive visualization, which can tightlyintegrate the knowledge and adaptability of human experts with thestatistical learning of machines [79, 84]. We comprehensively iden-tify a set of limitations to i-cNRL for contrastive network analysisin depth, which leads to the following design considerations forour visual analytics framework, ContraNA. In general, we aim toamplify the D iscovery, I nterpretability, I ntuitiveness, and F lexibility( DIIF ) in visual contrastive network analysis.
DC1:
Support the discovery of whether a target network isunique compared to a background network, and which part of thenetwork relates to the uniqueness.
The uniqueness of a target datasetrelative to the base is embedded in the contrastive representation Y T generated by CL-based representation learning methods, includingcNRL. Many previous works attempted to display this data to revealthe uniqueness [2, 3, 27]. However, because Y T only contains theinformation of the target network G T , reviewing only Y T is notsufficient to understand how well the CL method finds uniqueness.Also, it is difficult to identify which data points (i.e., network nodesin our case) highly relate to the found uniqueness. The visual an-alytics framework should support discovering the uniqueness andthe associated nodes by presenting the information in both the targetand background networks. DC2:
Enhance the interpretability of the features learned byNRL and the cPCs generated by CL.
Investigating the relationshipsamong the network features, cPCs, and the representation Y T is im-portant to interpret the uniqueness of G T . While i-cNRL is designed3 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 Figure 2: The analyst is using ContraNA to conduct a contrastive analysis of the Dolphin social network [69] (the target network) and theZacharys karate club network [94] (the background network). (a) A contrastive representation view shows contrastive representations of targetand background networks. (b) A feature contribution view visualizes network features generated by DeepGL and their contributions to each cPC(i.e., scaled cPC loadings). (c) A probability distribution view depicts target and background networks’ probability distributions of the selectednetwork feature in (b). (d)(e) A network layout view draws laid-out target and background networks, respectively. (f) The analyst can changeseveral settings of the algorithm and visualizations from the drop-down menu.Figure 3: Contrastive network analysis workflow with ContraNA. to provide interpretable network features and cPCs, understandingthem from i-cNRL’s direct outputs is not straightforward. For exam-ple, DeepGL could generate a sophisticated relational function suchas ( Φ + sum ◦ Φ max ◦ Φ − mean )( x ) . Moreover, examining cPC loadingsfor each feature would be time-consuming when DeepGL producesmany network features. The framework should provide visualiza-tions to facilitate easy understanding of the above information. DC3:
Offer intuitiveness in understanding a target network’suniqueness by relating it to common network visualizations.
Thecontrastive representation Y T generated could contain complicatedpatterns that are difficult to understand. Thus, it is not intuitiveenough to just view these patterns directly based on the i-cNRLresults in the embedding space. To help analyze such patterns, theframework should provide links between the results of i-cNRL andcommonly used visualizations for network analysis, such as laid-outnetworks and probability distributions of network centralities. DC4:
Provide the flexibility to interactively adjust the i-cNRLparameters to generate results based on the analysts’ interest.
Theresults of i-cNRL heavily depend on the parameters used for eachembedding step. For example, changing a value of the contrastparameter α might reveal different unique characteristics in G T . Foranalysts with advanced knowledge on NRL and CL, the frameworkshould provide abilities for interactively tuning the i-cNRL resultsbased on their needs. RAMEWORK O VERVIEW
Grounded by the
DIIF design considerations, we develop ContraNAwhich augments the back-end i-cNRL algorithm with interactivevisualization (Fig. 2), supporting visual contrastive network analysis.Fig. 3 shows a workflow of conducting contrastive network anal-ysis with ContraNA. The workflow starts from (A) generation of thei-cNRL results that includes NRL with DeepGL and CL with cPCA(Fig. 1). Afterward, the analyst can first (B) identify whether or notthere are any unique characteristics only found in a target networkfrom the contrastive representations visualized by ContraNA (Fig. 2-a). If such characteristics exist, to understand the uniqueness, theanalyst can (C) interpret the network features and cPCs generatedby i-cNRL with visualizations in Fig. 2-b. They can also (D) ana-lyze the contrastive representations, network features, and cPCs byrelating them with probability distributions (Fig. 2-c) and laid-outnetworks (Fig. 2-d, e). Based on findings during the exploration, theanalyst might want to adjust the parameters of i-cNRL.The above procedure is our expected main analysis workflow asindicated by the thick blue arrows in Fig. 3. However, the ContraNAUI provides the flexibility in the analysis activities, shown by thesolid gray arrows in Fig. 3. For example, the analyst might want tostart to (D) see laid-out networks in order to grasp the topologicaldifferences between target and background networks at a glance, andthen (B) examine the differences with the contrastive representations.Also, such an interactive analysis often requires to go back and forthbetween different views to validate findings obtained in one view.Due to the high computational cost of NRL with DeepGL (e.g.,20 seconds for a network of 6K nodes and 20K links), we decided tosupport the interactive parameter adjustment only for cPCA. Afterthe analyst updates DeepGL’s parameters and generates the networkfeatures, they can analyze the results with the ContraNA UI.We have developed ContraNA as a web application. For theback-end algorithms, we use Python to integrate the existing i-cNRLimplementation [36]. The front-end UI is implemented with a com-4 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 (a) G T : LC-multiple [75,93], G B : Combined-AP/MS [21,93] (b) G T : Dolphin [69], G B : Karate [94] Figure 4: Two-dimensional projections based on (contrastive) repre-sentations obtained with PCA (when α = ) and cPCA. bination of HTML5, JavaScript, D3 [16], and WebGL. D3 is used forthe feature contribution and probability distribution views (Fig. 2-b,c). For the other views (Fig. 2-a, d, e), we utilize WebGL to supportefficient rendering and interaction as networks often consist of manynodes and links (e.g., several thousand nodes). We use WebSocketto communicate between the front- and back-end modules. ONTRA
NA V
ISUAL I NTERFACE
As shown in Fig. 2, the ContraNA UI consists of four interactivelycoordinated views, including a contrastive representation view, a fea-ture contribution view, a probability distribution view, and a networklayout view, designed with the considerations in Sect. 4. Here, wedescribe the views provided by the UI through contrastive networkanalysis of two social networks, the Dolphin social network [69] as G T and Zachary’s karate club network [94] as G B . A demonstrationvideo of the interface is available at our online site [1]. With the results generated by i-cNRL, the first step of our anal-ysis workflow (Fig. 3-A), ContraNA’s contrastive representationview (Fig. 2-a) visualizes the results to reveal whether or not thereis uniqueness in the target network compared to the backgroundnetwork, serving as the following step (Fig. 3-B,
DC1-Discovery ). Visual Identification of Target Network’s Uniqueness.
Simi-lar to existing works [2, 3, 27, 80], a potential solution is comparingthe results of ordinary PCA and cPCA. For example, given the twoprotein interaction networks, LC-multiple [75, 93] and Combined-AP/MS [21, 93], Fig. 4-a1, a2 show contrastive representations Y T generated with i-cNRL using the contrastive parameter α = α =
138 (cPCA), respectively. In Fig. 4-a2, comparing withFig. 4-a1, we can see the emergence of a new cluster, as annotatedwith the red rectangle. It indicates that cPCA successfully findsdirections (i.e., cPCs) where G T has a higher variance than G B (i.e.,the uniqueness). However, in many cases, it is difficult to see clearpattern differences between the results of PCA and cPCA, as shownin Fig. 4-b1, b2 with the networks of dolphins [69] as G T and Karateclub members [94] as G B .The problem is mainly because we do not know how nodes in abackground dataset distribute in the embedding space generated byCL. Thus, we introduce a method that plots the contrastive represen-tations of target and background datasets, Y T and Y B , together. Asshown in Fig. 4-a3, a4, b3, b4, Y T and Y B are visualized as greencircles and brown triangles, respectively.When a network has high variance in the embedded space, itsnodes are widely distributed along cPCs. Thus, the uniqueness of atarget network G T can be identified by comparing the scatterednessof nodes in Y T and Y B . As shown in Fig. 4-a4, b4, cPCA revealsthat Y T has much higher scatteredness than Y B . Moreover, we caneasily grasp which parts of a target network have strong uniqueness. (a) Highlighting of G T (b) Highlighting of G B (c) Selection with lasso Figure 5: Node highlighting and selection supported in the contrastiverepresentation view. (a) Notations in DeepGL. (b) Visual representations in ContraNA.(c) Computation of the feature ( Φ + sum ◦ Φ sum ◦ Φ − mean )( x ) , where x is total-degree . Figure 6: Representations of network features in DeepGL and Con-traNA. Here, as an example, we use a complex feature (consistingof three relational feature operators) that DeepGL may produce. (a)and (b) represent the same feature: the sum of out-neighbors of thesum of all-neighbors of the mean of in-neighbors of total-degrees . (c)shows an example of the computational flow of this feature. In (c), thecircles and arrows represent nodes and directed links of a network.
Similar to other representation learning methods (e.g., PCA andMDS [86]), a distance in the embedding space of cPCA represents adissimilarity between nodes. Thus, when the target network nodesare highly unique, they are placed far away from the nodes in thebackground network (e.g., the nodes in the red boxes of Fig. 4-a4).
Integration into ContraNA.
We employ the above visualizationas the contrastive representation view of ContraNA (Fig. 2-a), wherethe values of a network feature selected in the feature contributionview (see Fig. 2-b and Sect. 6.2) are colorcoded with a purple-yellowscheme [82]. To encode nodes in target and background networks,we first explored different shapes, including circle, triangle, andsquares; however, circles and squares are hard to distinguish andtriangles require much higher rendering cost with WebGL than cir-cles and squares. We then used circles with different sizes andborders, with larger and black-border circles for the target networkand smaller and gray-border circles for the background network.Moreover, the analyst can highlight the target or the backgroundnetwork by hovering over the corresponding legend as shown inFig. 5-a, b. The contrastive representation view also provides funda-mental interactions, such as zooming, panning, and lasso selection(Fig. 5-c). From the different scatteredness of G T and G B nodes inFig. 2-a, we can decide that there exists uniqueness in the Dolphinnetwork. With the above observation from the contrastive representation view,we move on to interpret the network features and cPCs (Fig. 3-C,
DC2-Interpretability ) with the feature contribution view (Fig. 2-b).
Visual Representation of Network Features.
The left part ofthe feature contribution view lists all the network features generatedby DeepGL. They usually consist of a few relational feature oper-ators (RFOs), which are represented with mathematical notations(Fig. 6-a). However, it is difficult for analysts to interpret featureswith such notations. We thus design an intuitive visual representationof the features (Fig. 6-b).A network feature learned by DeepGL consists of the base feature(e.g., total-degree ), summary measures (e.g., mean), and neighbor5 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 types (e.g., in-neighbors). We use a gray rectangle and an ellipsewith text labels to denote a base feature and a summary measure,respectively. Then, we connect them with a line and, to indicate theneighbor type, annotate with a text label ( in , out , or all ). Also, forin- and out-neighbors, we use an arrowhead to indicate the direction.Lastly, we order them from left to right based on the computationalflow to obtain the feature value. The resultant representation inFig. 6-b visually summarizes the neighborhood relationships and thecomputational flow, which is further explained in Fig. 6-c. Visualization of cPC Loadings.
The right part of the featurecontribution view visualizes cPC loadings [34] described in Sect. 3.2as a heatmap. Each row and column correspond to a network featureand cPC, respectively. Similar to Fujiwara et al.’s work [34], wegenerate scaled cPC loadings (or feature contributions ) between [ − , ] by dividing each cPC’s loadings by their maximum absolutevalue. Then, we encode the scaled cPC loadings with a brown-to-blue-green diverging colormap [26, 48]. The magnitude of theloading represents how strongly a feature contributes to the corre-sponding cPC. For example, the feature at the eighth row in Fig. 2-b( F8 : the mean of all-neighbors’ eigenvector centralities [73]), has themost influence on cPC1. Also, the sign of the loading indicates thecontributed direction along the cPC ( + : positive; − : negative). Forexample, in Fig. 2-a where each node is colored by F8 , we can seethat the feature values of G T generally vary from low to high alongthe positive x -direction.By default, ContraNA automatically selects the feature that moststrongly contributes to cPC1 (e.g., F8 in Fig. 2-b) and highlightsthe corresponding row in yellow. The analyst can select a differentfeature, and all other views are updated based on the selected feature(e.g., node colors in the contrastive representation view).By using the contrastive representation and feature contributionviews together, we discover that the uniqueness of the Dolphinnetwork G T highly relates to F8 . From the nodes colored by thefeature values (Fig. 2-a), we can see that the nodes around the top-left have low values while the nodes around the bottom-right tend tohave higher values. With above results, we further analyze the uniqueness by relating F8 to common network visualizations (Fig. 3-D). ContraNA pro-vides two perspectives for network analysis ( DC3-Intuitiveness ):probability distributions and laid-out networks [10]. Probabilitydistributions are often used to compare the distributions of target andbackground networks’ centralities (e.g., whether the degree distribu-tion follows the power law [10]), and laid-out networks are helpfulfor viewing the topological differences (e.g., whether multiple com-munities exist).
Linking with Probability Distributions.
The probability distri-bution view (Fig. 2-c) shows the distributions of the selected featurevalues in the feature contribution view (i.e., F8 in Fig. 2-b), fortarget and background networks. Its x - and y -coordinates representa (scaled) feature value and its probability (or relative frequency),respectively. Both logarithmic and linear scales for the y -coordinateare supported. We colorcode the probability distribution lines withthe same colors used for the node borders in the contrastive represen-tation view (i.e., black: target network, gray: background network). Linking with Network Layouts.
The network layout view inFig. 2-d, e visualizes laid-out target and background networks, withthe scalable force-directed placement [51]. Same as the contrastiverepresentation view, each node is colored based on the selectedfeature in the feature contribution view (e.g., F8 in Fig. 2-b) andoutlined in black (target network) or gray (background network).The network layout view also supports several basic interactionssuch as zooming, panning, and lasso selection, and is fully linkedwith other views. For example, by reviewing Fig. 2-a, b, d together,we notice that the two node groups found previously (i.e., nodes with Figure 7: Visualization of intermediate computational results of featureF15. small and high F8 values, placed around the top-left and bottom-rightin Fig. 2-a) seem to correspond to distinct communities at the bottom-left and top-right in Fig. 2-d. This can be confirmed by performing alasso selection on the nodes in Fig. 2-a, as demonstrated in Fig. 5-c. Understanding Complicated Network Features.
The linkingsabove can be utilized to further help understand the network featurethat consists of multiple RFOs. As shown in Fig. 7, by hovering overeither the base feature or summary measure in the feature contribu-tion view, the network layout view and the contrastive representationview show the intermediate computational results of the feature val-ues. For instance, Fig. 7 (from left to right) visualizes
PageRank values of G T ’s nodes, the maximum of all-neighbors of PageRank values, and the mean of all-neighbors of them. Thus, the analyst canvisually understand how the base feature values spread across theneighbors and how the final network feature values are derived.Through the analysis from Fig. 3-A to D, we can conclude thatthe Dolphin network G T has unique characteristics relative to theKarate network G B . The uniqueness highly relates to F8: eigenvec-tor centralities of each node’s neighbors, and it clearly reveals theseparation of the two communities in G T , which cannot be seen in G B . The cPCA used in i-cNRL automatically selects the contrastive pa-rameter α and computes cPCs to generate the optimized contrastiverepresentations, i.e., maximizing the variation in X T while simul-taneously minimizing the variation in X B [36] (Fig. 1). However,the analyst may want to loosen or strengthen the reduction of thevariation of X B in order to elucidate the found patterns or discoverdifferent patterns. For example, around the top-left in Fig. 2-a, anorange node, with a high value of F8 , is mixed up with the nodeswith lower values (as annotated in the green box in Fig. 8-b). Also,the resultant cPCs might not apt to interpret visually found patterns.For example, in Fig. 2-a, the value of F8 tends to increase alongthe diagonal line, but not along cPC1 (the x -axis). To handle suchcases, ContraNA supports interactive adjustments of α and cPCs( DC4-Flexibility ). Adjustment of Contrastive Parameter.
ContraNA allows theanalyst to interactively change the contrastive parameter α witha range slider (Fig. 2-f), based on the efficiency of cPCA (e.g.,the completion time is less than 3ms for 10,000 nodes with 10features [34]). However, the update of α in cPCA causes an arbitrarysign flipping for each cPC, similar to PCA [33,87]. Fig. 8-a shows anexample of the flipping along both horizontal and vertical directionswhen α is changed, making it difficult to follow.To address this, we employ a similar solution used for PCA [87].For each of cPC1 and cPC2, we compute the cosine similarity be-tween the coordinates of all nodes before and after the update; thenif the similarity is negative, we flip the sign generated by cPCA.Fig. 8-b shows the result with the sign adjustment. As α decreasesto 38, the orange node annotated with the green rectangle moves to-ward the right-bottom and the separation of nodes with low (purple)and higher values (pink, orange, and yellow) becomes more salient. Adjustment of Contrastive Principal Components.
We intro-duce an interactive method for customizing cPCs, which can be6 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 (a) Without sign adjustment.(b) With sign adjustment.
Figure 8: Changes of node projection by updating contrastive pa-rameter α (a) without and (b) with sign adjustment. Correspondinganimations are available online [1].Figure 9: Adjustment of the cPCs. (a) and (b) show feature contribu-tions of Features 7–10 and the contrastive representation view beforeand after the adjustment, respectively. used for both PCA and cPCA. First, in the contrastive representationview, a preferable axis direction for cPC1 can be drawn as a straightline (Fig. 9-a). We then rotate the projection based on the anglebetween the drawn line and cPC1 (Fig. 9-b). As a result, we alsoneed to update the cPC loadings shown in Fig. 2-b. Similar to therotation in ordinary PCA [76], the cPC loadings can be obtained bysimply multiplying a rotation matrix with the above user-definedangle. For example, Fig. 9-a, b show a subset of the cPC loadingscorresponding to F7-10 before and after the rotation. We can seethat F8 has a strong contribution to both cPC1 and cPC2 and F10 has a stronger contribution to cPC1 than before.Note that we can also use a method developed by Kwon et al. [60]for general scatterplots, including cPCA projection results. It gen-erates new axes based on the user-drawn freeform line over theplot and nonlinear transformation. However, we use the above lin-ear transformation, so that we can update cPC loadings, which areimportant to interpret the result of cPCA.
ASE S TUDIES
In Sect. 6, we have shown the effectiveness of ContraNA through anexample of comparing two social networks. Here, we demonstratetwo additional case studies, including an evaluation of a networkmodel and a comparison of protein interaction networks.
Figure 10: Case study 1. (FC) shows a subset of network featuresvisualized in the feature contribution view (note: a full set of them in-cludes a feature consisting of multiple RFOs). (CR1, TG1, BG1) showthe contrastive representation, target network layout, background net-work layout views after selecting in-degree, respectively. Similarly,(CR2, TG2, BG2) are the results after selecting k -core. As a default,ContraNA selects k -core as the most contributed feature to cPC1. Network modeling is essential to simulate and understand real-worldnetworks (e.g., how they grow and shrink). It also can be used toperform what-if analysis (e.g., what elimination of a hub node willcause), as well as to generate synthetic datasets [42]. ContraNAcan help evaluate network models by comparing them with real-world networks. In this case study, we focus on peer-to-peer (P2P)networks, where a precise network model is essential for analyzingthe robustness of a P2P network [66]. P2P networks are often scale-free [66]; thus, we use the Price’s model [73] as an evaluation target.As listed in Table 1 (see Sect. 8), to identify which characteristicsthe Price’s model does not simulate well, we set a real-world P2Pnetwork (p2p-Gnutella08) as G T , and the network generated withthe Price’s model (Price 2) as G B .As shown in Fig. 10-CR2, the contrastive representation viewindicates the differences in the node distributions of G T and G B . TheP2P network ( G T ) has clear groups of nodes, unlike Price 2 ( G B ).From Fig. 10-FC, in-degree , total-degree , and k -core are identified asmain contribution features. After selecting in-degree in Fig. 10-FCto review the related information with the other views (Fig. 10-CR1,TG1, BG1), we notice that G T has a region where nodes have amuch higher in-degree than the other nodes, as annotated with thegreen boxes in Fig. 10-CR1, TG1. Similar findings appear whenthe total-degree is selected in Fig. 10-FC. As for the k -core number (Fig. 10-CR2, TG2, BG2), there is no obvious difference of thisfeature in G B , but a clear distinction with 8 groups of nodes in G T ,as annotated in Fig. 10-CR2. Moreover, from Fig. 10-FC, we cansee that cPC1 is more related to the k -core values and cPC2 is morerelated to the degree of nodes. Therefore, unique characteristics inthe P2P network have been identified.Also, the result above reveals more variations of the k -core number in the P2P network G T . The k -core number informs that a node atleast connects to other k nodes [73], indicating that the Price’s modelpresents a significant difference in the network robustness from theP2P network. This issue arises because the Price’s model formsa network by always adding a new node with a fixed number oflinks; as a result, the generated nodes have a constant k -core number .Therefore, to better simulate the P2P network, we should develop oruse a model that can generate multiple k -core numbers , such as thedual-Barab´asi-Albert model [71] or its extension [36]. In this case study, we compare two interactome networks, LC-multiple and Combined-AP/MS [93], which represent protein-7 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020
Figure 11: Case study 2. (FC) shows a subset of features in thefeature contribution view. (CR1, TG1, BG1) show the contrastiverepresentation, target network layout, background network layoutviews after selecting F8. Similarly, (CR2, TG2, BG2) are the resultsafter selecting F9. (cid:13) – (cid:13) at the bottom show the zoomed regionsfrom TG2 and BG2. protein interactions of the yeast S. cerevisiae . While LC-multiple isthe literature-curated (LC) interactome from several low-throughput experiments [75, 93], Combined-AP/MS is generated through a high-throughput approach, specifically, affinity purification/massspectrometry (AP/MS) [21, 93]. Low and high-throughput ap-proaches have different pros and cons in capturing the protein in-teractions [90, 93], and thus they produce different interactomes. Acomparison of these interactomes is fundamental to assess the qual-ity and characteristics of each approach [93]. As listed in Table 1, weset LC-multiple and Combined-AP/MS as G T and G B , respectively.As shown in Fig. 11-CR2, TG2, BG2, ContraNA automaticallyselects F9 (because of its high contribution) and generates relatedvisualizations. A difference is revealed in the scatteredness of thetarget and background networks in Fig. 11-CR2. We also notice thatLC-multiple G T has relatively high feature values towards both leftand right directions of cPC1, as annotated with the light blue andgreen. This indicates that cPC1 is not dominantly decided by F9 .Therefore, we select the secondary contributed feature F8 , and theresults are shown in Fig. 11-CR1, TG1, BG1. From Fig. 11-CR1,we can see that only the area annotated light blue in Fig. 11-CR2has high values of F8 . F8 and F9 are related to the eigenvector and Katz centralities . Both centralities measure how strongly a nodeinfluences other nodes; however, the eigenvector centrality tends tobe high only when a node is in a strongly connected region whilethe
Katz centrality can be high even when a node is in a weaklyconnected region [73]. Thus, we can expect that the nodes annotatedwith the light blue and green in Fig. 11-CR2 are in strongly andweakly connected regions, respectively.To visually confirm the above patterns, we select the annotatednodes from Fig. 11-CR2 and zoom into the related regions in Fig. 11-TG2, as detailed by Fig. 11- (cid:13) , (cid:13) , (cid:13) . Here we show only two fromall the regions related to the nodes annotated with the green colorin Fig. 11-CR2, TG2. Similarly, in Fig. 11- (cid:13) , we show the regionwhere the nodes have high F9 values in Fig. 11-BG2. We can seethat the nodes in Fig. 11- (cid:13) , (cid:13) are strongly connected, but not inFig. 11- (cid:13) , (cid:13) . From these observations, we can confirm that theuniqueness is derived from the fact that G T has two different typesof nodes linked to high Katz centrality node(s) in either strongly orweakly connected region, which cannot be seen in G B . This findingindicates that using LC-multiple or Combined-AP/MS to identifyimportant proteins for S. cerevisiae could reach different conclusions. Therefore, additional validation would be needed before decidingtheir importance based on only one dataset.
ONTROLLED U SER S TUDY
In addition to the case studies, we conducted a controlled user studyto assess the usefulness of ContraNA for contrastive network analy-sis. We aimed to answer these research questions: (Q1)
Can analystseffectively identify unique characteristics in a target network (com-pared to a background network), and (Q2)
Can analysts properlyinterpret and explain the found uniqueness? We expected that Q1 would be primarily addressed by the contrastive representation view,and that all the other coordinated views would help answer Q2 . Weprovide the materials used for the study online [1], including thedatasets listed in Table 1, their visualized results with ContraNA,and questionnaires. As far as we know, ContraNA is the first framework designed forcontrastive network analysis, and thus we were not able to finda baseline system to compare against. Therefore, we design thefollowing study to evaluate the usability of ContraNA in terms ofdiscovering a target network’s uniqueness and interpreting it.
Datasets.
As shown in Table 1, we generated random networks(Random 1, 2) with Gilbert’s random graph [10] and scale-freenetworks (Price 1, 2) with the Price’s preferential attachment mod-els [73], as well as used several public datasets. We categorized theanalysis tasks into three by carefully selecting target and backgroundnetworks: (a) no uniqueness is in G T ( is N/A ), (b) theuniqueness in G T can be identified and interpreted with a networkfeature containing only the base feature ( = ≥ Participants.
We recruited 12 participants (4 females and 8males; aged 18–44) at a local university, with 10 from computerscience and 2 from political science. There were 1 postdoc-fellow,10 PhDs, and 1 Master’s. We pre-screened participants to ensurethat they have fundamental knowledge of network science. Theirself-reported familiarity with network analysis had the median of5 ( ), on a scale of 1 (not familiar) to 7 (use regularly). Outof 7 network centralities/measures used in the study (i.e., degree,closeness, betweenness, eigenvector, Katz centralities, PageRank, and k -core number [73]), participants’ knowledge of these had themedian of 3 ( ). Apparatus.
The study was conducted on an iMac (4 GHz IntelCore i7, 16GB 1,600 MHz DDR3) with a 27-inch display (5 , × ,
880 pixels), connected with an Apple Magic Mouse 2. The UI waspresented with Google Chrome in full-screen mode. Because therefinement of contrastive representations (Sect. 6.4) was not relevantto our study tasks, we disabled the related functionalities.
Tasks and Design.
Based on Q1 and Q2 , given target and back-ground networks, participants were asked to perform comparativeanalysis using ContraNA and complete two subtasks, (ST1) and (ST2) : (ST1) identifies whether or not the target network has anyuniqueness compared to the background network, and (ST2) ex-plains the found uniqueness (if any) or the reason of concludingthere is no uniqueness. ST1 required a selection from options of Yes , No , and I’m not sure ; for ST2, participants were asked to writedown their explanation. We employed a within-subjects designfor our study. Each participant completed three comparative net-work analysis tasks in our main study, using three different pairs ofnetworks (Tasks A, B, and C in Table 1). The order of tasks wascounterbalanced across participants.
Procedure.
At the beginning, participants provided their demo-graphics and backgrounds on a survey. A brief tutorial was thenpresented including explanations of the definition of the uniqueness,the above 7 network centralities/measures, the usage of ContraNA,8 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020
Table 1: Networks used for the controlled user study and case studies,where n and l represent the numbers of nodes and links, respectively. Target Network G T Background Network G B Tutorial 1 Price1 Random1 0( n = l = n = l = n =
100 , l = n =
100 , l = n = , l = , n = , l = , n = l = , n = l = , n =
233 , l = , n =
246 , l = , n = , l = , n = , l = , n =
62 , l = n =
34 , l = G T . Figure 12: Accuracy (left) and completion time (right) for each subtask. and 3 concrete analysis examples. Afterward, participants completeda training session, allowing them to get familiar with ContraNA andthe task, followed by the real study consisting of three tasks. Thedatasets used in the tutorials, training, and study tasks are shown inTable 1. Think aloud protocol was used during the training and tasksessions. They were allowed to ask questions about the ContraNAUI and the network centralities and measures. No time limit was setfor the tasks. Lastly, participants provided their feedback with theNASA TLX [49], a questionnaire about ContraNA’s visual interface,and a semi-structured interview. The whole study lasted around 1hour per participant.
This section reports our controlled study results including task accu-racy, completion time, and participants’ subjective feedback.
Accuracy.
The accuracy for each subtask is shown in Fig. 12-left. Two network science experts independently rated participants’explanations in ST2 with a scale of 1 (the worst) to 5 (the best) basedon correctness and comprehensiveness. Weighted Cohen’s kappacoefficient indicates high reliability of the ratings ( κ = .
83, in therange of 0 . . almost perfect agreement ) [20].In general, Task B has the highest mean accuracy for both ST1(100%) and ST2 (92%), which might be because the uniquenessof the target network can be understood easily with the base fea-ture. However, for ST1, a Cochran’s Q test [70] does not show anysignificant differences across tasks. For ST2, a Friedman test [70]reveals significant differences ( χ = . p < . p < .
05) that has the most difficulty. Additionally, par-ticipants’ scores of ST2 show a weak positive correlation (Pearson’scorrelation coefficient ρ = .
31) with the numbers of network cen-tralities/measures they knew, which generally represent their level ofexpertise in network science. Thus, higher expertise seems to helpprovide better explanations.
Completion Time.
Fig. 12-right shows the completion time foreach task. However, a Friedman test does not show any significantdifference across tasks. There is a weak negative correlation ( ρ = − .
33) between the completion times and the numbers of knownnetwork centralities/measures (i.e., the expertise helped finish tasksfaster). For Tasks A and B, ST2 (2.7 minutes and 3.5 minutes,
Figure 13: NASA TLX results (the lower the better).Figure 14: Histograms of participants’ ratings on the overall impres-sion and usefulness of each UI function (the higher the better). Num-bers over the bins represent the frequency. Median ratings are indi-cated in gray. respectively) took longer than ST1 (both 2.5 minutes). But for TaskC, it is the opposite (ST1: 3.5 minutes, ST2: 2.7 minutes). From ourobservation, the reason might be that participants tried to find theexplanation (ST2) before deciding their answer to ST1.
Subjective Feedback.
Fig. 13 lists participants’ ratings with theNASA TLX. Generally, ST2 has higher mean values than ST1 ineach task; however, a Wilcoxon signed-rank exact test does notshow any significant difference in each pair of subtasks. Participantsexpressed relatively high mental demand and effort for performingthe tasks, which is plausible because the network analysis needshigh concentration. Fig. 14 shows the questionnaire results on theimpression of ContraNA. Overall, participants felt that ContraNA iseasy to learn, easy to use, and useful to perform ST1 and ST2. Forthe usefulness of each UI function, the contrastive representation,feature contribution, and network layout views receive high ratings,especially the contrastive representation view, whereas the probabil-ity distribution view has relatively low scores. Also, a Friedman test( χ = . p < . p < .
05) and network layout ( p < . “visually more clear” ( p4, 6 ) and “more intuitiveto understand” (other 8 participants). The rest of the participantspreferred DeepGL’s notation because using mathematical symbolshas less ambiguity. Eleven participants applauded the usefulnessof ContraNA’s visualization of intermediate computational results(Fig. 7), which was used to understand complicated network fea-tures: “Those are particularly useful because you can see the levelsof how these [i.e., features] are getting computed like that” ( p3 ).Two participants with expert knowledge mentioned that they wantedto use the opposite order from the current representation (i.e., fromleft to right, placing RFOs first and then a base feature) because theymentally converted each network feature in this order. However,others stated that they were used to understand each feature from abase feature.9 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 ISCUSSION
We have presented ContraNA and validated it with case studies anda controlled user study. Here, we provide a thematic discussion onadditional aspects of ContraNA as well as the studies.
Limitations in Visual Scalability.
While the studies indicate theusefulness and effectiveness of ContraNA, it is not without limi-tations. ContraNA employs scatterplots, node-link diagrams, andheatmaps in its interface, but these techniques suffer from scalabilityissues. We can enhance these techniques with filtering, aggregation,and focus+context methods to mitigate the issues [23]. A specificscalability issue in ContraNA is the visual representation of networkfeatures, where we use rectangles in the feature contribution view,ellipses, lines with text labels. This may limit the number of RFOsto display in a network feature. However, this is not a major issue,because NRL methods (including DeepGL) that generate featuresbased on relationships of node neighbors are generally utilized withonly a few hops (typically 2 or 3) of neighbors [58, 77]. Another is-sue could be caused when NRL produces a large number of features(e.g., 100 features). This issue can be addressed by only displayingfeatures that highly contribute to cPCs as such features are mostimportant to interpret the cNRL results.
Ambiguity of Uniqueness.
In spite of high mental demand andeffort, participants achieved high accuracy in identifying ( Q1 ) andexplaining ( Q2 ) the uniqueness in a target network when it actuallyexists (Tasks B and C). However, when a target network did nothave clear uniqueness (Task A), the accuracy for ST1 was relativelylow, though there was no significant difference (Sect. 8.2). Potentialreasons might be associated with the ambiguity of uniqueness andparticipants’ expectation, as noted by p1 : “It wasn’t too difficult [tolearn and use the contrastive representation view] but I had a ques-tion of how much separation is enough to define uniqueness.” Whilethe contrastive representation view in Task A showed the similarscatteredness between target and background networks, participantswere able to find some small regions that seemed to relate to theuniqueness if they had an expectation for uniqueness. We found thatall 5 participants who answered
Yes for Task A-ST1 did not providea convincing explanation, with mean accuracy for Task A-ST2 52%(Fig. 12). How to better define and inform a threshold of containingthe uniqueness should be addressed in the further work.
Importance of Interpretability and cNRL.
One notable resultis that participants spent similar time in completing ST1 and ST2.This surprises us because we expected that ST1 would be finishedmuch faster because they only needed to review the contrastive rep-resentation view and select an answer, while ST2 required the use ofmultiple views and writing an explanation. From our observation,we noticed that although they quickly recognized the uniquenessfrom the contrastive representation view, before selecting the answer,they tried to understand the reasons behind to convince themselves.This points out the importance of providing the interpretability in al-gorithms, including NRL and CL methods. This fact also influencedthe mean accuracy of Task C-ST1. Three participants chose
I’m notsure because they were not able to completely understand why thetarget network was unique while the potential uniqueness was found,which may be due to their lower expertise in network science.All the views except for the probability distribution view seemedto be useful according to participants. From our interviews, severalparticipants mentioned that for easier tasks (e.g., Task B) it wasnot necessary to use the probability distribution view; for moredifficult tasks (e.g., Task C), the probability distribution view wasnot helpful to reveal the uniqueness. This indicates the limitationof network comparison based on probability distributions, which isa popular analysis approach, and the necessity of more advancedembedding based approaches, such as cNRL. Further, when askedabout how to perform similar tasks without ContraNA, participantsprovided approaches of either comparing probability distributionsof basic network centralities or comparing laid-out networks. Also, they mentioned that they might be able to find the uniqueness withtheir stated approach but it would “be awful” ( p9 ) and “take longer” ( p1 ), and “I might miss some uniqueness” ( p5 ). In contrast, usingContraNA is “much easier because it supports a lot of stuff youneed to deal with... Comparing the target and background in thecontrastive representation view is really helpful. If you see spreadingpatterns [of a target], it might be unique.” ( p11 ). Usage with Other Algorithms.
ContraNA employs i-cNRL be-cause of its interpretability; however, most of ContraNA functionali-ties are generic enough to be well adapted with other NRL and CLmethods in the architecture. For example, if the interpretability isnot required, DeepGL can be replaced with GraphSAGE [46], whereonly network features are changed. Thus, ContraNA is still applica-ble by updating the visual representations of features in the featurecontribution view. Similarly, we can switch cPCA with the otherCL methods, such as cVAE [3, 80] which can find uniqueness in atarget dataset even when its data points and latent features have non-linear relationships. As we cannot obtain the features’ contributions,in this case, we can simply remove the heatmap from the featurecontribution view. Also, once other interpretable CL methods aredeveloped, we do not need any changes to integrate them into Con-traNA. Another potential extension is cooperating with link featurelearning, which is also supported by DeepGL. In this case, we justneed to add visual encodings of links to the views of ContraNA.
Adaption for Application Domains.
As presented, the linkingwith laid-out networks is important to intuitively understand unique-ness. Networks are often visualized in a specific manner accordingto the application domain. For example, when analyzing brain net-works, neuroscientists often use adjacency-matrix based visualiza-tions or 2D/3D node-link diagrams [30]. This is because the formeris useful to find correlated brain regions with matrix-reordering al-gorithms [12] and the latter can help relate analysis results to theactual locations in a brain. By customizing the network layout views,ContraNA can support such analysis tasks in this specific domain.Also, as shown in Sect. 8.2, the way to understand network featuresgenerated by DeepGL is different by the analyst. Therefore, weshould consider adding settings to customize the representation ofnetwork features based on the analyst’s preference.
10 C
ONCLUSION AND F UTURE W ORK
We have presented ContraNA, a visual analytics framework fornetwork comparison, which utilizes two machine learning schemes—network representation learning and contrastive learning—togetherwith an intuitive visual interface. ContraNA provides the capabilityfor effectively identifying and understanding unique characteristicsof one network relative to another, supporting four key capabilities asoutlined by
DIIF . As our case studies indicate, ContraNA promisesto extract insights from networks found in various application do-mains. Our controlled user study also reflects the usefulness andeffectiveness of ContraNA with carefully-designed analysis tasks. Inthe future, to provide more variations for contrastive network analy-sis, we plan to extend the framework for comparison of two groupsof multiple networks, including dynamic network comparison. Wealso wish to adapt ContraNA to other cNRL algorithms (e.g., basedon GraphSAGE [46], cVAE [80], etc.) and application domains (e.g.,brain network analysis). A CKNOWLEDGMENTS
This research was partially carried out at FXPAL. This researchis sponsored in part by the U.S. National Science Foundationthrough grant IIS-1741536, the Natural Sciences and EngineeringResearch Council of Canada through the Discovery Grant, and FX-PAL through its internship program.10 o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 R EFERENCES [1] The supplementary materials: Videos, datasets, and user study materi-als. https://takanori-fujiwara.github.io/s/contrana/ .[2] A. Abid, M. J. Zhang, V. K. Bagaria, and J. Zou. Exploring patternsenriched in a dataset with contrastive principal component analysis.
Nature Communications , 9(1):2134, 2018.[3] A. Abid and J. Zou. Contrastive variational autoencoder enhancessalient features. arXiv:1902.04601 , 2019.[4] B. Alper, B. Bach, N. Henry Riche, T. Isenberg, and J.-D. Fekete.Weighted graph comparison techniques for brain connectivity analysis.In
Proc. CHI , pp. 483–492, 2013.[5] S. Aminikhanghahi and D. J. Cook. A survey of methods for timeseries change point detection.
Knowledge and Information Systems ,51(2):339–367, 2017.[6] K. Andrews, M. Wohlfahrt, and G. Wurzinger. Visual graph compari-son. In
Proc. IV , pp. 62–67, 2009.[7] B. Bach, N. Henry-Riche, T. Dwyer, T. Madhyastha, J.-D. Fekete,and T. Grabowski. Small MultiPiles: Piling time to explore temporalpatterns in dynamic networks.
Computer Graphics Forum , 34(3):31–40,2015.[8] B. Bach, E. Pietriga, and J.-D. Fekete. GraphDiaries: Animated transi-tions andtemporal navigation for dynamic networks.
IEEE Trans. onVisualization and Computer Graphics , 20(5):740–754, 2013.[9] B. Bach, C. Shi, N. Heulot, T. Madhyastha, T. Grabowski, and P. Drag-icevic. Time Curves: Folding time to visualize patterns of temporalevolution in data.
IEEE Trans. on Visualization and Computer Graph-ics , 22(1):559–568, 2016.[10] A.-L. Barab´asi et al.
Network Science . Cambridge university press,2016.[11] F. Beck, M. Burch, S. Diehl, and D. Weiskopf. A taxonomy andsurvey of dynamic graph visualization.
Computer Graphics Forum ,36(1):133–159, 2017.[12] M. Behrisch, B. Bach, N. Henry Riche, T. Schreck, and J.-D. Fekete.Matrix reordering methods for table and network visualization.
Com-puter Graphics Forum , 35(3):693–716, 2016.[13] A. Benavoli, G. Corani, and F. Mangili. Should we really use post-hoc tests based on mean-ranks?
J. of Machine Learning Research ,17(1):152–161, 2016.[14] A. Bezerianos, F. Chevalier, P. Dragicevic, N. Elmqvist, and J.-D.Fekete. GraphDice: A system for exploring multivariate social net-works.
Computer Graphics Forum , 29(3):863–872, 2010.[15] G. Bhanot, A. Gara, P. Heidelberger, E. Lawless, et al. Optimizing tasklayout on the Blue Gene/L supercomputer.
IBM J. of Research andDevelopment , 49(2.3):489–500, 2005.[16] M. Bostock, V. Ogievetsky, and J. Heer. D data-driven documents. IEEE Trans. on Visualization and Computer Graphics , 17(12):2301–2309, 2011.[17] H. Cai, V. W. Zheng, and K. C.-C. Chang. A comprehensive surveyof graph embedding: Problems, techniques, and applications.
IEEETrans. on Knowledge and Data Engineering , 30(9):1616–1637, 2018.[18] H. Chen and B. M. Sharp. Content-rich biological network constructedby mining pubmed abstracts.
BMC Bioinformatics , 5(1):147, 2004.[19] D. Coelho, I. Chase, and K. Mueller. PeckVis: A visual analytics toolto analyze dominance hierarchies in small groups.
IEEE Trans. onVisualization and Computer Graphics , 26(4):1650–1660, 2020.[20] J. Cohen. Weighted kappa: Nominal scale agreement provision forscaled disagreement or partial credit.
Psychological Bulletin , 70(4):213,1968.[21] S. R. Collins, P. Kemmeren, X.-C. Zhao, J. F. Greenblatt, et al. Towarda comprehensive atlas of the physical interactome of Saccharomycescerevisiae.
Molecular & Cellular Proteomics , 6(3):439–450, 2007.[22] T. Crnovrsanin, C. Muelder, R. Faris, D. Felmlee, and K.-L. Ma. Vi-sualization techniques for categorical analysis of social networks withmultiple edge sets.
Social Networks , 37:56–64, 2014.[23] W. Cui. Visual analytics: A comprehensive overview.
IEEE Access ,7:81555–81573, 2019.[24] A. Dal Col, P. Valdivia, F. Petronetto, F. Dias, C. T. Silva, and L. G.Nonato. Wavelet-based visual analysis of dynamic networks.
IEEETrans. on Visualization and Computer Graphics , 24(8):2456–2469, 2017.[25] T. N. Dang, N. Pendar, and A. G. Forbes. TimeArcs: Visualizing fluctu-ations in dynamic networks.
Computer Graphics Forum , 35(3):61–69,2016.[26] M. Davis. Palettable. https://jiffyclub.github.io/palettable/ . Accessed: 2020-3-31.[27] A.-H. Dirie, A. Abid, and J. Zou. Contrastive multivariate singularspectrum analysis. In
Proc. Allerton Conference , pp. 1122–1127, 2019.[28] F. Emmert-Streib, M. Dehmer, and Y. Shi. Fifty years of graph match-ing, network alignment and network comparison.
Information Sciences ,346:180–197, 2016.[29] P. Federico, W. Aigner, S. Miksch, F. Windhager, and L. Zenk. A visualanalytics approach to dynamic social networks. In
Proc. of i-KNOW ,pp. 1–8, 2011.[30] A. Fornito, A. Zalesky, and E. Bullmore.
Fundamentals of BrainNetwork Analysis . Academic Press, 2016.[31] M. Freire, C. Plaisant, B. Shneiderman, and J. Golbeck. Manynets: aninterface for multiple network analysis and visualization. In
Proc. CHI ,pp. 213–222, 2010.[32] T. Fujiwara, J.-K. Chou, A. M. McCullough, C. Ranganath, and K.-L.Ma. A visual analytics system for brain functional connectivity com-parison across individuals, groups, and time points. In
Proc. PacificVis ,pp. 250–259, 2017.[33] T. Fujiwara, J.-K. Chou, S. Shilpika, P. Xu, L. Ren, and K.-L. Ma. Anincremental dimensionality reduction method for visualizing streamingmultidimensional data.
IEEE Trans. on Visualization and ComputerGraphics , 26(1):418–428, 2020.[34] T. Fujiwara, O.-H. Kwon, and K.-L. Ma. Supporting analysis of dimen-sionality reduction results with contrastive learning.
IEEE Trans. onVisualization and Computer Graphics , 26(1):45–55, 2020.[35] T. Fujiwara, J. K. Li, M. Mubarak, C. Ross, C. D. Carothers, R. B. Ross,and K.-L. Ma. A visual analytics system for optimizing the performanceof large-scale networks in supercomputing systems.
Visual Informatics ,2(1):98–110, 2018.[36] T. Fujiwara, J. Zhao, F. Chen, Y. Yu, and K.-L. Ma. Interpretablecontrastive learning for networks. arXiv:2005.12419 , 2020.[37] C. Gaiteri, S. Mostafavi, C. J. Honey, P. L. De Jager, and D. A. Bennett.Genetic variants in alzheimer disease-molecular and brain networkapproaches.
Nature Reviews Neurology , 12(7):413, 2016.[38] R. Ge and J. Zou. Rich component analysis. In
Proc. ICML , pp.1502–1510, 2016.[39] N. Gehlenborg, S. I. O’donoghue, N. S. Baliga, A. Goesmann, M. A.Hibbs, H. Kitano, et al. Visualization of omics data for systems biology.
Nature Methods , 7:S56–S68, 2010.[40] M. Gleicher. Considerations for visualizing comparison.
IEEE Trans.on Visualization and Computer Graphics , 24(1):413–423, 2018.[41] M. Gleicher, D. Albers, R. Walker, I. Jusufi, C. D. Hansen, and J. C.Roberts. Visual comparison for information visualization.
InformationVisualization , 10(4):289–309, 2011.[42] A. Goldenberg, A. X. Zheng, S. E. Fienberg, and E. M. Airoldi. Asurvey of statistical network models.
Foundations and Trends® inMachine Learning , 2(2):129–233, 2010.[43] R. Gove. Gragnostics: Fast, interpretable features for comparing graphs.In
Proc. IV , pp. 201–209, 2019.[44] A. Grover and J. Leskovec. node2vec: Scalable feature learning fornetworks. In
Proc. KDD , pp. 855–864, 2016.[45] M. Hajij, B. Wang, C. Scheidegger, and P. Rosen. Visual detection ofstructural changes in time-varying graphs using persistent homology.In
Proc. PacificVis , pp. 125–134, 2018.[46] W. Hamilton, Z. Ying, and J. Leskovec. Inductive representationlearning on large graphs. In
Proc. NIPS , pp. 1024–1034, 2017.[47] M. Harrigan, D. Archambault, P. Cunningham, and N. Hurley. EgoNav:Exploring networks through egocentric spatializations. In
Proc. AVI ,pp. 563–570, 2012.[48] M. Harrower and C. A. Brewer. Colorbrewer.org: An online tool forselecting colour schemes for maps.
Cartographic Journal , 40(1):27–37,2003.[49] S. G. Hart and L. E. Staveland. Development of nasa-tlx (task loadindex): Results of empirical and theoretical research. In P. A. Hancockand N. Meshkati, eds.,
Human Mental Workload , vol. 52 of
Advances o appear in IEEE Conference on Visual Analytics Science and Technology (VAST) 2020 in Psychology , pp. 139–183. North-Holland, 1988.[50] N. Henry, J.-D. Fekete, and M. J. McGuffin. NodeTrix: a hybridvisualization of social networks. IEEE Trans. on Visualization andComputer Graphics , 13(6):1302–1309, 2007.[51] Y. Hu. Efficient, high-quality force-directed graph drawing.
Mathemat-ica Journal , 10(1):37–71, 2005.[52] Y. Jia, F. Nie, and C. Zhang. Trace Ratio Problem Revisited.
IEEETrans. on Neural Networks , 20(4):729–735, 2009.[53] M. John and M. Baumann. A visual approach for the comparativeanalysis of character networks in narrative texts. In
Proc. PacificVis ,pp. 247–256, 2019.[54] I. T. Jolliffe. Principal component analysis and factor analysis. In
Principal Component Analysis , pp. 115–128. Springer, 1986.[55] P. Kerpedjiev, N. Abdennur, F. Lekschas, C. McCallum, K. Dinkla,H. Strobelt, J. M. Luber, S. B. Ouellette, A. Azhir, N. Kumar, et al. Hi-Glass: web-based visual exploration and analysis of genome interactionmaps.
Genome biology , 19(1):1–12, 2018.[56] S. P. Kesavan, T. Fujiwara, J. K. Li, C. Ross, M. Mubarak, C. D.Carothers, R. B. Ross, and K.-L. Ma. A visual analytics frameworkfor reviewing streaming performance data. In
Proc. PacificVis , pp.206–2015, 2020.[57] A. N. Khambhati, J. D. Medaglia, E. A. Karuza, S. L. Thompson-Schill,and D. S. Bassett. Subgraphs of functional brain networks identify dy-namical constraints of cognitive control.
PLOS Computational Biology ,14(7):e1006234, 2018.[58] T. N. Kipf and M. Welling. Semi-supervised classification with graphconvolutional networks. arXiv:1609.02907 , 2016.[59] D. Koop, J. Freire, and C. T. Silva. Visual summaries for graph collec-tions. In
Proc. PacificVis , pp. 57–64, 2013.[60] B. C. Kwon, H. Kim, E. Wall, J. Choo, H. Park, and A. Endert.AxiSketcher: Interactive nonlinear axis mapping of visualizationsthrough user drawings.
IEEE Trans. on Visualization and ComputerGraphics , 23(1):221–230, 2017.[61] O.-H. Kwon, T. Crnovrsanin, and K.-L. Ma. What would a graphlook like in this layout? a machine learning approach to large graphvisualization.
IEEE Trans. on Visualization and Computer Graphics ,24(1):478–488, 2018.[62] V. Larivi`ere, Y. Gingras, and ´E. Archambault. Canadian collabora-tion networks: A comparative analysis of the natural sciences, socialsciences and the humanities.
Scientometrics , 68(3):519–533, 2006.[63] A. Lee, D. Archambault, and M. Nacenta. Dynamic Network Plaid: Atool for the analysis of dynamic networks. In
Proc. of ACM CHI , pp.1–14, 2019.[64] J. Leskovec and A. Krevl. SNAP Datasets: Stanford large networkdataset collection. http://snap.stanford.edu/data , 2014. Ac-cessed: 2020-4-10.[65] D. Liu, F. Guo, B. Deng, H. Qu, and Y. Wu. egoComp: A node-link-based technique for visual comparison of ego-networks.
InformationVisualization , 16(3):179–189, 2017.[66] L. Liu, J. Xu, D. Russell, P. Townend, and D. Webster. Efficient andscalable search on scale-free p2p networks.
Peer-to-Peer Networkingand Applications , 2(2):98–108, 2009.[67] S. Liu, X. Wang, J. Chen, J. Zhu, and B. Guo. TopicPanorama: A fullpicture of relevant topics. In
Proc. VAST , pp. 183–192, 2014.[68] X. Liu and H.-W. Shen. The effects of representation and juxtapositionon graphical perception of matrix visualization. In
Proc. CHI , pp.269–278, 2015.[69] D. Lusseau, K. Schneider, O. J. Boisseau, P. Haase, et al. The bot-tlenose dolphin community of doubtful sound features a large propor-tion of long-lasting associations.
Behavioral Ecology and Sociobiology ,54(4):396–405, 2003.[70] E. McCrum-Gardner. Which is the correct statistical test to use?
BritishJ. of Oral and Maxillofacial Surgery , 46(1):38–41, 2008.[71] N. Moshiri. The dual-Barab´asi-Albert model. arXiv:1810.10538 , 2018.[72] S. Murugesan, K. Bouchard, J. Brown, M. Kiran, D. Lurie, B. Hamann,and G. H. Weber. State-based network similarity visualization.
Infor-mation Visualization , 19(2):96–113, 2020.[73] M. Newman.
Networks . Oxford university press, 2018.[74] N. Prˇzulj. Biological network comparison using graphlet degree distri-bution.
Bioinformatics , 23(2):e177–e183, 2007. [75] T. Reguly, A. Breitkreutz, L. Boucher, B.-J. Breitkreutz, et al. Com-prehensive curation and analysis of global interaction networks insaccharomyces cerevisiae.
J. of Biology , 5(4):11, 2006.[76] M. B. Richman. Rotation of principal components.
J. of Climatology ,6(3):293–335, 1986.[77] R. A. Rossi, R. Zhou, and N. Ahmed. Deep inductive graph represen-tation learning.
IEEE Trans. on Knowledge and Data Engineering ,32(3):438–452, 2020.[78] S. Rufiange and M. J. McGuffin. DiffAni: Visualizing dynamic graphswith a hybrid of difference maps and animation.
IEEE Trans. onVisualization and Computer Graphics , 19(12):2556–2565, 2013.[79] D. Sacha, L. Zhang, M. Sedlmair, J. A. Lee, J. Peltonen, D. Weiskopf,S. C. North, and D. A. Keim. Visual interaction with dimensionality re-duction: A structured literature analysis.
IEEE Trans. on Visualizationand Computer Graphics , 23(1):241–250, 2017.[80] K. A. Severson, S. Ghosh, and K. Ng. Unsupervised learning withcontrastive latent variable models. In
Proc. AAAI , vol. 33, pp. 4862–4869, 2019.[81] L. Shi, H. Tong, and X. Mu. Brainquest: Perception-guided brainnetwork comparison. In
Proc. ICDM , pp. 379–388, 2015.[82] N. Smith and S. van der Walt. mpl colormaps. https://bids.github.io/colormap/ . Accessed: 2020-3-31.[83] J. Stehl´e, N. Voirin, A. Barrat, C. Cattuto, et al. High-resolutionmeasurements of face-to-face contact patterns in a primary school.
PlOS ONE , 6(8), 2011.[84] G. K. Tam, V. Kothari, and M. Chen. An analysis of machine-andhuman-analytics in classification.
IEEE Trans. on Visualization andComputer Graphics , 23(1):71–80, 2017.[85] M. Tantardini, F. Ieva, L. Tajoli, and C. Piccardi. Comparing methodsfor comparing networks.
Scientific Reports , 9(1):17557, 2019.[86] W. S. Torgerson. Multidimensional scaling: I. Theory and method.
Psychometrika , 17(4):401–419, 1952.[87] C. Turkay, E. Kaya, S. Balcisoy, and H. Hauser. Designing progressiveand interactive analytics processes for high-dimensional data analysis.
IEEE Trans. on Visualization and Computer Graphics , 23(1):131–140,2017.[88] S. van den Elzen, D. Holten, J. Blaas, and J. J. van Wijk. Reducingsnapshots to points: A visual analytics approach to dynamic networkexploration.
IEEE Trans. on Visualization and Computer Graphics ,22(1):1–10, 2016.[89] T. von Landesberger, M. Gorner, and T. Schreck. Visual analysisof graphs with multiple connected components. In
Proc. VAST , pp.155–162, 2009.[90] M. R. Wilkins. Hares and tortoises: The high-versus low-throughputproteomic race.
Electrophoresis , 30(S1):S150–S155, 2009.[91] X. Yang, L. Shi, M. Daianu, H. Tong, Q. Liu, and P. Thompson.Blockwise human brain network visual comparison using nodetrixrepresentation.
IEEE Trans. on Visualization and Computer Graphics ,23(1):181–190, 2017.[92] V. Yoghourdjian, T. Dwyer, K. Klein, K. Marriott, and M. Wybrow.Graph Thumbnails: Identifying and comparing multiple graphs ata glance.
IEEE Trans. on Visualization and Computer Graphics ,24(12):3081–3095, 2018.[93] H. Yu, P. Braun, M. A. Yıldırım, I. Lemmens, et al. High-quality binaryprotein interaction map of the yeast interactome network.
Science ,322(5898):104–110, 2008.[94] W. W. Zachary. An information flow model for conflict and fission insmall groups.
J. of Anthropological Research , 33(4):452–473, 1977.[95] D. Zhang, J. Yin, X. Zhu, and C. Zhang. Network representationlearning: A survey.
IEEE Trans. on Big Data , 2018.[96] Z. Zhang, P. Cui, and W. Zhu. Deep learning on graphs: A survey.
IEEETrans. on Knowledge and Data Engineering , 2020 (Early Access).[97] J. Zhao, M. Glueck, F. Chevalier, Y. Wu, and A. Khan. Egocentricanalysis of dynamic networks with egolines. In
Proc. of ACM CHI , pp.5003–5014, 2016.[98] J. Zhao, Z. Liu, M. Dontcheva, A. Hertzmann, and A. Wilson. Ma-trixWave: Visual comparison of event sequence data. In
Proc. of ACMCHI , p. 259268, 2015.[99] J. Y. Zou, D. J. Hsu, D. C. Parkes, and R. P. Adams. Contrastivelearning using spectral methods. In
Proc. NIPS , pp. 2238–2246, 2013., pp. 2238–2246, 2013.