Diane Staheli

Visualization evaluation for cyber security: trends and future directions

The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. This research effort has created many tools and techniques that could be applied to improve cyber security, yet the community has not yet established unified standards for evaluating these approaches to predict their operational validity. In this paper, we survey and categorize the evaluation metrics, components, and techniques that have been utilized in the past decade of VizSec research literature. We also discuss existing methodological gaps in evaluating visualization in cyber security, and suggest potential avenues for future research in order to help establish an agenda for advancing the state-of-the-art in evaluating cyber security visualizations.

Static graph challenge: Subgraph isomorphism

The rise of graph analytic systems has created a need for ways to measure and compare the capabilities of these systems. Graph analytics present unique scalability difficulties. The machine learning, high performance computing, and visual analytics communities have wrestled with these difficulties for decades and developed methodologies for creating challenges to move these communities forward. The proposed Subgraph Isomorphism Graph Challenge draws upon prior challenges from machine learning, high performance computing, and visual analytics to create a graph challenge that is reflective of many real-world graph analytics processing systems. The Subgraph Isomorphism Graph Challenge is a holistic specification with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. Subgraph isomorphism is amenable to both vertex-centric implementations and array-based implementations (e.g., using the Graph-BLAS.org standard). The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed graph challenge is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance have been measured. Specifications, data, and software are publicly available at GraphChallenge.org.

Unlocking user-centered design methods for building cyber security visualizations

User-centered design can aid visualization designers to build better, more practical tools that meet the needs of cyber security users. The cyber security visualization research community can adopt a variety of design methods to more efficiently and effectively build tools. We demonstrate how previous cyber visualization research has omitted a discussion of effectiveness and process in the explanation of design methods. In this paper, we discuss three design methods and illustrate how each method informed two real-world cyber security visualization projects which resulted in successful deployments to users.

Streaming graph challenge: Stochastic block partition

An important objective for analyzing real-world graphs is to achieve scalable performance on large, streaming graphs. A challenging and relevant example is the graph partition problem. As a combinatorial problem, graph partition is NP-hard, but existing relaxation methods provide reasonable approximate solutions that can be scaled for large graphs. Competitive benchmarks and challenges have proven to be an effective means to advance state-of-the-art performance and foster community collaboration. This paper describes a graph partition challenge with a baseline partition algorithm of sub-quadratic complexity. The algorithm employs rigorous Bayesian inferential methods based on a statistical model that captures characteristics of the real-world graphs. This strong foundation enables the algorithm to address limitations of well-known graph partition approaches such as modularity maximization. This paper describes various aspects of the challenge including: (1) the data sets and streaming graph generator, (2) the baseline partition algorithm with pseudocode, (3) an argument for the correctness of parallelizing the Bayesian inference, (4) different parallel computation strategies such as node-based parallelism and matrix-based parallelism, (5) evaluation metrics for partition correctness and computational requirements, (6) preliminary timing of a Python-based demonstration code and the open source C++ code, and (7) considerations for partitioning the graph in streaming fashion. Data sets and source code for the algorithm as well as metrics, with detailed documentation are available at GraphChallenge.org.

BubbleNet: a cyber security dashboard for visualizing patterns

The field of cyber security is faced with ever‐expanding amounts of data and a constant barrage of cyber attacks. Within this space, we have designed BubbleNet as a cyber security dashboard to help network analysts identify and summarize patterns within the data. This design study faced a range of interesting constraints from limited time with various expert users and working with users beyond the network analyst, such as network managers. To overcome these constraints, the design study employed a user‐centered design process and a variety of methods to incorporate user feedback throughout the design of BubbleNet. This approach resulted in a successfully evaluated dashboard with users and further deployments of these ideas in both research and operational environments. By explaining these methods and the process, it can benefit future visualization designers to help overcome similar challenges in cyber security or alternative domains.

VAST Challenge 2015: Mayhem at dinofun world

A fictitious amusement park and a larger-than-life hometown football hero provided participants in the VAST Challenge 2015 with an engaging yet complex storyline and setting in which to analyze movement and communication patterns. The datasets for the 2015 challenge were large—averaging nearly 10 million records per day over a three day period—with a simple straightforward structured format. The simplicity of the format belied a complex wealth of features contained in the data that needed to be discovered and understood to solve the tasks and questions that were posed. Two Mini-Challenges and a Grand Challenge compose the 2015 competition. Mini-Challenge 1 contained structured location and date-time data for park visitors, against which participants were to discern groups and their activities. Mini-Challenge 2 contained structured communication data consisting of metadata about time-stamped text messages sent between park visitors. The Grand Challenge required participants to use both movement and communication data to hypothesize when a crime was committed and identify the most likely suspects from all the park visitors. The VAST Challenge 2015 received 71 submissions, and the datasets were downloaded, at least partially, from 26 countries.

Keynote: Human-Centric Cyber

In this keynote, I will discuss the importance of human factors in cyber security and highlight lessons learned from conducting user-centered design activities with cyber security analysts. As network traffic volume, interconnectedness of devices, and sophistication of cyber threats all continue to increase, so do concerns about the complexity of providing cyber security. Many research efforts focus on the technology aspects of cyber security; few focus on studying the challenges faced by the human ecosystem of analysts, operators, and senior leaders. User-centered design can help uncover unmet needs and gather requirements to build effective systems to support those that perform cyber security work. Design methods in this domain can help establish user needs, identify opportunities for technology to assist, and evaluate concepts - in this, talk we will discuss examples of each. Ultimately, by embracing the human element of cyber, and positioning the human as the focal point of the research process, we can help the technology community be more efficient at building effective tools. We encourage future cyber security projects to broaden the research methodologies, methods, and techniques at their disposal in order to more completely explore this space. The talk will draw from research experience and field work with users on a number of cyber security research projects. Topics covered will include formative user research and the user-centered design process [1, 2], situation awareness prototyping efforts [3, 4], and evaluation methods for cyber security visualization tools [5].