John Homer

Improving Attack Graph Visualization through Data Reduction and Attack Grouping

Various tools exist to analyze enterprise network systems and to produce attack graphs detailing how attackers might penetrate into the system. These attack graphs, however, are often complex and difficult to comprehend fully, and a human user may find it problematic to reach appropriate configuration decisions. This paper presents methodologies that can 1) automatically identify portions of an attack graph that do not help a user to understand the core security problems and so can be trimmed, and 2) automatically group similar attack steps as virtual nodes in a model of the network topology, to immediately increase the understandability of the data. We believe both methods are important steps toward improving visualization of attack graphs to make them more useful in configuration management for large enterprise networks. We implemented our methods using one of the existing attack-graph toolkits. Initial experimentation shows that the proposed approaches can 1) significantly reduce the complexity of attack graphs by trimming a large portion of the graph that is not needed for a user to understand the security problem, and 2) significantly increase the accessibility and understandability of the data presented in the attack graph by clearly showing, within a generated visualization of the network topology, the number and type of potential attacks to which each host is exposed.

Aggregating vulnerability metrics in enterprise networks using attack graphs

Quantifying security risk is an important and yet difficult task in enterprise network security management. While metrics exist for individual software vulnerabilities, there is currently no standard way of aggregating such metrics. We present a model that can be used to aggregate vulnerability metrics in an enterprise network, producing quantitative metrics that measure the likelihood breaches can occur within a given network configuration. A clear semantic model for this aggregation is an important first step toward a comprehensive network security metric model. We utilize existing work in attack graphs and apply probabilistic reasoning to produce an aggregation that has clear semantics and sound computation. We ensure that shared dependencies between attack paths have a proportional effect on the final calculation. We correctly reason over cycles, ensuring that privileges are evaluated without any self-referencing effect. We introduce additional modeling artifacts in our probabilistic graphical model to capture and account for hidden correlations among exploit steps. The paper shows that a clear semantic model for aggregation is critical in interpreting the results, calibrating the metric model, and explaining insights gained from empirical evaluation. Our approach has been rigorously evaluated using a number of network models, as well as data from production systems.

SAT-solving approaches to context-aware enterprise network security management

Enterprise network security management is a complex task of balancing security and usability, with trade-offs often necessary between the two. Past work has provided ways to identify intricate attack paths due to misconfiguration and vulnerabilities in an enterprise system, but little has been done to address how to correct the security problems within the context of various other requirements such as usability, ease of access, and cost of countermeasures. This paper presents an approach based on Boolean satisfiability solving (SAT solving) that can reason about attacks, usability requirements, cost of actions, etc. in a unified, logical framework. Preliminary results show that the approach is both effective and efficient.

On Novices' Interaction with Compiler Error Messages: A Human Factors Approach

The difficulty in understanding compiler error messages can be a major impediment to novice student learning. To alleviate this issue, multiple researchers have run experiments enhancing compiler error messages in automated assessment tools for programming assignments. The conclusions reached by these published experiments appear to be conducting. We examine these experiments and propose five potential reasons for the inconsistent conclusions concerning enhanced compiler error messages: (1) students do not read them, (2) researchers are measuring the wrong thing, (3) the effects are hard to measure, (4) the messages are not properly designed, (5) the messages are properly designed, but students do not understand them in context due to increased cognitive load. We constructed mixed-methods experiments designed to address reasons 1 and 5 with a specific automated assessment tool, Athene, that previously reported inconclusive results. Testing student comprehension of the enhanced compiler error messages outside the context of an automated assessment tool demonstrated their effectiveness over standard compiler error messages. Quantitative results from a 60 minute one-on-one think-aloud study with 31 students did not show substantial increase in student learning outcomes over the control. However, qualitative results from the one-on-one think-aloud study indicated that most students are reading the enhanced compiler error messages and generally make effective changes after encountering them.

An Empirical Study of Iterative Improvement in Programming Assignments

As automated tools for grading programming assignments become more widely used, it is imperative that we better understand how students are utilizing them. Other researchers have provided helpful data on the role automated assessment tools (AATs) have played in the classroom. In order to investigate improved practices in using AATs for student learning, we sought to better understand how students iteratively modify their programs toward a solution by analyzing more than 45,000 student submissions over 7 semesters in an introductory (CS1) programming course. The resulting metrics allowed us to study what steps students took toward solutions for programming assignments. This paper considers the incremental changes students make and the correlating score between sequential submissions, measured by metrics including source lines of code, cyclomatic (McCabe) complexity, state space, and the 6 Halstead measures of complexity of the program. We demonstrate the value of throttling and show that generating software metrics for analysis can serve to help instructors better guide student learning.

Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.

One common frustration students face when first learning to program in a compiled language is the difficulty in interpreting the compiler error messages they receive. Attempts to improve error messages have produced differing results. Two recently published papers showed conflicting results, with one showing measurable change in student behavior, and the other showing no measurable change. We conducted an experiment comparable to these two over the course of several semesters in a CS1 course. This paper presents our results in the context of previous work in this area. We improved the clarity of the compiler error messages the students receive, so that they may more readily understand their mistakes and be able to make effective corrections. Our goal was to help students better understand their syntax mistakes and, as a reasonable measure of our success, we expected to document a decrease in the number of times students made consecutive submissions with the same compilation error. By doing this, we could demonstrate that this enhancement is effective. After collecting and thoroughly analyzing our own experimental data, we found that--despite anecdotal stories, student survey responses, and instructor opinions testifying to the tools helpfulness--enhancing compiler error messages shows no measurable benefit to students. Our results validate one of the existing studies and contradict another. We discuss some of the reasons for these results and conclude with projections for future research.

Augmenting attack graphs to represent data link and network layer vulnerabilities

Attack graphs enable system stakeholders to understand the stepping stones or exploitation procedures that an adversary could potentially execute to impact the confidentiality, integrity, and availability of a network system. These graphs are used to assess risk and to determine components that, when hardened, contribute most to risk reduction. While these graphs are powerful and widely used in enterprise network systems they focus on application vulnerabilities; they currently do not incorporate weaknesses in the network backbone (e.g., routing) that could lead to traffic hijacking, spoofing, eavesdropping, and several others. In this paper, we describe our work in augmenting the MulVAL attack graph software to incorporate network layer misconfigurations. Through a case study, we show how our modular data pipeline, leveraging previous work in network layer attack impact prediction, can aid system stakeholders in identifying risk and deciding on risk reduction strategies.

Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools

Most novice programmers are not explicitly aware of the problem-solving process used to approach programming problems and cannot articulate to an instructor where they are in that process. Many are now arguing that this skill, called metacognitive awareness, is crucial for novice learning. However, novices frequently learn in university CS1 courses that employ automated assessment tools (AATs), which are not typically designed to provide the cognitive scaffolding necessary for novices to develop metacognitive awareness. This paper reports on an experiment designed to understand what difficulties novice programmers currently face when learning to code with an AAT. We describe the experiences of CS1 students who participated in a think-aloud study where they were observed solving a programming problem with an AAT. Our observations show that some students mentally augmented the tool when it did not explicitly support their metacognitive awareness, while others stumbled due to the tools lack of such support. We use these observations to formulate difficulties faced by novices that lack metacognitive awareness, compare these results to other related studies, and look toward future work in modifying AATs.