Jason Tyler King

Modifying without a trace: general audit guidelines are inadequate for open-source electronic health record audit mechanisms

Without adequate audit mechanisms, electronic health record (EHR) systems remain vulnerable to undetected misuse. Users could modify or delete protected health information without these actions being traceable. The objective of this paper is to assess electronic health record audit mechanisms to determine the current degree of auditing for non-repudiation and to assess whether general audit guidelines adequately address non-repudiation. We derived 16 general auditable event types that affect non-repudiation based upon four publications. We qualitatively assess three open-source EHR systems to determine if the systems log these 16 event types. We find that the systems log an average of 12.5% of these event types. We also generated 58 black-box test cases based on specific auditable events derived from Certification Commission for Health Information Technology criteria. We find that only 4.02% of these tests pass. Additionally, 20% of tests fail in all three EHR systems. As a result, actions including the modification of patient demographics and assignment of user privileges can be executed without a trace of the user performing the action. The ambiguous nature of general auditable events may explain the inadequacy of auditing for non-repudiation. EHR system developers should focus on specific auditable events for managing protected health information instead of general events derived from guidelines.

Hidden in plain sight: Automatically identifying security requirements from natural language artifacts

Natural language artifacts, such as requirements specifications, often explicitly state the security requirements for software systems. However, these artifacts may also imply additional security requirements that developers may overlook but should consider to strengthen the overall security of the system. The goal of this research is to aid requirements engineers in producing a more comprehensive and classified set of security requirements by (1) automatically identifying security-relevant sentences in natural language requirements artifacts, and (2) providing context-specific security requirements templates to help translate the security-relevant sentences into functional security requirements. Using machine learning techniques, we have developed a tool-assisted process that takes as input a set of natural language artifacts. Our process automatically identifies security-relevant sentences in the artifacts and classifies them according to the security objectives, either explicitly stated or implied by the sentences. We classified 10,963 sentences in six different documents from healthcare domain and extracted corresponding security objectives. Our manual analysis showed that 46% of the sentences were security-relevant. Of these, 28% explicitly mention security while 72% of the sentences are functional requirements with security implications. Using our tool, we correctly predict and classify 82% of the security objectives for all the sentences (precision). We identify 79% of all security objectives implied by the sentences within the documents (recall). Based on our analysis, we develop context-specific templates that can be instantiated into a set of functional security requirements by filling in key information from security-relevant sentences.

Using templates to elicit implied security requirements from functional requirements - a controlled experiment

Context: Security requirements for software systems can be challenging to identify and are often overlooked during the requirements engineering process. Existing functional requirements of a system can imply the need for security requirements. Systems having similar security objectives (e.g., confidentiality) often also share security requirements that can be captured in the form of reusable templates and instantiated in the context of a system to specify security requirements. Goal: We seek to improve the security requirements elicitation process by automatically suggesting appropriate security requirement templates implied by existing functional requirements. Method: We conducted a controlled experiment involving 50 graduate students enrolled in a software security course to evaluate the use of automatically-suggested templates in eliciting implied security requirements. Participants were divided into treatment (automatically-suggested templates) and control groups (no templates provided). Results: Participants using our templates identified 42% of all the implied security requirements in the oracle as compared to the control group, which identified only 16% of the implied security requirements. Template usage increased the efficiency of security requirements identified per unit of time. Conclusion: Automatically-suggested templates helped participants (security non-experts) think about security implications for the software system and consider more security requirements than they would have otherwise. We found that participants need more incentive than just a participatory grade when completing the task. Further, we recommend to ensure task completeness, participants either need a step-driven (i.e., wizard) approach or progress indicators to identify remaining work.

Establishing a baseline for measuring advancement in the science of security: an analysis of the 2015 IEEE security & privacy proceedings

To help establish a more scientific basis for security science, which will enable the development of fundamental theories and move the field from being primarily reactive to primarily proactive, it is important for research results to be reported in a scientifically rigorous manner. Such reporting will allow for the standard pillars of science, namely replication, meta-analysis, and theory building. In this paper we aim to establish a baseline of the state of scientific work in security through the analysis of indicators of scientific research as reported in the papers from the 2015 IEEE Symposium on Security and Privacy. To conduct this analysis, we developed a series of rubrics to determine the completeness of the papers relative to the type of evaluation used (e.g. case study, experiment, proof). Our findings showed that while papers are generally easy to read, they often do not explicitly document some key information like the research objectives, the process for choosing the cases to include in the studies, and the threats to validity. We hope that this initial analysis will serve as a baseline against which we can measure the advancement of the science of security.

Enabling forensics by proposing heuristics to identify mandatory log events

Software engineers often implement logging mechanisms to debug software and diagnose faults. As modern software manages increasingly sensitive data, logging mechanisms also need to capture detailed traces of user activity to enable forensics and hold users accountable. Existing techniques for identifying what events to log are often subjective and produce inconsistent results. The objective of this study is to help software engineers strengthen forensic-ability and user accountability by 1) systematically identifying mandatory log events through processing of unconstrained natural language software artifacts; and 2) proposing empirically-derived heuristics to help determine whether an event must be logged. We systematically extract each verb and object being acted upon from natural language software artifacts for three open-source software systems. We extract 3,513 verb-object pairs from 2,128 total sentences studied. Two raters classify each verb-object pair as either a mandatory log event or not. Through grounded theory analysis of discussions to resolve disagreements between the two raters, we develop 12 heuristics to help determine whether a verb-object pair describes an action that must be logged. Our heuristics help resolve 882 (96%) of 919 disagreements between the two raters. In addition, our results demonstrate that the proposed heuristics facilitate classification of 3,372 (96%) of 3,513 extracted verb-object pairs as either mandatory log events or not.

Measuring the forensic-ability of audit logs for nonrepudiation

Forensic analysis of software log files is used to extract user behavior profiles, detect fraud, and check compliance with policies and regulations. Software systems maintain several types of log files for different purposes. For example, a system may maintain logs for debugging, monitoring application performance, and/or tracking user access to system resources. The objective of my research is to develop and validate a minimum set of log file attributes and software security metrics for user nonrepudiation by measuring the degree to which a given audit log file captures the data necessary to allow for meaningful forensic analysis of user behavior within the software system. For a log to enable user nonrepudiation, the log file must record certain data fields, such as a unique user identifier. The log must also record relevant user activity, such as creating, viewing, updating, and deleting system resources, as well as software security events, such as the addition or revocation of user privileges. Using a grounded theory method, I propose a methodology for observing the current state of activity logging mechanisms in healthcare, education, and finance, then I quantify differences between activity logs and logs not specifically intended to capture user activity. I will then propose software security metrics for quantifying the forensic-ability of log files. I will evaluate my work with empirical analysis by comparing the performance of my metrics on several types of log files, including both activity logs and logs not directly intended to record user activity. My research will help software developers strengthen user activity logs for facilitating forensic analysis for user nonrepudiation.

Log your CRUD: design principles for software logging mechanisms

According to a 2011 survey in healthcare, the most commonly reported breaches of protected health information involved employees snooping into medical records of friends and relatives. Logging mechanisms can provide a means for forensic analysis of user activity in software systems by proving that a user performed certain actions in the system. However, logging mechanisms often inconsistently capture user interactions with sensitive data, creating gaps in traces of user activity. Explicit design principles and systematic testing of logging mechanisms within the software development lifecycle may help strengthen the overall security of software. The objective of this research is to observe the current state of logging mechanisms by performing an exploratory case study in which we systematically evaluate logging mechanisms by supplementing the expected results of existing functional black-box test cases to include log output. We perform an exploratory case study of four open-source electronic health record (EHR) logging mechanisms: OpenEMR, OSCAR, Tolven eCHR, and WorldVistA. We supplement the expected results of 30 United States government-sanctioned test cases to include log output to track access of sensitive data. We then execute the test cases on each EHR system. Six of the 30 (20%) test cases failed on all four EHR systems because user interactions with sensitive data are not logged. We find that viewing protected data is often not logged by default, allowing unauthorized views of data to go undetected. Based on our results, we propose a set of principles that developers should consider when developing logging mechanisms to ensure the ability to capture adequate traces of user activity.

Developing Software Engineering Skills using Real Tools for Automated Grading

Situated learning theory supports engaging students with materials and resources that reflect professional standards and best practices. Starting with our introductory courses, we incorporate situated learning to support student engagement in software engineering practices and processes through the use of industrial strength open-source tools in several classes throughout the undergraduate computer science curriculum at NC State University. Additionally, these tools support several logistical and educational needs in computer science classrooms, including assignment submission systems and automated grading. In this tools paper, we present our Canary Framework for supporting software engineering practices through the use of Eclipse for development; GitHub for submission and collaboration; and Jenkins for continuous integration and automated grading. These tools are used in five of ten core courses by more than 3000 students over ten semesters. While the use of these tools in education is not unique, we want to share our model of using professional tools in a classroom setting and our experiences on how this framework can support multiple courses throughout the curriculum and at scale.

Teaching Software Engineering Skills in CS1.5: Incorporating Real-world Practices and Tools (Abstract Only)

Students learn best in environments where they can meaningfully engage with materials that emulate real-world scenarios. Incorporating software engineering best practices and supporting tools in introductory courses provides students the opportunity to engage in course materials as a novice member of the profession. We support student engagement with industry tools to support software engineering best practices for tutorials, in-class labs, and programming projects. The goal of the research is to improve student learning, engagement in the course and profession, and retention through the use of software engineering practices and tools that introduce students to the software engineering profession. A prior study on the incorporation of in-class laboratories, supported with software engineering best practices, on linear data structures showed an increase in engagement, but did not show a difference on student learning when compared with active learning lectures. We are currently expanding the study by incorporating in-class laboratories across a full semester of a CS1.5 class at NC State University. The poster presents the preliminary results from Fall 2015.

Automating Software Engineering Best Practices Using an Open Source Continuous Integration Framework (Abstract Only)

Ideally, software engineering courses should adequately reflect real-world software development so that students obtain a better understanding and experience with practices and techniques used in industry. Our objective is to improve software engineering courses by incorporating best practices for automated software engineering and facilitating rapid feedback for students using an open source continuous integration framework for evaluating student software development. The open source Jenkins Continuous Integration Server is the core of our framework, which provides a consistent environment for building student projects, executing automated test cases, calculating code coverage, executing static analysis, and generating reports for students. By using continuous integration, a common tool in real-world software development, we can incorporate software engineering best practices, introduce students to continuous integration in practice, and provide formative feedback to students throughout the software development lifecycle. We found that 76% or more of students in each of the classes that deploy our framework reported that using Jenkins increased their productivity, and that 84% or more of students in each of the classes reported that using Jenkins increased their code quality.