Katsiaryna Labunets

The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals

[Context and motivation] To remedy the lack of security expertise, industrial security risk assessment methods come with catalogues of threats and security controls. [Question/problem] We investigate in both qualitative and quantitative terms whether the use of catalogues of threats and security controls has an effect on the actual and perceived effectiveness of a security risk assessment method. In particular, we assessed the effect of using domain-specific versus domain-general catalogues on the actual and perceived efficacy of a security risk assessment method conducted by non-experts and compare it with the effect of running the same method by security experts but without catalogues.

On the Equivalence Between Graphical and Tabular Representations for Security Risk Assessment

Context: Many security risk assessment methods are proposed both in academia (typically with a graphical notation) and industry (typically with a tabular notation).Question: We compare methods based on those two notations with respect to their actual and perceived efficacy when both groups are equipped with a domain-specific security catalogue (as typically available in industry risk assessments).

Model comprehension for security risk assessment: an empirical comparison of tabular vs. graphical representations

Tabular and graphical representations are used to communicate security risk assessments for IT systems. However, there is no consensus on which type of representation better supports the comprehension of risks (such as the relationships between threats, vulnerabilities and security controls). Cognitive fit theory predicts that spatial relationships should be better captured by graphs. In this paper we report the results of two studies performed in two countries with 69 and 83 participants respectively, in which we assessed the effectiveness of tabular and graphical representations with respect to extraction correct information about security risks. The experimental results show that tabular risk models are more effective than the graphical ones with respect to simple comprehension tasks and in some cases are more effective for complex comprehension tasks. We explain our findings by proposing a simple extension of Vessey’s cognitive fit theory as some linear spatial relationships could be also captured by tabular models.

Graphical vs. tabular notations for risk models: on the role of textual labels and complexity

[Background] Security risk assessment methods in industry mostly use a tabular notation to represent the assessment results whilst academic works advocate graphical methods. Experiments with MSc students showed that the tabular notation is better than an iconic graphical notation for the comprehension of security risks. [Aim] We investigate whether the availability of textual labels and terse UML-style notation could improve comprehensibility. [Method] We report the results of an online comprehensibility experiment involving 61 professionals with an average of 9 years of working experience, in which we compared the ability to comprehend security risk assessments represented in tabular, UML-style with textual labels, and iconic graphical modeling notations. [Results] Tabular notation are still the most comprehensible notion in both recall and precision. However, the presence of textual labels does improve the precision and recall of participants over iconic graphical models. [Conclusion] Tabular representation better supports extraction of correct information of both simple and complex comprehensibility questions about security risks than the graphical notation but textual labels help.

Estimating the Assessment Difficulty of CVSS Environmental Metrics: An Experiment

[Context] The CVSS framework provides several dimensions to score vulnerabilities. The environmental metrics allow security analysts to downgrade or upgrade vulnerability scores based on a company’s computing environments and security requirements. [Question] How difficult is for a human assessor to change the CVSS environmental score due to changes in security requirements (let alone technical configurations) for PCI-DSS compliance for networks and systems vulnerabilities of different type? [Results] A controlled experiment with 29 MSc students shows that given a segmented network it is significantly more difficult to apply the CVSS scoring guidelines on security requirements with respect to a flat network layout, both before and after the network has been changed to meet the PCI-DSS security requirements. The network configuration also impact the correctness of vulnerabilities assessment at system level but not at application level. [Contribution] This paper is the first attempt to empirically investigate the guidelines for the CVSS environmental metrics. We discuss theoretical and practical key aspects needed to move forward vulnerability assessments for large scale systems.

Towards empirical evaluation of automated risk assessment methods

Security risk assessment methods are numerous, and it might be confusing for organizations to select one. Researchers have conducted empirical studies with established methods in order to find factors that influence their effectiveness and ease of use. In this paper we evaluate the recent TREsPASS semi-automated risk assessment method with respect to the factors identified as critical in several controlled experiments. We also argue that automation of risk assessment raises new research questions that need to be thoroughly investigated in future empirical studies.

Which security catalogue is better for novices

Several catalogues of security threats and controls have been proposed to help organizations in identifying critical risks and improve their risk posture against real world threats. But the role that these catalogues play in a security risk assessment has not yet been investigated. In this paper we report an experiment with 18 MSc students conducted to compare the effect of using domain-specific and domain-general catalogues of threats and security controls on the actual efficacy and perception of a security risk assessment method. The experimental results show that there is no difference in the actual efficacy of the method when applied with the two types of catalogues. In contrast, the perceived usefulness of the method is higher for the participants who have used the domain-specific catalogues. In addition, the domain-specific catalogues are perceived as easier to use by the participants.

Empirical Assessment of Security Requirements and Architecture: Lessons Learned

Over the past three years, our groups at the University of Leuven and the University of Trento have been conducting a number of experimental studies. In particular, two common themes can be easily identified within our work. First, we have investigated the value of several threat modeling and risk assessment techniques. The second theme relates to the problem of preserving security over time, i.e., security evolution. Although the empirical results obtained in our studies are interesting on their own, the main goal of this chapter is to share our experience. The objective is to provide useful, hands-on insight on this type of research work so that the work of other researchers in the community would be facilitated. The contribution of this chapter is the discussion of the challenges we faced during our experimental work. Contextually, we also outline those solutions that worked out in our studies and could be reused in the field by other studies.

No search allowed: what risk modeling notation to choose?

[Background] Industry relies on the use of tabular notations to document the risk assessment results, while academia encourages to use graphical notations. Previous studies revealed that tabular and graphical notations with textual labels provide better support for extracting correct information about security risks in comparison to iconic graphical notation. [Aim] In this study we examine how well tabular and graphical risk modeling notations support extraction and memorization of information about risks when models cannot be searched. [Method] We present results of two experiments with 60 MSc and 31 BSc students where we compared their performance in extraction and memorization of security risk models in tabular, UML-style and iconic graphical modeling notations. [Result] Once search is restricted, tabular notation demonstrates results similar to the iconic graphical notation in information extraction. In memorization task tabular and graphical notations showed equivalent results, but it is statistically significant only between two graphical notations. [Conclusion] Three notations provide similar support to decision-makers when they need to extract and remember correct information about security risks.

Teaching predictive modeling to junior software engineers---seminar format and its evaluation: poster

Due to the increased importance of machine learning in software and security engineering, effective trainings are needed that allow software engineers to learn the required basic knowledge to understand and successfully apply prediction models fast. In this paper, we present a two-days seminar to teach machine learning-based prediction in software engineering and the evaluation ofits learning effects based on Blooms taxonomy. As a teaching scenario for the practical part, we used a paper reporting a research study on the application ofmachine learning techniques to predict vulnerabilities in the code. The results of the evaluation showed that the seminar is an appropriate format for teaching predictive modeling to software engineers. The participants were very enthusiastic and self-motivated to learn about the topic and the empirical investigation based on Blooms taxonomy showed positive learning effects on the knowledge, comprehension, application, analysis, and evaluation level.