Yasemin Acar

On the ecological validity of a password study

The ecological validity of password studies is a complex topic and difficult to quantify. Most researchers who conduct password user studies try to address the issue in their study design. However, the methods researchers use to try to improve ecological validity vary and some methods even contradict each other. One reason for this is that the very nature of the problem of ecological validity of password studies is hard to study, due to the lack of ground truth. In this paper, we present a study on the ecological validity of password studies designed specifically to shed light on this issue. We were able to compare the behavior of 645 study participants with their real world password choices. We conducted both online and laboratory studies, under priming and non-priming conditions, to be able to evaluate the effects of these different forms of password studies. While our study is able to investigate only one specific password environment used by a limited population and thus cannot answer all questions about ecological validity, it does represent a first important step in judging the impact of ecological validity on password studies.

You Get Where You're Looking for: The Impact of Information Sources on Code Security

Vulnerabilities in Android code -- including but not limited to insecure data storage, unprotected inter-component communication, broken TLS implementations, and violations of least privilege -- have enabled real-world privacy leaks and motivated research cataloguing their prevalence and impact. Researchers have speculated that appification promotes security problems, as it increasingly allows inexperienced laymen to develop complex and sensitive apps. Anecdotally, Internet resources such as Stack Overflow are blamed for promoting insecure solutions that are naively copy-pasted by inexperienced developers. In this paper, we for the first time systematically analyzed how the use of information resources impacts code security. We first surveyed 295 app developers who have published in the Google Play market concerning how they use resources to solve security-related problems. Based on the survey results, we conducted a lab study with 54 Android developers (students and professionals), in which participants wrote security-and privacy-relevant code under time constraints. The participants were assigned to one of four conditions: free choice of resources, Stack Overflow only, official Android documentation only, or books only. Those participants who were allowed to use only Stack Overflow produced significantly less secure code than those using, the official Android documentation or books, while participants using the official Android documentation produced significantly less functional code than those using Stack Overflow. To assess the quality of Stack Overflow as a resource, we surveyed the 139 threads our participants accessed during the study, finding that only 25% of them were helpful in solving the assigned tasks and only 17% of them contained secure code snippets. In order to obtain ground truth concerning the prevalence of the secure and insecure code our participants wrote in the lab study, we statically analyzed a random sample of 200,000 apps from Google Play, finding that 93.6% of the apps used at least one of the API calls our participants used during our study. We also found that many of the security errors made by our participants also appear in the wild, possibly also originating in the use of Stack Overflow to solve programming problems. Taken together, our results confirm that API documentation is secure but hard to use, while informal documentation such as Stack Overflow is more accessible but often leads to insecurity. Given time constraints and economic pressures, we can expect that Android developers will continue to choose those resources that are easiest to use, therefore, our results firmly establish the need for secure-but-usable documentation.

Comparing the Usability of Cryptographic APIs

Potentially dangerous cryptography errors are well-documented in many applications. Conventional wisdom suggests that many of these errors are caused by cryptographic Application Programming Interfaces (APIs) that are too complicated, have insecure defaults, or are poorly documented. To address this problem, researchers have created several cryptographic libraries that they claim are more usable, however, none of these libraries have been empirically evaluated for their ability to promote more secure development. This paper is the first to examine both how and why the design and resulting usability of different cryptographic libraries affects the security of code written with them, with the goal of understanding how to build effective future libraries. We conducted a controlled experiment in which 256 Python developers recruited from GitHub attempt common tasks involving symmetric and asymmetric cryptography using one of five different APIs. We examine their resulting code for functional correctness and security, and compare their results to their self-reported sentiment about their assigned library. Our results suggest that while APIs designed for simplicity can provide security benefits – reducing the decision space, as expected, prevents choice of insecure parameters – simplicity is not enough. Poor documentation, missing code examples, and a lack of auxiliary features such as secure key storage, caused even participants assigned to simplified libraries to struggle with both basic functional correctness and security. Surprisingly, the availability of comprehensive documentation and easy-to-use code examples seems to compensate for more complicated APIs in terms of functionally correct results and participant reactions, however, this did not extend to security results. We find it particularly concerning that for about 20% of functionally correct tasks, across libraries, participants believed their code was secure when it was not. Our results suggest that while new cryptographic libraries that want to promote effective security should offer a simple, convenient interface, this is not enough: they should also, and perhaps more importantly, ensure support for a broad range of common tasks and provide accessible documentation with secure, easy-to-use code examples.

VCCFinder: Finding Potential Vulnerabilities in Open-Source Projects to Assist Code Audits

Despite the security communitys best effort, the number of serious vulnerabilities discovered in software is increasing rapidly. In theory, security audits should find and remove the vulnerabilities before the code ever gets deployed. However, due to the enormous amount of code being produced, as well as a the lack of manpower and expertise, not all code is sufficiently audited. Thus, many vulnerabilities slip into production systems. A best-practice approach is to use a code metric analysis tool, such as Flawfinder, to flag potentially dangerous code so that it can receive special attention. However, because these tools have a very high false-positive rate, the manual effort needed to find vulnerabilities remains overwhelming. In this paper, we present a new method of finding potentially dangerous code in code repositories with a significantly lower false-positive rate than comparable systems. We combine code-metric analysis with metadata gathered from code repositories to help code review teams prioritize their work. The paper makes three contributions. First, we conducted the first large-scale mapping of CVEs to GitHub commits in order to create a vulnerable commit database. Second, based on this database, we trained a SVM classifier to flag suspicious commits. Compared to Flawfinder, our approach reduces the amount of false alarms by over 99 % at the same level of recall. Finally, we present a thorough quantitative and qualitative analysis of our approach and discuss lessons learned from the results. We will share the database as a benchmark for future research and will also provide our analysis tool as a web service.

Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security

Online programming discussion platforms such as Stack Overflow serve as a rich source of information for software developers. Available information include vibrant discussions and oftentimes ready-to-use code snippets. Previous research identified Stack Overflow as one of the most important information sources developers rely on. Anecdotes report that software developers copy and paste code snippets from those information sources for convenience reasons. Such behavior results in a constant flow of community-provided code snippets into production software. To date, the impact of this behaviour on code security is unknown. We answer this highly important question by quantifying the proliferation of security-related code snippets from Stack Overflow in Android applications available on Google Play. Access to the rich source of information available on Stack Overflow including ready-to-use code snippets provides huge benefits for software developers. However, when it comes to code security there are some caveats to bear in mind: Due to the complex nature of code security, it is very difficult to provide ready-to-use and secure solutions for every problem. Hence, integrating a security-related code snippet from Stack Overflow into production software requires caution and expertise. Unsurprisingly, we observed insecure code snippets being copied into Android applications millions of users install from Google Play every day. To quantitatively evaluate the extent of this observation, we scanned Stack Overflow for code snippets and evaluated their security score using a stochastic gradient descent classifier. In order to identify code reuse in Android applications, we applied state-of-the-art static analysis. Our results are alarming: 15.4% of the 1.3 million Android applications we analyzed, contained security-related code snippets from Stack Overflow. Out of these 97.9% contain at least one insecure code snippet.

SoK: Lessons Learned from Android Security Research for Appified Software Platforms

Android security and privacy research has boomed in recent years, far outstripping investigations of other appified platforms. However, despite this attention, research efforts are fragmented and lack any coherent evaluation framework. We present a systematization of Android security and privacy research with a focus on the appification of software systems. To put Android security and privacy research into context, we compare the concept of appification with conventional operating system and software ecosystems. While appification has improved some issues (e.g., market access and usability), it has also introduced a whole range of new problems and aggravated some problems of the old ecosystems (e.g., coarse and unclear policy, poor software development practices). Some of our key findings are that contemporary research frequently stays on the beaten path instead of following unconventional and often promising new routes. Many security and privacy proposals focus entirely on the Android OS and do not take advantage of the unique features and actors of an appified ecosystem, which could be used to roll out new security mechanisms less disruptively. Our work highlights areas that have received the larger shares of attention, which attacker models were addressed, who is the target, and who has the capabilities and incentives to implement the countermeasures. We conclude with lessons learned from comparing the appified with the old world, shedding light on missed opportunities and proposing directions for future research.

Why eve and mallory (also) love webmasters: a study on the root causes of SSL misconfigurations

Previous research showed that the SSL infrastructure is a fragile system: X.509 certificate validation fails for a non-trivial number of HTTPS-enabled websites resulting in SSL warning messages presented to users. Studies revealed that warning messages do not provide easy-to-understand information or are ignored by webbrowser users. SSL warning messages are a critical component in the HTTPS infrastructure and many attempts have been made to improve these warning messages. However, an important question has not received sufficient attention yet: Why do webmasters (deliberately) deploy non-validating, security-critical X.509 certificates on publicly available websites? In this paper, we conduct the first study with webmasters operating non-validating X.509 certificates to understand their motives behind deploying those certificates. We extracted the non-validating certificates from Googles webcrawler body of X.509 certificates, informed webmasters about the problem with the X.509 certificate configuration on their website and invited a random sample of the respective webmasters to participate in our study. 755 webmasters participated, allowing us insight into their motives. While one third of them admitted to having misconfigured their webserver accidentally, two thirds of them gave reasons for deliberately using a non-validating X.509 certificate.

How Internet Resources Might Be Helping You Develop Faster but Less Securely

In this experimental study, Android developers using Stack Overflow to solve common security issues were more likely to produce functional--but less secure--code. Given todays time constraints and economic pressures, developers need improved official documentation thats both secure and usable.

A Summary of Survey Methodology Best Practices for Security and Privacy Researchers

“Given a choice between dancing pigs and security, users will pick dancing pigs every time,” warns an oft-cited quote from well-known security researcher Bruce Schneier [132]. This issue of understanding how to make security tools and mechanisms work better for humans (often categorized as usability, broadly construed) has become increasingly important over the past 17 years [7], [159], as illustrated by the growing body of research. Usable security and privacy research has improved our understanding of how to help users stay safe from phishing attacks [12], [62], [77], [105], [109], [129], [138], create strong passwords [39], [73], [130], [152], and control access to their accounts [16], [33], [93], [139], as just three examples. One key technique for understanding and improving how human decision making affects security is the gathering of self-reported data from users. This data is typically gathered via survey and interview studies, and serves to inform the broader security and privacy community about user needs, behaviors, and beliefs. The quality of this data, and the validity of subsequent research results, depends on the choices researchers make when designing their experiments. Contained here is a set of essential guidelines for conducting self-report usability studies distilled from prior work in survey methodology and related fields. Other fields that rely on selfreport data, such as the health and social sciences, have established guidelines and recommendations for collecting high quality self-report data [10], [42], [55], [57], [70], [82], [98], [103], [119], [136], [148], [149].

Developers Need Support, Too: A Survey of Security Advice for Software Developers

Increasingly developers are becoming aware of the importance of software security, as frequent high-profile security incidents emphasize the need for secure code. Faced with this new problem, most developers will use their normal approach: web search. But are the resulting web resources useful and effective at promoting security in practice? Recent research has identified security problems arising from Q&A resources that help with specific secure-programming problems, but the web also contains many general resources that discuss security and secure programming more broadly, and to our knowledge few if any of these have been empirically evaluated. The continuing prevalence of security bugs suggests that this guidance ecosystem is not currently working well enough: either effective guidance is not available, or it is not reaching the developers who need it. This paper takes a first step toward understanding and improving this guidance ecosystem by identifying and analyzing 19 general advice resources. The results identify important gaps in the current ecosystem and provide a basis for future work evaluating existing resources and developing new ones to fill these gaps.