Cynthia Bailey Lee

Are user runtime estimates inherently inaccurate

Computer system batch schedulers typically require information from the user upon job submission, including a runtime estimate. Inaccuracy of these runtime estimates, relative to the actual runtime of the job, has been well documented and is a perennial problem mentioned in the job scheduling literature. Typically users provide these estimates under circumstances where their job will be killed after the provided amount of time elapses. Also, users may be unaware of the potential benefits of providing accurate estimates, such as increased likelihood of backfilling. This study examines user behavior when the threat of job killing is removed, and when a tangible reward for accuracy is provided. We show that under these conditions, about half of users provide an improved estimate, but there is not a substantial improvement in the overall average accuracy.

Halving fail rates using peer instruction: a study of four computer science courses

Peer Instruction (PI) is a teaching method that supports student-centric classrooms, where students construct their own understanding through a structured approach featuring questions with peer discussions. PI has been shown to increase learning in STEM disciplines such as physics and biology. In this report we look at another indicator of student success the rate at which students pass the course or, conversely, the rate at which they fail. Evaluating 10 years of instruction of 4 different courses spanning 16 PI course instances, we find that adoption of the PI methodology in the classroom reduces fail rates by a per-course average of 61% (20% reduced to 7%) compared to standard instruction (SI). Moreover, we also find statistically significant improvements within-instructor. For the same instructor teaching the same course, we find PI decreases the fail rate, on average, by 67% (from 23% to 8%) compared to SI. As an in-situ study, we discuss the various threats to the validity of this work and consider implications of wide-spread adoption of PI in computing programs.

Precise and realistic utility functions for user-centric performance analysis of schedulers

Utility functions can be used to represent the value users attach to job completion as a function of turnaround time. Most previous scheduling research used simple synthetic representations of utility, with the simplicity being due to the fact that real user preferences are difficult to obtain, and perhaps concern that arbitrarily complex utility functions could in turn make the scheduling problem intractable. In this work, we advocate a flexible representation of utility functions that can indeed be arbitrarily complex. We show that a genetic algorithm heuristic can improve global utility by analyzing these functions, and does so tractably. Since our previous work showed that users indeed have and can articulate complicated utility functions, the result here is relevant. We then provide a means to augment existing workload traces with realistic utility functions for the purpose of enabling realistic scheduling simulations.

Can peer instruction be effective in upper-division computer science courses?

Peer Instruction (PI) is an active learning pedagogical technique. PI lectures present students with a series of multiple-choice questions, which they respond to both individually and in groups. PI has been widely successful in the physical sciences and, recently, has been successfully adopted by computer science instructors in lower-division, introductory courses. In this work, we challenge readers to consider PI for their upper-division courses as well. We present a PI curriculum for two upper-division computer science courses: Computer Architecture and Theory of Computation. These courses exemplify several perceived challenges to the adoption of PI in upper-division courses, including: exploration of abstract ideas, development of high-level judgment of engineering design trade-offs, and exercising advanced mathematical sophistication. This work includes selected course materials illustrating how these challenges are overcome, learning gains results comparing these upper-division courses with previous lower-division results in the literature, student attitudinal survey results (N = 501), and pragmatic advice to prospective developers and adopters. We present three main findings. First, we find that these upper-division courses achieved student learning gains equivalent to those reported in successful lower-division computing courses. Second, we find that student feedback for each class was overwhelmingly positive, with 88% of students recommending PI for use in other computer science classes. Third, we find that instructors adopting the materials introduced here were able to replicate the outcomes of the instructors who developed the materials in terms of student learning gains and student feedback.

Performance modeling of HPC applications

Publisher Summary This chapter discusses the performance modeling of high performance computing (HPC) applications. Performance models of applications enable HPC system designers and centers to gain insight into the most optimal hardware for their applications, giving them valuable information into the components of hardware that for a certain investment of time/money will give the most benefit for the applications slated to run on the new system. The task of developing accurate performance models for scientific application on such complex systems can be difficult. The chapter briefly review a framework developed that provides an automated means for carrying out performance modeling investigations. The ongoing work to lower the overhead required for obtaining application signatures is discussed, and how one can increased the level-of-detail of convolutions with resulting improvements in modeling accuracy is explained. The chapter discusses how these technology advances enabled performance studies to explain why performance of applications, such as POP (Parallel Ocean Program), NLOM (Navy Layered Ocean Model), and Cobalt60, vary on different machines and quantifies the performance effect of various components of the machines. The chapter concludes by generalizing these results to show how this applications performance would likely improve if the underlying target machines were improved in various dimensions.

Computer science concept inventories: past and future

Concept Inventories (CIs) are assessments designed to measure student learning of core concepts. CIs have become well known for their major impact on pedagogical techniques in other sciences, especially physics. Presently, there are no widely used, validated CIs for computer science. However, considerable groundwork has been performed in the form of identifying core concepts, analyzing student misconceptions, and developing CI assessment questions. Although much of the work has been focused on CS1 and a CI has been developed for digital logic, some preliminary work on CIs is underway for other courses. This literature review examines CI work in other STEM disciplines, discusses the preliminary development of CIs in computer science, and outlines related research in computer science education that contributes to CI development.

Peer instruction in computing: the role of reading quizzes

Peer Instruction has recently gained interest in computing as an effective active learning pedagogy. The general focus of PI research has been on the in-class portion of PI: multiple choice questions and group discussion. Here, our focus is the reading quizzes completed by students for purposes of class preparation. These quizzes contain content questions but also ask for difficulties or confusion with course material. Consistent with expectations, we demonstrate that providing correct responses to quiz questions positively correlates with other course assessments. Somewhat counter-intuitively, we find that identifying confusions, noting problematic sections, or asking questions about the reading are also correlated with lab grades.

Experience report: CS1 in MATLAB for non-majors, with media computation and peer instruction

As computer programming is increasingly considered an essential literacy skill for all students, MATLAB courses in particular can play a role in introducing non-major students to a tool commonly used in many of their fields. This paper reports on our departments experience introducing a CS1 in MATLAB for non-majors course. The course assumed no prior programming experience and no training in linear algebra. Without linear algebra and without the ability to do domain-specific tailoring, we turned to Media Computation to contextualize the skills and motivate students. Media Computation is an approach to programming instruction that focuses on manipulation of visual, audio, and video media. The course design also featured the Peer Instruction lecture format, in which lectures are punctuated by frequent questions that students answer individually and in small groups. To our knowledge, this represents the first time that Media Computation and Peer Instruction pedagogies have been comprehensively adapted to a MATLAB course. This work shares selected materials designed for this course, and reports outcomes of the two terms the course has been offered.

A Multi-institutional Study of Peer Instruction in Introductory Computing

Peer Instruction (PI) is a student-centric pedagogy in which students move from the role of passive listeners to active participants in the classroom. Over the past five years, there have been a number of research articles regarding the value of PI in computer science. The present work adds to this body of knowledge by examining outcomes from seven introductory programming instructors: three novices to PI and four with a range of PI experience. Through common measurements of student perceptions, we provide evidence that introductory computing instructors can successfully implement PI in their classrooms. We find encouraging minimum (74%) and average (92%) levels of success as measured through student valuation of PI for their learning. This work also documents and hypothesizes reasons for comparatively poor survey results in one course, highlighting the importance of the choice of grading policy (participation vs. correctness) for new PI adopters.

An Improved Algorithm for Selecting Ground Motions to Match a Conditional Spectrum

ABSTRACT This paper describes an algorithm to efficiently select ground motions from a database while matching a target mean, variance, and correlations of response spectral values at a range of periods. The approach improves an earlier algorithm by Jayaram et al. [2011]. Key steps in the process are to screen a ground motion database for suitable motions, statistically simulate response spectra from a target distribution, find motions whose spectra match each statistically simulated response spectrum, and then perform an optimization to further improve the consistency of the selected motions with the target distribution. These steps are discussed in detail, and the computational expense of the algorithm is evaluated. A brief example selection exercise is performed, to illustrate the type of results that can be obtained. Source code for the algorithm has been provided, along with metadata for several popular databases of recorded and simulated ground motions, which should facilitate a variety of exploratory and research studies.