Vicki L. Almstrum

A multi-national, multi-institutional study of assessment of programming skills of first-year CS students

In computer science, an expected outcome of a students education is programming skill. This working group investigated the programming competency students have as they complete their first one or two courses in computer science. In order to explore options for assessing students, the working group developed a trial assessment of whether students can program. The underlying goal of this work was to initiate dialog in the Computer Science community on how to develop these types of assessments. Several universities participated in our trial assessment and the disappointing results suggest that many students do not know how to program at the conclusion of their introductory courses. For a combined sample of 216 students from four universities, the average score was 22.89 out of 110 points on the general evaluation criteria developed for this study. From this trial assessment we developed a framework of expectations for first-year courses and suggestions for further work to develop more comprehensive assessments.

Exploring the role of visualization and engagement in computer science education

Visualization technology can be used to graphically illustrate various concepts in computer science. We argue that such technology, no matter how well it is designed, is of little educational value unless it engages learners in an active learning activity. Drawing on a review of experimental studies of visualization effectiveness, we motivate this position against the backdrop of current attitudes and best practices with respect to visualization use. We suggest a new taxonomy of learner engagement with visualization technology. Grounded in Blooms well-recognized taxonomy of understanding, we suggest metrics for assessing the learning outcomes to which such engagement may lead. Based on these taxonomies of engagement and effectiveness metrics, we present a framework for experimental studies of visualization effectiveness. Interested computer science educators are invited to collaborate with us by carrying out studies within this framework.

RUNESTONE, an international student collaboration project

Students will eventually work in a global market; what better preparation can be provided for international collaboration than...international collaboration? The RUNESTONE project is developing and evaluating the notion of incorporating international group projects into the undergraduate computer science curriculum. RUNESTONE adds new dimensions to student teamwork, requiring students to handle collaboration that is remote, cross-cultural and linguistically challenging. RUNESTONE is a three year project, with the prototype version running in Winter 1998 with students at Uppsala University, Sweden, and Grand Valley State University, Michigan, USA. The 1998 pilot study will be followed by a full-scale implementation in 1999 and another in 2000.

Concept inventories in computer science for the topic discrete mathematics

This report describes concept inventories, specialized assessment instruments that enable educational researchers to investigate student (mis)understandings of concepts in a particular domain. While students experience a concept inventory as a set of multiple-choice items taken as a test, this belies its purpose, its careful development, and its validation. A concept inventory is not intended to be a comprehensive instrument, but rather a tool that probes student comprehension of a carefully selected subset of concepts that give rise to the most common and pervasive mismodelings. The report explains how concept inventories have been developed and used in other STEM fields, then outlines a project to explore the feasibility of concept inventories in the computing field. We use the domain of discrete mathematics to illustrate a suggested plan of action.

An international student/faculty collaboration: the Runestone project

Students of today need to be prepared to work in globally distributed organizations. Part of that preparation involves teaching students to work effectively in teams to solve problems. Students also must be able to work with individuals located at distant sites where there is no or very little face-to-face interaction. The Runestone project, an international collaboration between two universities, adds new dimensions to student teamwork, requiring students to handle collaboration that is remote, cross-cultural, and technically challenging. Runestone is a three-year project funded by the Swedish Council for the Renewal of Undergraduate Education. A pilot study in 1998 was followed by a full-scale implementation in 1999 with another implementation ongoing in 2000.Each time this global cooperation project is run, both students and faculty learn important lessons in how to work with each other in a virtual environment. This paper discusses both student and faculty learning outcomes for Runestone 1999.

What is the attraction to computing

The strongest motivators include a sense of accomplishment from solving problems and programming; the weakest include being captivated by the Web and a passion for playing computer games.

Support for teaching formal methods

This report describes a growth path for the area referred to as formal methods within the computing education community. We define the term formal methods and situate it within our field by highlighting its role in Computing Curricula 1991, Computing Curricula 2001, and the SoftWare Engineering Body Of Knowledge (SWEBOK). The working group proposes an enhancement to an existing web resource, which is a rich collection of materials and links related to formal methods. The new resource is designed to provide a bridge between the general computing education community and the formal methods community. The goal is to allow the latter to provide useful support for the former for the ultimate benefit of all of our students. Eventually, the working group aspires to see the concepts of formal methods integrated seamlessly into the computing curriculum so that it is not necessary to separate them in our discussions.

Interaction factors in software development performance in distributed student teams in computer science

This research in progress paper compares the characteristics of high and low performance distributed student teams doing software development in Computer Science. The distributed student teams were involved in a software development project that was part of a Computer Science course at two universities located in different countries.We developed a set of categories to examine the email communication of distributed student teams. This paper tracks the progression and changes in the categories coded for each teams communication throughout the projects timeline, particularly during key decision periods in the software development cycle.

Evaluation: turning technology from toy to tool: report of the working group on evaluation

Evaluation is an educational process, not an end in itselfi we learn in order to help our students learn. This paper presents a pragmatic perspective on evaluation, viewing it as a matter of trade-offs. The space of possible evaluation approaches is analysed in terms of trade-offs among desired evidence, costs, and other constraints. This approach is illustrated with example scenarios and a list of selected resources is provided. Aim of the Working Group This working group set out to consider how pragmatic, empirical evaluation can be used to harness technology for teaching Computer Science and Information Systems. Educators reject the tendency to adopt ‘technology for technology’s sake’ and want to analyze technology in terms of its suitability for a teaching purpose and its impact—both costs and benefits—on teaching practice and outcomes. The question is not ‘Can we use technology in teaching?’, but ‘Can we use technology to enhance teaching and improve learning?’ Empirical evaluation and technology can form a powerful partnership to enhance teaching purposefully and usably. The working group explored the parameters of an effective partnership. Introduction Computer Science and Information Systems (CS/IS) are rife with examples of technology-driven projects that fail to address fundamental issues, with systems designed by introspection, with software evaluated by market share alone, with good ideas neglected after poor initial implementations. Evaluation is often Permission to make digitalmard copy of part or atl of this work for personal or classroom use is ranted without fee provided that copies are not made f or distributed for pro d or wmmercial advantage, the copyright notice, the titte of the publication and its date appear, and notice is given that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Integrating Tech. into C.S.E. 6/96 Barcelona, Spain 01996 ACM 0-89791 -844-4/96/0009 ...

Best practices in software engineering project class management

3.50 Diane M. Miller University of Southern Mississippi, USA dmmiller@medea. gp.usm.edu Marian Petre (joint chair) Open University, UK M.Petre@open.ac.uk Paul Schragger Villanova University, USA schragge@monet. vilI.edul Fred Springsteel University of Missouri, USA csfreds @mizzoul .missouri.edu seen as an expensive, time-consuming, esoteric process with little practical relevance. But principled, practical evaluation— empirical study of actual practice, perhaps within a tightly focused question or a particular task—can identify crucial issues, debunk inappropriate folklore, give substance to intuition, disambiguate causes, and make the difference between failure and success. The introduction of new technologies increases the importance of evaluation in order to untangle the snarl of factors and influences that impinge on how technology is used in context. Unless educational technology can address educational objectives, the ‘nifty’ ideas it encompasses are no more than fashion. Evaluators need to base their analyses and designers neecl to base thleir designs on real practice; not everything that is ‘intuitive’ or ‘sexy’ is appropriate within real teaching environments. Evaluation offers a means of putting technology into perspective, so that it is viewed as a tool for addressing real problems—a means, rather than an end in itself. Technology as toy and tool The current leading-edge technologies, such as videoconferencing, multi-media, software vi sualizatiou, and Internetenabled applications (World Wide Web, electronic mail, bulletin board systems, etc.), are perceived to have immediate potential for use as educational tools. However, it is all too easy to mis-aplply these technologies, using them as flashy toys or interesting playthings. Technology-led adoption follows a ‘we have it—let’s use it’ enthusiasm. But that can be a blind alley for evaluation: often the need for an answer expires before we have a chance to ask the question. We should pursue an education-led deliberation: ‘We have it—but is it appropriate for this purpose?’ Technology remains a toy when it is used merely because it is attractive and exciting, but its real potential is unexplored. Technology is often introduced into education to attract and excite, without any more than an assumption that it might be useful. But, if applied without deliberative study of its use in context and without the evaluation of the technology’s impact on this use, ‘educational’ technology remains a toy.