Dragutin Petkovic

Work in progress: A machine learning approach for assessment and prediction of teamwork effectiveness in software engineering education

One of the challenges in effective software engineering (SE) education is the lack of objective assessment methods of how well student teams learn the critically needed teamwork practices, defined as the ability: (i) to learn and effectively apply SE processes in a teamwork setting, and (ii) to work as a team to develop satisfactory software (SW) products. In addition, there are no effective methods for predicting learning effectiveness in order to enable early intervention in the classroom. Most of the current approaches to assess achievement of SE teamwork skills rely solely on qualitative and subjective data taken as surveys at the end of the class and analyzed only with very rudimentary data analysis. In this paper we present a novel approach to address the assessment and prediction of student learning of teamwork effectiveness in software engineering education based on: a) extracting only objective and quantitative student team activity data during their team class project; b) pairing these data with related independent observations and grading of student team effectiveness in SE process and SE product components in order to create “training database” and c) applying a machine learning (ML) approach, namely random forest classification (RF), to the above training database in order to create ML models, ranked factors and rules that can both explain (e.g. assess) as well as provide prediction of the student teamwork effectiveness. These student team activity data are being collected in joint and already established (since 2006) SE classes at San Francisco State University (SFSU), Florida Atlantic University (FAU) and Fulda University, Germany (Fulda), from approximately 80 students each year, working in about 15 teams, both local and global (with students from multiple schools).

High precision prediction of functional sites in protein structures.

We address the problem of assigning biological function to solved protein structures. Computational tools play a critical role in identifying potential active sites and informing screening decisions for further lab analysis. A critical parameter in the practical application of computational methods is the precision, or positive predictive value. Precision measures the level of confidence the user should have in a particular computed functional assignment. Low precision annotations lead to futile laboratory investigations and waste scarce research resources. In this paper we describe an advanced version of the protein function annotation system FEATURE, which achieved 99% precision and average recall of 95% across 20 representative functional sites. The system uses a Support Vector Machine classifier operating on the microenvironment of physicochemical features around an amino acid. We also compared performance of our method with state-of-the-art sequence-level annotator Pfam in terms of precision, recall and localization. To our knowledge, no other functional site annotator has been rigorously evaluated against these key criteria. The software and predictive models are incorporated into the WebFEATURE service at http://feature.stanford.edu/wf4.0-beta.

Using Learning Analytics to Assess Capstone Project Teams

Machine-learning-based analytics are an effective tool to help assess student teamwork skills and predict learning outcomes in software engineering courses.

SETAP: Software engineering teamwork assessment and prediction using machine learning

Effective teaching of teamwork skills in local and globally distributed Software Engineering (SE) teams is recognized as an important part of the education of current and future software engineers. Effective methods for assessment and early prediction of learning effectiveness in SE teamwork are not only a critical part of teaching but also of value in industrial training and project management. This paper presents a novel analytical approach to the assessment and, most importantly, the prediction of learning outcomes in SE teamwork based on data from our joint software engineering class concurrently taught at San Francisco State University (SFSU), Florida Atlantic University (FAU) and Fulda University, Germany (Fulda). Our approach focuses on assessment and prediction of SE teamwork in terms of ability of student teams to apply best SE processes and develop SE products. It differs from existing work in the following aspects: a) it develops and uses only objective and quantitative measures of team activity from multiple sources, such as statistics of student time use, software engineering tool use, and instructor observations; b) it leverages powerful machine learning (ML) techniques applied to team activity measurements to identify quantitative and objective factors which can assess and predict learning of software engineering teamwork skills at the team level. In this paper we provide the following contributions: a) we present in detail for the first time the full team activity measurement data set we developed, consisting of over 40 objective and quantitative measures extracted from student teams working on class projects; b) we present a ML framework which applies the Random Forest (RF) algorithm to the team activity measurements and team outcomes, focusing on predicting teams that are likely to fail; c) we describe in detail our now fully implemented and operational data processing pipeline, consisting of data collection methods from multiple sources, ML training database creation, and ML analysis subsystems; and finally d) we present very preliminary results of ML analysis results based on the data from our joint software engineering classes in Fall 2012, and Spring 2013, with the data from 17 student teams. While our ML training database is currently small, it continuously grows. Our preliminary results, verified with two independent accuracy measures, show that RF is able to predict SE Process and SE Product team performance in intuitively explainable manner.

Microenvironment-Based Protein Function Analysis by Random Forest

Machine learning-based prediction of protein functions plays a key role in bioinformatics and pharmaceutical research, facilitating swift discovery of new drugs in high-throughput settings. This paper presents an adaptation of Random Forest to the structure-based protein function prediction. Our system represents proteins 3D physicochemical structural information in microenvironment descriptors whose spatial resolution is much finer than other sequence-based protein descriptors. We prepare our datasets for seven active sites from five protein function classes by using multiple public data banks and train Random Forest classifiers to identify these seven function models in proteins. This paper presents two experiment studies: 1) a 5-fold stratified cross-validation for comparing Random Forest with Naive Bayes and Support Vector Machine and 2) systematic comparison of Random Forests two variable importance measures. Promising results of these studies demonstrate a potential for Random Forest to improve the accuracy of the current protein function assays.

Toward objective and quantitative assessment and prediction of teamwork effectiveness in software engineering courses

Forward Today’s software engineering projects require teamwork which students practice in upper division software engineering courses. However, do they really ‘learn’ teamwork practices? This month’s column addresses this question. While reading this article please think about how the concepts presented might be effectively applied in a corporate setting. Mark & Peter Introduction One of the critical challenges in effective software engineering (SE) education is the lack of objective assessment methods of how well student teams learn the critically needed teamwork practices, defined as the ability: (i) to learn and effectively apply SE processes in a teamwork setting, and (ii) to work as a team to develop satisfactory software (SW) products. In addition, there are no effective methods for predicting learning effectiveness in order to enable early intervention in the classroom. This is further complicated with the emergence of global SW teams. Current approaches to assess achievement of SE teamwork skills rely on qualitative and subjective data taken as surveys at the end of the class with only rudimentary data analysis. In this article we present initial progress in our research to address the assessment and prediction of student learning of teamwork effectiveness in SE education. Our novel approach is based on: a) extracting only objective and quantitative student team activity data during their team class project; b) pairing this data with related independent observations and grading of student team effectiveness in SE process and SE product components in order to create “training database”; and c) applying a machine learning (ML) approach, namely random forest classification (RF), to the above training database in order to create ML models, ranked factors and rules that can both explain and assess, as well as provide prediction of the student teamwork effectiveness. Student team activity data are collected from ongoing and synchronously offered SE classes at San Francisco State University (SFSU), Florida Atlantic University (FAU) and Fulda University, Germany (Fulda), for approximately 80 students each year, working in about 15 teams, where student teams are both local and global (with students from multiple schools). In this article we summarize our approach and present preliminary data analysis results which served to test the concept, data gathering and ML tools we intend to use. We believe that success in this project will transform teaching (e.g. assessment) of critically important SE teamwork and will be of benefit to managing SE projects in industry.

GeneDive: A gene interaction search and visualization tool to facilitate precision medicine

Obtaining relevant information about gene interactions is critical for understanding disease processes and treatment. With the rise in text mining approaches, the volume of such biomedical data is rapidly increasing, thereby creating a new problem for the users of this data: information overload. A tool for efficient querying and visualization of biomedical data that helps researchers understand the underlying biological mechanisms for diseases and drug responses, and ultimately helps patients, is sorely needed. To this end we have developed GeneDive, a web-based information retrieval, filtering, and visualization tool for large volumes of gene interaction data. GeneDive offers various features and modalities that guide the user through the search process to efficiently reach the information of their interest. GeneDive currently processes over three million gene-gene interactions with response times within a few seconds. For over half of the curated gene sets sourced from four prominent databases, more than 80% of the gene set members are recovered by GeneDive. In the near future, GeneDive will seamlessly accommodate other interaction types, such as gene-drug and gene-disease interactions, thus enabling full exploration of topics such as precision medicine. The GeneDive application and information about its underlying system architecture are available at http://www.genedive.net.

Using the random forest classifier to assess and predict student learning of Software Engineering Teamwork

The overall goal of our Software Engineering Teamwork Assessment and Prediction (SETAP) project is to develop effective machine-learning-based methods for assessment and early prediction of student learning effectiveness in software engineering teamwork. Specifically, we use the Random Forest (RF) machine learning (ML) method to predict the effectiveness of software engineering teamwork learning based on data collected during student team project development. These data include over 100 objective and quantitative Team Activity Measures (TAM) obtained from monitoring and measuring activities of student teams during the creation of their final class project in our joint software engineering classes which ran concurrently at San Francisco State University (SFSU), Fulda University (Fulda) and Florida Atlantic University (FAU). In this paper we provide the first RF analysis results done at SFSU on our full data set covering four years of our joint SE classes. These data include 74 student teams with over 380 students, totaling over 30000 discrete data points. These data are grouped into 11 time intervals, each measuring important phases of project development during the class (e.g. early requirement gathering and design, development, testing and delivery). We briefly elaborate on the methods of data collection and describe the data itself. We then show prediction results of the RF analysis applied to this full data set. Results show that we are able to detect student teams who are bound to fail or need attention in early class time with good (about 70%) accuracy. Moreover, the variable importance analysis shows that the features (TAM measures) with high predictive power make intuitive sense, such as late delivery/late responses, time used to help each other, and surprisingly statistics on commit messages to the code repository, etc. In summary, we believe we demonstrate the viability of using ML on objective and quantitative team activity measures to predict student learning of software engineering teamwork, and point to easy-to-measure factors that can be used to guide educators and software engineering managers to implement early intervention for teams bound to fail. Details about the project and the complete ML training database are downloadable from the project web site.

Work in progress — Elassys: Online tool for teamwork analysis and assessment in software engineering education

Teaching software engineering (SE) is now critical part of all major curricula in computer science programs. The ultimate goals of such programs include development of teamwork practices and techniques important for software (SW) development and application of modern SE practices and processes. One of the most challenging parts in these activities is the assessment process whose goal is to evaluate students achievement of those learning objectives, namely adherence to the software engineering process and their ability to develop adequate SW product. The challenges of the assessment process include the following: a) Fairness of the assessment process — reflection of individual performance and contribution as well as of ability of students to work in collaborative environment and support teamwork; b) Effectiveness, efficiency and ease of implementation; c) Fast and timely access to the relevant feedback about performance and collaboration of the students; d) Automation of the process and availability of the recorded data for further analysis. To address these questions we have designed a tool Elassys for assessment and analysis of teamwork and individual student performance in software engineering projects.

BioMedia: multimedia information systems for biology research, education, and collaboration

The long-term goals of the recently started Biomedia project at SFSU are to provide multimedia information systems and applications for the research and education needs of several projects in the SFSU Biology Department. These applications involve a considerable amount of images and image sequence data, in addition to traditional text, genomic, and experimental measurement data. Our systems will allow biology researchers and students to store, index, annotate, search, visualize, analyze, collaborate, and share a large amount of heterogeneous biomedical data. Our initial focus in BioMedia is the creation of collaborative WWW site for the Hedgehog gene pathway. The Hedgehog (Hh) protein super family constitutes a group of closely related secreted proteins that control many crucial processes during the embryogenesis of tissues. The overall goals of the Hh WWW Site project are as follows: a) to provide a WWW site to be used by researchers and students studying the Hedgehog gene pathway and made available to broad community, and b) to provide advanced and innovative functionality enabling easy usage and management, community based content submission and updates, and asynchronous collaboration between researchers and students. In this paper we present the status and first results in building and researching technologies necessary for this WWW site.