Chung Keung Poon

Toward More Robust Automatic Analysis of Student Program Outputs for Assessment and Learning

Automated analysis and assessment of students programs, typically implemented in automated program assessment systems (APASs), are very helpful to both students and instructors in modern day computer programming classes. The mainstream of APASs employs a black-box testing approach which compares students program outputs with instructor-prepared outputs. A common weakness of existing APASs is their inflexibility and limited capability to deal with admissible output variants, that is, outputs produced by acceptable correct programs that differ from the instructors. This paper proposes a more robust framework for automatically modelling and analysing student program output variations based on a novel hierarchical program output structure called HiPOS. Our framework assesses student programs by means of a set of matching rules tagged to the HiPOS, which produces a better verdict of correctness. We also demonstrate the capability of our framework by means of a pilot case study using real student programs.

Automatic Assessment via Intelligent Analysis of Students' Program Output Patterns.

Automatic assessment of computer programming exercises offers a number of benefits to both learners and educators, including timely and customised feedback, as well as saving of human effort in grading. However, due to the high variety of programs submitted by students, exact matching between the expected output and different output variants is undesirable and how to do the matching properly is a challenging and practical problem. Existing approaches to address this problem adopt various inexact matching strategies, but typically they are unscalable, incapable of expressing a diversity of program outputs, or require high level of expertise. In this paper, we propose Hierarchical Program Output Structure (HiPOS), which provides higher expressiveness and flexibility, to model the program output. Based on HiPOS, we design different levels of matching rules in the matching rule hierarchy to determine the admissible program output variants in a flexible and scalable manner. We conducted experiments and compare our approach of automatic assessment to human judgement. The results show that our proposed method achieved an accuracy of 0.8467 in determining the admissible program output variants.

An automatic approach for discovering skill relationship from learning data

We have developed a method called skill2vec, which applies big data techniques to automatically analyze the learning data to discover skill relationship, leading to a more objective and data-informed decision making. Skill2vec is a neural network architecture which can transform a skill to a new vector space called embedding. The embedding can facilitate the comparison and visualization of different skills and their relationship. We conducted a pilot experiment using benchmark dataset to demonstrate the effectiveness of our method.

Comparative study on heterogeneous profiling sources for second language learners

Stimulated by the arrival of the big data era, various and heterogeneous data sources such as data in social networks, mobile devices and sensor data for users have emerged, mirroring characteristics and preferences of data owners. These data sources are often used to construct user profiles so as to facilitate personalized services like recommendations or personalized data access. In the context of second language learning, learner data involve learning logs, standard test results, and individual learning preferences and styles. Given its attribute of reflecting the characteristics of learners, such data can be exploited to build the learner profiles. However, these data sources possibly include noises or bias, and hence influence the reliability of the correspondingly constructed learner profiles. Consequently, the inaccurate profiles may result in ineffective learning tasks that are generated by e-Learning systems. To tackle this issue, it is significant and critical to evaluate the accuracy of learner profiles. In a response to this call, we propose a novel metric named “profile mean square error” to examine the accuracy of learner profiles founded upon diverse sources. We also demonstrate how to construct various learner profiles though applying different data sources such as learning logs, standard test results, and personal learning preferences in e-Learning systems and pedagogical activities. Moreover, we conduct an experimental study among some second language learners, the results of which illustrate that the most accurate profiles are generated from multiple data sources if they are integrated in a rational way.