Colin Higgins

Automated assessment and experiences of teaching programming

This article reports on the design, implementation, and usage of the CourseMarker (formerly known as CourseMaster) courseware Computer Based Assessment (CBA) system at the University of Nottingham. Students use CourseMarker to solve (programming) exercises and to submit their solutions. CourseMarker returns immediate results and feedback to the students. Educators author a variety of exercises that benefit the students while offering practical benefits. To date, both educators and students have been hampered by CBA software that has been constructed to assess text-based or multiple-choice answers only. Although there exist a few CBA systems with some capability to automatically assess programming coursework, none assess Java programs and none are as flexible, architecture-neutral, robust, or secure as the CourseMarker CBA system.

The CourseMarker CBA System: Improvements over Ceilidh

This document reports on the results of re-designing and re-implementing the Ceilidh courseware system. It highlights the limitations identified in the thirteen years of Ceilidhs use at the University of Nottingham. It also illustrates how most of these limitations have been resolved by re-designing Ceilidhs architecture and improving various aspects of the marking and administrating processes. The new system, entitled CourseMarker, offers enhanced functionality by adding useful features that have long been needed by Ceilidhs community. The paper concludes with an evaluation of the changes and a brief report on the experience of CourseMarkers use over the last three years. Finally, recent developments and future directions are discussed.

Off-line Recognition of Handwritten Arabic Words Using Multiple Hidden Markov Models

Hidden Markov Models (HMM) have been used with some success in recognising printed Arabic words. In this paper, a complete scheme for unconstrained Arabic handwritten word recognition based on a Multiple discriminant Hidden Markov Models is presented and discussed. The overall engine of this combination of a global feature scheme with an HMM module, is a system able to classify Arabic-Handwritten words and has been tested on one hundred different writers. The system first attempts to remove some of the variation in the images that do not affect the identity of the handwritten word. Next, the system codes the skeleton and edge of the word such that feature information about the strokes in the skeleton is extracted. Then, a classification process based on a rule based classifier is used as that a global recognition engine to classify words into eight groups. Finally, for each group, the HMM approach is used for trial classification. The output is a word in the lexicon. A detailed experiment has been carried out, and successful recognition results are reported.

A data base for Arabic handwritten text recognition research

In this paper we present a new database for off-line Arabic handwriting recognition, together with associated preprocessing procedures. We have developed a new database for the collection, storage and retrieval of Arabic handwritten text (AHDB). This is an advance both in terms of the size of the database as well as the number of different writers involved. We further designed an innovative, simple yet powerful, in place tagging procedure for our database. It enables us to easily extract the bitmaps of words. We also constructed a preprocessing class, which contains some useful preprocessing operations. In this paper the most popular words in Arabic writing were identified for the first time, using an associated program.

Formative computer based assessment in diagram based domains

This paper presents an approach to conducting formative assessment of student coursework within diagram-based domains using Computer Based Assessment (CBA) technology. Formative assessment is perceived as a resource-intensive assessment mode and its usage is in steep decline in higher education. CBA technology developed out of the desire to automate assessment due to the necessity of assessing students with decreasing unit-resource; it can overcome the decline in formative assessment by automating those processes which are considered resource-intensive.The system described is based upon the CourseMarker CBA system (formerly CourseMaster / Ceilidh) and the DATsys object-oriented framework for CBA-oriented diagram editors. This paper outlines requirements for obtaining good formative assessment using CBA software and documents a live system which assessed student Entity Relationship diagrams within an undergraduate Database Systems course. Results are presented and considerable extensions proposed.

Off-line recognition of handwritten Arabic words using multiple hidden Markov models

A complete scheme for unconstrained Arabic handwritten word recognition based on a multiple hidden Markov models (HMM) is presented. The overall engine of this combination of a global feature scheme with a HMM module, is a system able to classify Arabic-handwritten words. The system first removes some of the variation in the images. Next, it codes the skeleton and edge of the word such that features are extracted. Then, a rule-based classifier is used as a global recognition engine. Finally, for each group, the HMM approach is used for trial classification. The output is a word in the lexicon

Preface to the special issue on automated assessment of programming assignments

Programming computers is becoming an increasingly popular activity, not only for computer science students but across a large number of other disciplines. But funding and other resources for teaching programming have not, for the most part, kept pace with the continuous growth in the number of students studying programming. Academic institutions face the challenge of providing their students with better quality teaching while minimizing the amount of additional work for staff. While traditional teaching methods can be enhanced by audio/visual means and advances in online learning, in the area of assessment the problems continue to persist. Computer-based assessment (CBA), which over the years has become an increasingly important teaching tool, can help educators solve these problems. For the past twenty years, educators have reported on the practical and pedagogic benefits of using automated assessment tools to assess student coursework in programming. The purpose of this issue of JERIC is to explain cutting-edge research on the automated assessment of student programs, and other kinds of programming assignments, to a wider audience. It is devoted to both fully automated assessment and to partial student program assessment by machine. We do not provide an overview of the field in this introductory statement. A recent survey by Ala-Mutka [Ala-Mutka 2005], the review of Douce et al. in this issue [Douce et al. 2006], and several other articles in this issue provide a comprehensive overview and offer various ways to classify the multitude of known work. However, to introduce the articles in this issue, the editors found it useful to distinguish three categories of systems: (1) to assess program-tracing skills; (2) to assess program-writing skills; and (3) assess intelligent programming tutors. The first group of systems attempts to assess students’ knowledge of programming language semantics by presenting students with a program and asking them to trace it. The answer to this type of problem results from its execution: What was printed? What was the final state of the variables in the data structures? The ability to automatically evaluate student answers stems from the system’s ability to execute the program or the algorithm with the same data and compare that result with the one entered by the student. The QuizPACK system reported by Brusilovsky and Sosnovsky [2006] in this issue _________________________________________________________________________________________

A Database for Arabic Printed Character Recognition

Electronic Document Management (EDM) technology is being widely adopted as it makes for the efficient routing and retrieval of documents. Optical Character Recognition (OCR) is an important front end for such technology. Excellent OCR now exists for Latin based languages, but there are few systems that read Arabic, which limits the penetration of EDM into Arabic-speaking countries. In developing an OCR system for Arabic it is necessary to create a database of Arabic words. Such a database has many uses as well as in training and testing a recognition system. This paper provides a comprehensive study and analysis of Arabic words and explains how such a database was constructed. Unlike earlier studies, this paper describes a database developed using a large number of collected Arabic words (6 million). It also considers connected segments or Pieces of Arabic Words (PAWs) as well as Naked Pieces of Arabic Word (NPAWs); PAWS without diacritics. Background information concerning the Arabic language is also presented.

Building a multi-modal Arabic corpus (MMAC)

Traditionally, a corpus is a large structured set of text, electronically stored and processed. Corpora have become very important in the study of languages. They have opened new areas of linguistic research, which were unknown until recently. Corpora are also key to the development of optical character recognition (OCR) applications. Access to a corpus of both language and images is essential during OCR development, particularly while training and testing a recognition application. Excellent corpora have been developed for Latin-based languages, but few relate to the Arabic language. This limits the penetration of both corpus linguistics and OCR in Arabic-speaking countries. This paper describes the construction and provides a comprehensive study and analysis of a multi-modal Arabic corpus (MMAC) that is suitable for use in both OCR development and linguistics. MMAC currently contains six million Arabic words and, unlike previous corpora, also includes connected segments or pieces of Arabic words (PAWs) as well as naked pieces of Arabic words (NPAWs) and naked words (NWords); PAWs and Words without diacritical marks. Multi-modal data is generated from both text, gathered from a wide variety of sources, and images of existing documents. Text-based data is complemented by a set of artificially generated images showing each of the Words, NWords, PAWs and NPAWs involved. Applications are provided to generate a natural-looking degradation to the generated images. A ground truth annotation is offered for each such image, while natural images showing small paragraphs and full pages are augmented with representations of the text they depict. A statistical analysis and verification of the dataset has been carried out and is presented. MMAC was also tested using commercial OCR software and is publicly and freely available.

A cursive script-recognition system based on human reading models

The human reading process is undoubtedly extremely complex; however, much work has been carried out in determining possible mechanisms behind it. A computer recognition system that makes use of some of the proposed models of human reading has been developed at the University of Nottingham. With it, we attempt to solve the problem of recognising handwriting on-line. The system, called NuScript, is based on the blackboard paradigm of artificial intelligence (AI). It initially uses easily extracted features to reduce a large lexicon to a smaller list of candidate words. Later stages use increasingly sophisticated knowledge sources, based on a diverse set of AI paradigms and other pattern-recognition techniques, to determine and subsequently refine a confidence value for each candidate. A description of the elements of the human recognition models on which the system is based is followed by a general description of the computer recognition system as a whole.