Theodoros Mastoras

Extending LMS to Support IRT-Based Assessment Test Calibration

Developing unambiguous and challenging assessment material for measuring educational attainment is a time-consuming, labor-intensive process. As a result Computer Aided Assessment (CAA) tools are becoming widely adopted in academic environments in an effort to improve the assessment quality and deliver reliable results of examinee performance. This paper introduces a methodological and architectural framework which embeds a CAA tool in a Learning Management System (LMS) so as to assist test developers in refining items to constitute assessment tests. An Item Response Theory (IRT) based analysis is applied to a dynamic assessment profile provided by the LMS. Test developers define a set of validity rules for the statistical indices given by the IRT analysis. By applying those rules, the LMS can detect items with various discrepancies which are then flagged for review of their content. Repeatedly executing the aforementioned procedure can improve the overall efficiency of the testing process.

Performance Evaluation of the Small Sample Dichotomous IRT Analysis in Assessment Calibration

Item Response Theory (IRT) provides the accepted framework for examining student responses to individual test items so as to assess their quality. Moreover, IRT proves especially valuable in both improving items, which will be reused in future tests, and eliminating the ambiguous or misleading ones. However, to ensure all IRT parameters are correctly estimated, every single item needs to be tested on a large number of examinees to define its properties. As a result, IRT tends to be shunned by teaching staff who only have access to a relatively small number of students. Nevertheless, the accuracy of parameter estimates is of lesser importance in assessment calibration, when items whose parameters exceed a threshold value are flagged for revision. This study uses simulated data sets under various simulation conditions and introduces two new quality indices together with their respective IRT goodness-of-fit tests as a means to explore the feasibility of applying IRT-based assessment calibration to small sample sizes.

LMS assessment: using IRT analysis to detect defective multiple-choice test items

Due to the computerisation of assessment tests, the use of Item Response Theory IRT has become commonplace for educational assessment development, evaluation and refinement. When used appropriately by a Learning Management System LMS, IRT can improve the assessment quality, increase the efficiency of the testing process and provide in-depth descriptions of item properties. This paper introduces a methodological and architectural framework which embeds an IRT analysis tool in an LMS so as to extend its functionality with assessment optimisation support. By applying a set of validity rules to the statistical indices produced by the IRT analysis, the enhanced LMS is able to detect several defective items from an item pool which are then reported for reviewing of their content. Assessment refinement is achieved by repeatedly employing this process until all flawed items are eliminated.