Bernard P. Veldkamp

MULTIDIMENSIONAL ADAPTIVE TESTING WITH CONSTRAINTS ON TEST CONTENT

The case of adaptive testing under a multidimensional response model with large numbers of constraints on the content of the test is addressed. The items in the test are selected using a shadow test approach. The 0–1 linear programming model that assembles the shadow tests maximizes posterior expected Kullback-Leibler information in the test. The procedure is illustrated for five different cases of multidimensionality. These cases differ in (a) the numbers of ability dimensions that are intentional or should be considered as “nuisance dimensions” and (b) whether the test should or should not display a simple structure with respect to the intentional ability dimensions.

A Model for the Construction of Country-Specific Yet Internationally Comparable Short-Form Marketing Scales

In the last few decades, the measurement of marketing constructs has improved tremendously. Our discipline has also started to systematically catalogue our measurement knowledge in influential handbooks of marketing scales. However, at least two important issues remain. First, existing scales are often too long for administration in nonstudent samples or in applied studies. Second, existing (U.S.-developed) scales may contain items that are not informative about the underlying construct in particular countries, whereas relevant items tapping into local cultural expressions of the construct in question may be missing. To address these issues, we propose a new model that yields country-specific yet fully cross-nationally comparable short forms of unidimensional marketing scales. The procedure is based on hierarchical item response theory and optimal test design. The procedure is flexible in the sense that the researcher can specify various constraints on item content, scale length, and measurement precision. Because our procedure allows inclusion of country-specific (or “emic”) items in standardized (or “etic”) scales, it presents an important step toward resolving the emic-etic dilemma that has plagued international marketing research for decades.

Conditional Item-Exposure Control in Adaptive Testing Using Item-Ineligibility Probabilities:

Two conditional versions of the exposure-control method with item-ineligibility constraints for adaptive testing in van der Linden and Veldkamp (2004) are presented. The first version is for unconstrained item selection, the second for item selection with content constraints imposed by the shadow-test approach. In both versions, the exposure rates of the items are controlled using probabilities of item ineligibility given θ that adapt the exposure rates automatically to a goal value for the items in the pool. In an extensive empirical study with an adaptive version of the Law School Admission Test, the authors show how the method can be used to drive conditional exposure rates below goal values as low as 0.025. Obviously, the price to be paid for minimal exposure rates is a decrease in the accuracy of the ability estimates. This trend is illustrated with empirical data.

Attention paid to feedback provided by a computer-based assessment for learning on information literacy

Three studies are presented on attention paid to feedback provided by a computer-based assessment for learning on information literacy. Results show that the attention paid to feedback varies greatly. In general the attention focuses on feedback of incorrectly answered questions. In each study approximately fifty percent of the respondents paid attention to feedback of incorrect answers only. Approximately another twenty-five percent did not pay attention to feedback at all. Results suggest that differences in attention paid to feedback are influenced by task difficulty and test length. Supervision, however, does not seem to influence the average attention paid to feedback. On the other hand, results show that indirect and direct supervision lead to a greater impact of feedback provided by a computer-based assessment for learning as the number increases of students taking the test and, as a consequence, paying attention to feedback.

Designing Item Pools for Computerized Adaptive Testing

In existing computerized adaptive testing (CAT) programs, each successive item in the test is chosen to optimize an objective function. Examples of well-known objectives in CAT are maximizing the information in the test at the ability estimate for the examinee and minimizing the deviation of the information in the test from a target value at this estimate. In addition, item selection is required to realize a set of content specifications for the test. For example, item content may be required to follow a certain taxonomy, or, if the items have a multiple-choice format, their answer key distribution should deviate not too much from uniformity. Content specifications are generally defined in terms of combinations of attributes the items in the test should have. They are typically realized by imposing a set of constraints on the item-selection process. The presence of an objective function and constraints in CAT leads to the notion of CAT as constrained (sequential) optimization. For a more formal introduction to this notion, see van der Linden (2000).

Why work?: Aligning foci and dimensions of commitment along the axes of the competing values framework

Purpose – This paper aims to gain insight into the psychological dynamics of work motivation; specific work values are related to foci and dimensions of commitment to understand how different values influence peoples commitments. Design/methodology/approach – A survey from a hospital (n=222) was analyzed on the relationships between organizational values, individual values, affective and normative commitment on both an organizational and an occupational level. Findings – Results supported the proposition that dimensions of commitment can be aligned along the flexibility‐control axis of the competing values framework (CVF) and that the foci of commitment can be aligned along the internal‐external axis of the CVF but, in line with recent findings, congruence effects were absent. Research limitations/implications – Although small effect sizes were found, and results based on self‐report have to be handled cautiously, the hypothesized pattern was found for affective and normative organizational and occupational commitment. This gives insights into the values associated with these types of commitments. Practical implications – Specific values which are found to be related to specific kinds of commitment can direct strategies for influencing commitments in organizational contexts. Originality/value – As specific values are linked to specific commitments, practitioners gain insight into which values may lead to a specific kind of commitment, enabling them to develop strategies for enhancing commitment, and academics see how the CVF corresponds to newly developed insights into commitment research.

Optimizing Balanced Incomplete Block Designs for Educational Assessments

A popular design in large-scale educational assessments as well as any other type of survey is the balanced incomplete block design. The design is based on an item pool split into a set of blocks of items that are assigned to sets of “assessment booklets.” This article shows how the problem of calculating an optimal balanced incomplete block design can be formulated as a problem in combinatorial optimization. Several examples of structural and practical requirements for balanced incomplete block designs are shown to be linear constraints on the optimization problem. In addition, a variety of possible objective functions to optimize the design is discussed. The technique is demonstrated using the 1996 Grade 8 Mathematics National Assessment of Educational Progress (NAEP) as a case study.

An integer programming approach to item bank design

An integer programming approach to item bank design is presented that can be used to calculate an optimal blueprint for an item bank, in order to support an existing testing program. The results are optimal in that they minimize the effort involved in producing the items as revealed by current item writing patterns. Also presented is an adaptation of the models, which can be used as a set of monitoring tools in item bank management. The approach is demonstrated empirically for an item bank that was designed for the Law School Admission Test.

Assembling a computerized adaptive testing item pool as a set of linear tests

Test-item writing efforts typically results in item pools with an undesirable correlational structure between the content attributes of the items and their statistical information. If such pools are used in computerized adaptive testing (CAT), the algorithm may be forced to select items with less than optimal information, that violate the content constraints, and/or have unfavorable exposure rates. Although at first sight somewhat counterintuitive, it is shown that if the CAT pool is assembled as a set of linear test forms, undesirable correlations can be broken down effectively. It is proposed to assemble such pools using a mixed integer programming model with constraints that guarantee that each test meets all content specifications and an objective function that requires them to have maximal information at a well-chosen set of ability values. An empirical example with a previous master pool from the Law School Admission Test (LSAT) yielded a CAT with nearly uniform bias and mean-squared error functions for the ability estimator and item-exposure rates that satisfied the target for all items in the pool.

The effect of regulation feedback in a computer-based formative assessment on information problem solving

This study examines the effect of regulation feedback in a computer-based formative assessment in the context of searching for information online. Fifty 13-year-old students completed two randomly selected assessment tasks, receiving automated regulation feedback between them. Student performance was (self-)graded by students and by experts. Expert, as well as student (self)grades showed a significant increase between Task 1 and Task 2. However, further analysis of the expert grades showed significant improvement in performance for girls only. Furthermore, the formative assessment system traced the number of searches and the number of websites consulted per student to complete the two assignments. On average, the results showed that students consulted significantly more websites for Task 2, compared to Task 1. The average number of searches did not differ significantly between Tasks 1 and 2. On the other hand, significant differences were found for those students who, during the evaluation of their performance on Task 1, explicitly stated that they would increase their searches. We examined the effect of regulation feedback in computer-based formative assessment on behaviour and performance.Focus on regulation feedback in Digital Information Skills Measurement tool (DIM).DIM traces student information problem solving behaviour.Applied learning analytics indicates behavioural changes.Performance improvement observed for female respondents only.