Theodore W. Frick

Observer Agreement and Reliabilities of Classroom Observational Measures

There appears to be a growing interest in the study of process variables in naturalistic classroom settings. Numerous observation systems have been developed and used in such investigations (Simon & Boyer, 1974). However, there is a corresponding paucity of published research that documents relationships between teaching-learning processes and educationally relevant pupil outcomes (Dunkin & Biddle, 1974; Heath & Nielson, 1974; Rosenshine, 1971; Smith, 1971; Soar, 1972). A reason for many insignificant findings may be that investigators have inadequately controlled or accounted for a number of sources of variance associated with observational studies (McGaw, Wardrop, & Bunda, 1972). The purpose of this paper is to clarify a number of common methodological errors made in observational studies and then to suggest practical means of minimizing such errors. Of primary concern are those errors introduced by observers and investigators that can limit reliabilities of observational measures. Prevalent confusion concerning the reliabilities of observational measures can be traced to the failure to separate two statistically related

User-centered design and usability testing of a web site: An illustrative case study

Administrators at a large midwestern university recognized that their World Wide Web site was rapidly becoming an important factor in recruiting new students. They also expected this Web site to serve many different types of information needs for existing students, faculty, staff, and alumni. an interdisciplinary team of faculty, graduate students, and staff was formed to evaluate the existing Web site. A group from this team first conducted a needs analysis to determine the kinds of information the target population was seeking. This analysis led to the creation of a new information structure for the Web site. Usability tests of the both the new and old designs were conducted on paper. Users were able to find answers to frequently asked questions much more rapidly and successfully with the new information structure. This structure was further refined through additional usability tests conducted on the Web itself. This descriptive case study illustrates the value of user-centered design and usability testing of World Wide Web sites.

Student Perceptions of Asynchronous Computer-Mediated Communication in Face-to-Face Courses

While there are many distance education studies of student satisfaction or perceptions of CMC, studies on residential student perceptions of CMC are rare. A paper survey was administered to 105 residential graduate and undergraduate students at a midwestern U.S. university. Results indicated that the majority of students preferred face-to-face discussion over CMC for most tasks; however, CMC was preferred overall for simple learning tasks. Content analyses of student responses to open-ended questions revealed that some students perceived face-to-face discussion to be faster, easier, and more convenient, while others perceived that CMC saves time and is more convenient. A discriminant analysis revealed several important factors that predicted those who preferred CMC for discussion. Students further commented that they would learn better from CMC if their instructors were more involved with and enthusiastic about CMC. Speed and convenience appear to be more important to students than whether discussion is face-to-face or CMC.

Changes in Student Motivation during Online Learning.

Self-directed e-learning (SDEL) refers to electronic learning environments where there are often no peer learners or instructors regularly available. Past studies suggest that lack of time and lack of motivation are primary causes of learner attrition in online settings. However, little is known about what influences motivational change during SDEL. We surveyed 368 adult learners from both higher education and corporate settings who had used commercial SDEL products. Results from stepwise regression analysis indicated that the best predictors of motivation to begin SDEL were perceived relevance, reported technology competence, and age. The best predictors of motivation during SDEL were perceived quality of instruction and learning (e-learning is right for me) and motivation to begin. Motivation during SDEL was the best predictor of positive change in motivation, which in turn predicted learner satisfaction with SDEL. Instructional design principles for sustaining learner motivation in SDEL are identified from the findings of the present study.

Computerized adaptive testing in instructional settings

Item response theory (IRT) has most often been used in research on computerized adaptive testing (CAT). Depending on the model used, IRT requires between 200 and 1,000 examinees for estimating item parameters. Thus, it is not practical for instructional designers to develop their own CAT based on the IRT model. Frick improved Walds sequential probability ratio test (SPRT) by combining it with normative expert systems reasoning, referred to as an EXSPRT-based CAT. While previous studies were based on re-enactments from historical test data, the present study is the first to examine how well these adaptive methods function in a real-time testing situation. Results indicate that the EXSPRT-I significantly reduced test lengths and was highly accurate in predicting mastery. EXSPRT is apparently a viable and practical alternative to IRT for assessing mastery of instructional objectives.

Analysis of Patterns in Time: A Method of Recording and Quantifying Temporal Relations in Education:

Analysis of patterns in time (APT) is a method for gathering information about observable phenomena such that probabilities of temporal patterns of events can be estimated empirically. If appropriate sampling strategies are employed, temporal patterns can be predicted from APT results. As an example of the fruitfulness of APT, it was discovered in a classroom observational study that elementary students were on task 97% of the time if some form of direct instruction was occurring also, whereas they were on task only 57% of the time during nondirect instruction. As a second example, APT results were used as a rule base for an expert system in adaptive computer-based testing. When two different computer tests were studied, average samples of 9 and 13 test items were required to make mastery and nonmastery decisions when items were selected at random. These decisions were, respectively, 94% and 98% accurate compared to those reached from two much larger test item pools. Finally, APT is compared to the linear models approach and event history analysis. The major difference is that in APT there is no mathematical model assumed to characterize relations among variables. In APT the model is the temporal pattern being investigated.

Artificial Tutoring Systems: What Computers Can and Can't Know.

After more than four decades, development of artificially intelligent tutoring systems has been constrained by two interrelated problems: knowledge representation and natural language understanding. G. S. Maccias epistemology of intelligent natural systems implies that computer systems will need to develop qualitative intelligence before these problems can be solved. Recent research on how human nervous systems develop provides evidence for the significance of qualitative intelligence. Qualitative intelligence is required for understanding of culturally bound meanings of signs used in communication among intelligent natural systems. S. I. Greenspan provides neurological and clinical evidence that emotion and sensation are vital to the growth of mind—capabilities that computer systems do not currently possess. Therefore, we must view computers in education as media through which a multitude of teachers can convey their messages. This does not mean that the role of classroom teachers is diminished. Teachers and students can be empowered by these additional learning resources.

Bayesian Adaptation during Computer-Based Tests and Computer-Guided Practice Exercises

One of the potential advantages of computer-based instruction (CBI) is individualization of instruction. However, this goal has not been fully realized in practice, due largely to limitations of natural language understanding and to combinatorial explosion. It is nonetheless possible to develop CBI programs which can adapt to students, depending on their performance, by adjusting the length of computer-guided practice exercises and computer-based tests. The validity of this approach is supported empirically. The number of questions can be significantly reduced for many individuals, while mastery and nonmastery decisions remain highly accurate.

Instructional Illustrations: Intended Meanings and Learner Interpretations

Abstract Instructional illustrations are widely used in textbooks and have been shown to have the potential to aid learning. However, illustrations that are not understood as their designers intend them to he may waste resources at best and interfere with learning at worst. Learners may recognize images but not understand illustrations when their meanings arc extended by the use of graphical devices. This study examines the interpretations made by 471 participants from 2 countries, 3“’ grade through adult, of simple instructional illustrations. The extent to which their interpretations match the intended meanings of the illustration designers and the characteristics oj their responses are reported.

Computerized Adaptive Mastery Tests as Expert Systems

Expert systems can be used to aid decision making. A computerized adaptive test is one kind of expert system, though not commonly recognized as such. A new approach, termed EXSPRT, was devised that combines uncertain inference in expert systems with sequential probability ratio test stopping rules. Two versions of EXSPRT were developed, one with random selection of items (EXSPRT-R) and one with intelligent selection (EXSPRT-I). Two empirical studies were conducted in which these two new methods were compared to the traditional SPRT and to an adaptive mastery testing (AMT) approach based on item response theory (IRT). The EXSPRT-I tended to be more efficient than the AMT, EXSPRT-R and SPRT models in terms of average test lengths. Although further research is needed, the EXSPRT-I initially appears to be a strong alternative to both IRT- and SPRT-based adaptive tests for making categorical decisions about examinee mastery of single instructional objectives. The EXSPRT-I is clearly less complex than IRT, both conceptually and mathematically. It also appears to require many fewer examinees to establish empirically a rule base when compared to the large numbers required to estimate parameters for item response functions in the IRT model.