Mingyu Feng

Addressing the assessment challenge with an online system that tutors as it assesses

Secondary teachers across the United States are being asked to use formative assessment data (Black and Wiliam 1998a,b; Roediger and Karpicke 2006) to inform their classroom instruction. At the same time, critics of US government’s No Child Left Behind legislation are calling the bill “No Child Left Untested”. Among other things, critics point out that every hour spent assessing students is an hour lost from instruction. But, does it have to be? What if we better integrated assessment into classroom instruction and allowed students to learn during the test? We developed an approach that provides immediate tutoring on practice assessment items that students cannot solve on their own. Our hypothesis is that we can achieve more accurate assessment by not only using data on whether students get test items right or wrong, but by also using data on the effort required for students to solve a test item with instructional assistance. We have integrated assistance and assessment in the ASSISTment system. The system helps teachers make better use of their time by offering instruction to students while providing a more detailed evaluation of student abilities to the teachers, which is impossible under current approaches. Our approach for assessing student math proficiency is to use data that our system collects through its interactions with students to estimate their performance on an end-of-year high stakes state test. Our results show that we can do a reliably better job predicting student end-of-year exam scores by leveraging the interaction data, and the model based on only the interaction information makes better predictions than the traditional assessment model that uses only information about correctness on the test items.

Online Mathematics Homework Increases Student Achievement

In a randomized field trial with 2,850 seventh-grade mathematics students, we evaluated whether an educational technology intervention increased mathematics learning. Assigning homework is common yet sometimes controversial. Building on prior research on formative assessment and adaptive teaching, we predicted that combining an online homework tool with teacher training could increase learning. The online tool ASSISTments (a) provides timely feedback and hints to students as they do homework and (b) gives teachers timely, organized information about students’ work. To test this prediction, we analyzed data from 43 schools that participated in a random assignment experiment in Maine, a state that provides every seventh-grade student with a laptop to take home. Results showed that the intervention significantly increased student scores on an end-of-the-year standardized mathematics assessment as compared with a control group that continued with existing homework practices. Students with low prior mathematics achievement benefited most. The intervention has potential for wider adoption.

Can we get better assessment from a tutoring system compared to traditional paper testing? can we have our cake (better assessment) and eat it too (student learning during the test)?

Dynamic assessment (DA) has been advocated as an interactive approach to conducting assessments to students in the learning systems Sternberg and others proposed to give students tests to see how much assistance it takes a student to learn a topic; and to use as a measure of their learning gain To researchers in the ITS community, it comes as no surprise that measuring how much assistance a student needs to complete a task successfully is probably a good indicator of this lack of knowledge However, a cautionary note is that conducting DA takes more time than simply administering regular test items to students In this paper, we report a study analyzing 40-minutes data of totally 1,392 students from two school years The result suggests that for the purpose of assessing student performance, it is more efficient for students to take DA than just having practice items.

A Web-based Authoring Tool for Intelligent Tutors: Blending Assessment and Instructional Assistance

Middle school mathematics teachers are often forced to choose between assisting students’ development and assessing students’ abilities because of limited classroom time available. To help teachers make better use of their time, a web-based system, called the Assistment system, was created to integrate assistance and assessment by offering instruction to students while providing a more detailed evaluation of their abilities to the teacher than is possible under current approaches. An initial version of the Assistment system was created and used in May, 2004 with approximately 200 students and over 1000 students currently use it once every two weeks. The hypothesis is that Assistments can assist students while also assessing them. This chapter describes the Assistment system and some preliminary results.

Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models in an Intelligent Tutoring System

Student modeling and cognitive diagnostic assessment are important issues that need to be addressed for the development and successful application of intelligent tutoring systems (ITS). ITS needs the construction of complex models to represent the skills that students are using and their knowledge states, and practitioners want cognitively diagnostic information at a finer grained level. Traditionally, most assessments treat all questions on the test as sampling a single underlying knowledge component. Can we have our cake and eat it, too? That is, can we have a good overall prediction of a high stakes test, while at the same time be able to tell teachers meaningful information about fine-grained knowledge components? In this paper, we introduce an online intelligent tutoring system that has been widely used. We then present some encouraging results about a fine-grained skill model with the system that is able to predict state test scores. This model allows the system track about 106 knowledge components for eighth grade math. In total, 921 eighth grade students were involved in the study. We show that our fine-grained model could improve prediction compared to other coarser grained models and an IRT-based model. We conclude that this intelligent tutoring system can be a good predictor of performance.

Implementation of an Intelligent Tutoring System for Online Homework Support in an Efficacy Trial

Much research has been done on the development of an intelligent tutoring system (ITS), and small empirical studies have demonstrated the effectiveness of ITS at promoting student learning. However, large-scale implementation of ITS in school settings has not been researched thoroughly. In this paper, we describe an ongoing randomized controlled trial (RCT) to evaluate the efficacy of a web-based tutoring system—the ASSISTments—as support for homework. The program is used in 46 middle schools in the state of Maine, to provide immediate feedback to students, and to provide reports to teachers to support homework review and instruction adaptation. We describe the challenges for the RCT, approaches used to understand implementation of the system, and findings on how the system is being used.

Using data mining findings to aid searching for better cognitive models

One key component of creating an intelligent tutoring system is forming a model that monitors student behavior Researchers in machine learning area have been using automatic/semi-automatic techniques to search for cognitive models One of the semi-automatic approaches is learning factor analysis, which involves human making hypothesis and identifying difficulty factors in the related items In this paper, we propose a hybrid approach in which we leverage findings from our previous educational data mining work to aid the search for a better cognitive model and thus, improve the efficiency of LFA Preliminary results suggest that our approach can lead to significantly better fitted cognitive models fast.

Adaptive Learning Goes to China

Adaptive learning, by definition, adjusts the content and guidance offered to individual learners. Studies have shown that adaptive systems can be effective learning tools. This paper introduces an adaptive learning system, “Yixue,” that was developed and deployed in China. It diagnostically assesses students’ mastery of fine-grained skills and presents them with instructional content that fits their characteristics and abilities. The Yixue system has been used by over 10,000 students in 17 cities in China for learning 12 subjects in middle school in 2017. The hypothesis is that the Yixue adaptive learning system will improve student learning outcomes compared to other learning systems. This paper describes major features of the Yixue system. A learning analysis of 1,355 students indicates that students learned from using the Yixue system and the results can generalize across students and skills. We also report a study that evaluates the efficacy of the Yixue math program in 8th and 9th grade.