Archive | 2021
Virtual Agents for Real Teachers: Applying AI to Support Professional Development of Proportional Reasoning
Abstract
Despite the critical role of teachers in the educational process, few advanced learning technologies have been developed to support teacher-instruction or professional development. This lack of support is particularly acute for middle school math teachers, where only 37% felt well prepared to scaffold instruction to address the needs of diverse students in a national sample. To address this gap, the Advancing Middle School Teachers’ Understanding of Proportional Reasoning project is researching techniques to apply pedagogical virtual agents and dialog-based tutoring to enhance teachers’ content knowledge and pedagogical content knowledge. This paper describes the design of a conversational, agent-based intelligent tutoring system to support teachers’ professional development. Pedagogical strategies are presented that leverage a virtual human facilitator to tutor pedagogical content knowledge (how to teach proportions to students), as opposed to content knowledge (understanding proportions). The roles for different virtual facilitator capabilities are presented, including embedding actions into virtual agent dialog, open-response versus choice-based tutoring, ungraded pop-up sub-activities (e.g. whiteboard, calculator, notetaking). Usability feedback for a small cohort of instructors pursuing graduate studies was collected. In this feedback, teachers rated the system ease of use and perceived usefulness moderately well, but also reported confusion about what to expect from the system in terms of flow between lessons and support by the facilitator. A key strategy for addressing students’ poor mathematics achievement is to improve professional learning opportunities for middle school teachers. Research indicates that teachers’ content and pedagogical content knowledge could be improved through professional development (CopurGencturk, Plowman, and Bai 2019) and that targeting both content knowledge and pedagogical content knowledge is more successful than either of them in isolation (Scher and O’Reilly 2009). Furthermore, gains in teachers’ content and pedagogical content knowledge have been linked to improvements in mathematics instruction that are associated with students’ mathematics achievement (Blazar 2015). Thus, professional development (PD) designed to enhance both content and pedagogical content knowledge has the poCopyright © 2021by the authors. All rights reserved. tential to improve the quality of mathematics instruction, which will improve students’ mathematics understanding. To research this issue, the Advancing Teachers Understanding of Proportional Reasoning (ATProportion) project is developing and testing a personalized, online professional development program for middle school teachers. As adult learners and as experts in education, teachers may react differently to virtual facilitator pedagogy which means that usability factors are still being established. In this paper, we describe the design and preliminary formative testing of the ATProportion system for professional development of proportional reasoning. When presenting the design, we focus on two aspects. First, iterating on the virtual facilitator pedagogy has been a central effort of the project, building capabilities that can be composed into distinct pedagogical patterns that could generalize to other domains. Second, developing content has also been a core focus, because math educational experts on our team require a high degree of control over content to make effective professional development training. As such, the content development process has been refined to help math experts directly author content wherever possible. Next, we present data from the formative testing. As this work is over a small sample and teachers are a diverse population, the results are not conclusive but they identify areas to explore further in the future. Background and Rationale The scalability of successful online professional development is constrained by the availability of quality interaction between users and facilitators (Pianta et al. 2008). These constraints can result in inadequate feedback and shallow discussion. In this work, we investigate virtual agents as a solution to overcome these problems. To support online professional development, a virtual facilitator must play both a pedagogical and a motivational role. Research indicates that virtual agents have been effective in both roles in various contexts (Schroeder, Adesope, and Gilbert 2013), but to our knowledge it has not yet been assessed in the context of a computer-based professional development program. This gap is important, because teachers are a special category of learner for two reasons. First, teachers must learn master both domain knowledge (the skills themselves) and also pedagogical domain knowledge (how to teach the skills, how different kinds of students understand a topic, etc.). Second, in-service teachers are adult professional learners, which is a stage that is overall not wellstudied by research on virtual agents or tutoring systems. Research on virtual agents to support learning, and mathematics learning in particular, have shown a number of key advantages over a more traditional “faceless” online system. First, virtual facilitators have been used to optimize learning from examples (Atkinson 2002), and professional development is taught through examples or case studies across many domains. Second, multimedia agents who communicate with learners via speech can increase learning gains (Schroeder, Adesope, and Gilbert 2013). This is relevant because high-quality mathematics instruction often talks through multiple steps of worked example (e.g., solving a word problem). Third, virtual agents can provide continuity and personalized support for learners as they learn from many different objects of joint attention, such as concept maps, equations, videos, and figures (Nye et al. 2018). An agent to provide continuity is especially relevant for mathematics instruction, where deep conceptual understanding often requires teaching multiple representations that may initially appear unrelated. This work also leverages virtual agents to deliver a dialogbased intelligent tutoring system (ITS). Open-response dialog-based ITS such as AutoTutor have shown learning gains on the order of 0.8σ (Nye, Graesser, and Hu 2014). While such ITS content is more time-consuming to develop, these resources have been particularly effective for learning deep, conceptual knowledge (Graesser, Lippert, and Hampton 2017). With that said, open-ended tutoring dialogs are not easily paired with procedural problem solving, due to the conversation and the task potentially having a different state, so dialog-based ITS for math often focus on conceptual knowledge or break the problem into multiple stages (Nye et al. 2018). Research on the impact of virtual agents on engagement and motivation is also relatively strong, but requires further investigation for teachers as learners. Overall, disengagement is often a problem for online learning (Feild et al. 2018) and virtual agents have shown the ability to increase motivation in computer-based learning (Sträfling et al. 2010). However, research has not studied how teachers specifically react to pedagogical agents and research with other adult professionals limited. For example, research on military training has shown some evidence for strong engagement during agent-based scenarios, but without a comparable traditional online learning control condition (Lane et al. 2013). Research on agents to support long-term engagement in medical interventions has shown benefits and give some evidence that agents engage adults, but the agents were not tutors (Bickmore, Schulman, and Yin 2010). One of the most popular voluntary learning apps (DuoLingo) employs an animated pedagogical agent, but the agent is only one of many engagement mechanisms in the app. Perhaps most important, learners’ reactions to agents are dependent on the type of agent (Baylor 2011) and their reactions to agents can impact learning outcomes (Schroeder et al. 2018). As such, the relevant question may not be if teachers can be motivated by a pedagogical agent but what kind of agent appearance and role is appropriate to engage mathematics teachers. System Design: ATProportion Based on this background work, the number of design criteria were identified as being essential for the success of the ATProportion professional development for proportional reasoning: 1. Virtual Facilitator: A virtual agent, whose interactions and appearance must be engaging and accepted by teachers. 2. Multi-Stage Examples: Walking a learner through multiple stages of a problem or concept, such as different steps, comparing examples, or multiple representations. 3. Conceptual Tutoring: Guiding a learner to discuss and explain their understanding of proportional reasoning. 4. Virtual Manipulatives: Multiple categories for objects of joint attention, such as images, videos, note-taking, whiteboards, and interactive objects. These elements represent the core capabilities of the system, which are the lessons that the teachers learn from. A number of other capabilities support this functionality, which will be described briefly. The system is structured in terms of two modules: Content Knowledge and Pedagogical Content Knowledge. Inside each module is a list of submodules, which train specific skills as shown in Figure 1. Each submodule contains a finite set of lessons, where completed lessons show a traffic light for their completion status and attemped lessons have a notepad link to review and edit any note-taking from the user. Lessons are adaptively recommended to the learner through a simple algorithm: each lessons is completed once in-order and then lessons with scores below passing are recommended up to once more if the submodule performance is below passing. While submodules may be completed out of order, the system will always recomme