Alvin Vista

Automatic Coding Procedures for Collaborative Problem Solving

This chapter examines the procedure followed in defi ning a scoring process to enable the reporting of individual student results for teachers to use in the classroom. The procedure begins with the identifi cation of task features that match elements of the skills frameworks, and is followed by the generation of simple rules to collect data points to represent these elements. The data points are extracted from log fi les generated by students engaged in the assessment tasks and consist of the documentation of each event, chat and action from each student. The chapter includes examples of the process for defi ning and generating global and local (task specifi c) indicators, and examples of how the indicators are coded, scored and interpreted. The development of coding and scoring of data generated when students engage in collaborative problem solving tasks is described. The data generated are captured in a process stream data fi le. Patterns of these data are coded as indicators of ele- ments defi ned in the conceptual framework outlined in Hesse et al. ( 2015 ; Chap. 2 ) and the relative complexity of indicators is used in a scoring process. The scored data are then used to calibrate the tasks. The calibrations form the basis of interpre- tation and these are used in forming reports for students and teachers. Figure 6.1 summarises the entire process from task development to the reporting of student ability based on a developmental framework.

Gender Differences in Variance and Means on the Naglieri Non-Verbal Ability Test: Data from the Philippines.

BACKGROUND Research on gender differences in intelligence has focused mostly on samples from Western countries and empirical evidence on gender differences from Southeast Asia is relatively sparse. AIMS This article presents results on gender differences in variance and means on a non-verbal intelligence test using a national sample of public school students from the Philippines. SAMPLE More than 2,700 sixth graders from public schools across the country were tested with the Naglieri Non-verbal Ability Test (NNAT). METHODS Variance ratios (VRs) and log-transformed VRs were computed. Proportion ratios for each of the ability levels were also calculated and a chi-square goodness-of-fit test was performed. An analysis of variance was performed to determine the overall gender difference in mean scores as well as within each of three age subgroups. RESULTS Our data show non-existent or trivial gender difference in mean scores. However, the tails of the distributions show differences between the males and females, with greater variability among males in the upper half of the distribution and greater variability among females in the lower half of the distribution. Descriptions of the results and their implications are discussed. CONCLUSIONS Results on mean score differences support the hypothesis that there are no significant gender differences in cognitive ability. The unusual results regarding differences in variance and the male-female proportion in the tails require more complex investigations.

Visualising and examining sequential actions as behavioural paths that can be interpreted as markers of complex behaviours

Abstract Visualisation of pathways taken by students through problem solving environments provides the potential to identify patterns of exploratory behaviour that may relate to different levels of proficiency or different cognitive approaches. This is particularly the case when problems are situated in complex online environments and are programmed to generate large scale data for analysis. Using process data from an online collaborative problem solving task, we visualised behavioural paths of 607 pairs of students as directed graphs. The empirical paths were then examined using exploratory network analysis based on four main aspects of exploration (prominence, branches, clusters, and shortest paths). The primary purpose of such analysis is to detect sequences that are potentially relevant for establishing particular paths as meaningful markers of complex behaviours. This visualisation approach enabled us to capture all actual (as opposed to possible ) pathways from the data. Although there may be an optimal pathway through a complex task according to criteria such as efficiency or correct solution, the added human factors bring a dynamic to the activity that provides a richer environment for understanding students’ collaborative processes, both cognitive and social. The method adopted in this study provides a prototype for exploration of other complex skillsets.

Sequential actions as markers of behavioural and cognitive processes

This paper presents a novel exploratory method to extract information about behavioural and cognitive processes that occur during the performance of complex tasks. The methodology is framed in a structure where process data on interactions between the task performer and the task space environment are captured in a data stream for big data analytics.We begin by describing a reductionist approach that involves deconstructing complex processes into constituent parts. We extend this approach by looking at sequences of these constituent parts with the aim of detecting patterns in a persons action pathways. We propose the exploration of these pathways as indicators of underlying behavioural and cognitive processes. Current sequence-based approaches are presented and a proposed approach in dealing with sequences as empirical pathways is demonstrated using data stream from an interactive and collaborative online-administered complex task. These empirical pathways were visualised as directed graphs and an exploratory network analysis is proposed to examine specific features of the emergent paths.General challenges encountered during the development of such a method are described and specific future challenges that we foresee in transitioning from an exploratory phase to a confirmatory phase are examined. Finally, possibilities for further research as well as future applications are discussed. An exploratory method to extract information on behavioural and cognitive processes.Performance in complex tasks can be systematically measured and analysed.Behavioural indicators are visualised as a network of directed graph.Exploring dominant paths in the network can yield process information.

Formative Assessment Policy and Its Enactment in the Philippines

In this chapter formative assessment is defined as a process of providing information to teachers to focus instruction on the improvement of student learning outcomes. The Department of Education in the Philippines in 2012 started the implementation of a new curriculum within a new structure which extended the education system from a Grade 1–10 to a K–12 structure to bring it into line with the developed world. A significant part of the curriculum reform is an assessment framework that includes formative approaches to assessment. The goal of the policy on formative assessment is to help teachers recognise relevant intervention practices that will improve student learning outcomes. An observation study of the link between assessment and teaching in a sample of 61 classroom lessons identified baseline practices and ways in which the emerging policy of the Department of Education in the Philippines could be promulgated. The observation study focused on classroom organisation, teacher instructional and assessment strategies, lesson structure, resources used by teachers, and student involvement in class work, as well as both formal and informal assessment practices. The observations were documented in a series of narratives aimed at identifying variation between teachers within grade level and disciplines. It emerged that a lesson structure which lingers from the previous curricular approach may be both the major inhibiting factor regarding formative use of assessment data and the most obvious opportunity for change.

The role of PS ability and RC skill in predicting growth trajectories of mathematics achievement

Abstract There are relatively few studies in Australia and South-East Asian region that combine investigating models of math growth trajectories with predictors such as reasoning ability and reading comprehension skills. Math achievement is one of the major components of overall academic achievement and it is important to determine what factors (especially domain-general factors) predict levels of achievement over time. This study presents large-scale data (N = 5,886) from Australia to examine the trajectories of growth in math achievement and how problem solving (PS) ability and reading comprehension (RC) skills predict this growth among government school students (grades 3–8) in Victoria. Latent growth modelling showed that PS ability predicts growth in math achievement and this relationship is partially mediated by RC skill. Both PS ability and RC skill predict initial status in math achievement, but only PS ability predicts growth. The data also fit a model in which an improvement in general reasoning ability of students allows those with lower initial levels in math achievement to catch up. This can be interpreted as evidence that improving growth rates in PS ability may lift growth rates in math achievement.

An impact-based filtering approach for literature searches:

This paper aims to present an alternative and simple method to improve the filtering of search results so as to increase the efficiency of literature searches, particularly for individual researchers who have limited logistical resources. The method proposed here is scope restriction using an impact-based filter, made possible by the emergence of freely available citation databases in conjunction with new tools to access and process the data efficiently. Results from a simulated search are compared between a conventional date-ranged filtered search and an impact-based filtered search using Publish or Perish as the citation analysis software. The utility and practical implications of this approach are further discussed.