Alejandra Martínez-Monés

A layered framework for evaluating on-line collaborative learning interactions

Evaluating on-line collaborative learning interactions is a complex task due to the variety of elements and factors that take place and intervene in the way a group of students comes together to collaborate in order to achieve a learning goal. The aim of this paper is to provide a better understanding of group interaction and determine how to best support the collaborative learning process. To that end, we propose a principled framework for the study and analysis of group interaction and group scaffolding which is built by combining different aspects and issues of collaboration, learning and evaluation. In particular, we define learning activity indicators at several levels of description which prompt to the application of a mixed interaction analysis scheme and the use of different data types and specific tools. At an initial layer, the basis of the approach is set by applying a qualitative process for evaluating the individual and group task performance as well as the group functioning and scaffolding. The interaction analysis process is completed by defining and applying two more layers: a social network analysis of the group activity and participation behaviour and a quantitative analysis of group effectiveness as regards task achievement and active interaction involvement. Our work defines a grounded and holistic conceptual model that describes on-line collaborative learning interactions sufficiently and applies it in a real, web-based, complex and long-term collaborative learning situation. An in-depth empirical evaluation of the conceptual model is fully discussed, which demonstrates the usefulness and value of the approach.

Multiple case studies to enhance project-based learning in a computer architecture course

The IEEE/Association for Computing Machinery (ACM) Computing Curricula and the Accreditation Board of Engineering and Technology (ABET) Evaluation Criteria 2000 emphasize the use of recurrent concepts and system design/evaluation through projects and case studies in the curriculum of Computer and Electrical Engineering. In addition, efficient teamwork, autonomy, and initiative are commonly required qualifications for a professional in this field. Project-based learning approaches that require the students to handle realistic case studies are adequate to pursue these objectives. However, these pedagogical approaches tend to be rejected because they promote deep learning but focus on a restricted set of concepts, whereas many engineering curricula require a broad range of concepts to be covered in each course. The introduction of multiple case studies carried out simultaneously in the same course by different teams of students can broaden the set of concepts studied, but collaboration at different levels must be strongly enforced to achieve effective learning. This paper describes a multiple-case-study project design that has been applied to a computer architecture course for four years. After systematically evaluating the experience, the authors conclude that students achieve a deep learning of the concepts required in their own case study, while they are able to generalize their knowledge to case studies of different characteristics from those considered during the course. Furthermore, a number of collaborative skills and attitudes are developed as a consequence of the proposed environment based on multiple levels of collaboration.

An Integrated Approach for Analysing and Assessing the Performance of Virtual Learning Groups

Collaborative distance learning involves a variety of elements and factors that have to be considered and measured in order to analyse and assess group and individual performance more effectively and objectively. This paper presents an approach that integrates qualitative, social network analysis (SNA) and quantitative techniques for evaluating online collaborative learning interactions. Integration of various different data sources, tools and techniques provides a more complete and robust framework for group modelling and guarantees a more efficient evaluation of group effectiveness and individual competence. Our research relies on the analysis of a real, long-term, complex collaborative experience, which is initially evaluated in terms of principled criteria and a basic qualitative process. At the end of the experience, the coded student interactions are further analysed through the SNA technique to assess participatory aspects, identify the most effective groups and the most prominent actors. Finally, the approach is contrasted and completed through a statistical technique which sheds more light on the results obtained that far. The proposal draws a well-founded line toward the development of a principled framework for the monitoring and analysis of group interaction and group scaffolding which can be considered a major issue towards the actual application of the CSCL proposals to real classrooms.

Supporting Members of a Learning Community Using Interaction Analysis Tools: The Example of the Kaleidoscope NoE Scientific Network

Communities of practice (CoP) are nowadays widely used for implementing collaborative learning approaches, under the scope of contemporary learning theories. Several communication and/or collaboration web modules are used in order to support the organizational structure and facilitate the operational purpose of such communities. Following the same approach, scientific networks (SN) are widely used as a common method of expanding research and sharing academic resources. The need for appropriate tools in order to support the members of a CoP or a SN and further facilitate their collaborative tasks, or even resulted learning is often highlighted by the literature. In this paper we propose the use of Interaction Analysis indicators for that matter. Our experience from the implementation of such indicators, supporting the kaleidoscope network of excellence is presented.

Supporting Teacher Orchestration in Ubiquitous Learning Environments: A Study in Primary Education

During the last decades, educational contexts have transformed into complex technological and social ecologies, with mobile devices expanding the scope of education beyond the traditional classroom, creating so-called Ubiquitous Learning Environments (ULEs). However, these new technological opportunities entail an additional burden for teachers, who need to manage and coordinate the resources involved in such complex educational scenarios in a process known as “orchestration”. This paper presents the evaluation of the orchestration support provided by GLUEPS-AR, a system aimed to help teachers in the coordination of across-spaces learning situations carried out in ULEs. The evaluation, following an interpretive research perspective, relied on a study where a pre-service teacher designed and enacted an authentic across-spaces learning situation in a primary school. The situation, which illustrates the orchestration challenges of ULEs, was aimed at fostering orienteering skills. It spanned five sessions taking place in the classroom, in the schools playground and at a nearby park, using multiple technologies and devices. The evaluation showed that GLUEPS-AR helped the teacher in the multiple aspects of orchestration, including implementation of his pedagogical ideas, adaptation in runtime, and sharing of orchestration load with students. Teacher awareness during outdoor activities was the main aspect to improve upon.

An Interaction-Aware Design Process for the Integration of Interaction Analysis into Mainstream CSCL Practices

The potential capabilities of computers to support analysis of interaction data have attracted the attention of the CSCL research community. This has led to the proposal of a number of interaction analysis tools, which process interaction data to meet different purposes. These may range from supporting researchers in ethnographic studies to providing advice to the students. However, after several years working with classroom-based CSCL experiences, we have found that both researchers and practitioners meet many difficulties to apply these potential benefits to their CSCL settings. Thus, the first goal of this chapter is to provide a systematic analysis of the problems that can be found when trying to apply interaction analysis tools to CSCL settings, which are then classified at into three levels, namely: application, architecture and design levels. Then, we outline the path for possible solutions to face these problems. According to this, the issues identified at the design level call for an IA-aware design process where we distinguish between co-design approaches that directly integrates the diverse needs of learning and analysis, and multi-perspective approaches that treat them independently at an initial stage. On the other hand, the problems at the application and architecture levels must be faced by technology-driven solutions, such as the use of decoupled architectures, either based on inter-process communication or on interchange of log file information. Several open issues have also been detected that need adequate solutions, as e.g., the semantic integration of log-files when multiple self-contained learning tools are used for an integrated analysis.

Data Flow between Tools: Towards a Composition-Based Solution for Learning Design

Data flow between tools cannot be specified using the current IMS Learning Design specification (LD). Nevertheless by specifying this data flow between tools, several degrees of activity automation may augment the system intervention opportunities for data flow management. Service automation, data flow automation and data flow validation may enhance the continuity of the learning design realization, reduce the students cognitive load and obtain system-support for error prone situations. In this paper a novel approach based on the composition of LD and a standard workflow technology is proposed. Unlike other current approaches, our approach maintains interoperability with both LD and workflow standards. Then an architectural solution based on the composition approach is presented.

Monitoring pattern-based CSCL scripts: a case study

Two strategies have been proposed in CSCL to foster effective collaboration: structuring the learning scenario by means of collaboration scripts and monitoring interactions among participants in order to detect and regulate potential deviations from the initial plan. In order to help teachers in this endeavor, we propose to combine these two approaches by means of a process where design takes into account the especial requirements posed by monitoring, and monitoring is informed by the characteristics of the scripts that must be met to achieve the learning goals. These desired features are obtained from the constraints defined by the collaborative-learning flow patterns on which the scripts are based. The result is an automated and higher-level view about the evolution of the learning process, integrating the data gathered from the different tools. This paper also presents a case study based on an authentic experience in higher education where these ideas were tried out.

Towards embedding assessment in CSCL scripts through selection and assembly of learning and assessment patterns

Assessment is one of the key elements of learning scenarios, both in individual and collaborative learning. Thus, embedding assessment in Computer Supported Collaborative Learning (CSCL) scripts can potentially enhance the enactment of collaborative learning sessions, by explicitly indicating how assessment is to be carried out. However, assessment design has also been recognized as a demanding task for non-expert instructional designers or practitioners. To tackle this problem, the use of learning and assessment patterns has already been reported as a way of supporting script design for non-experts. In this context, the combination of patterns of different nature (learning-oriented and assessment-oriented) can be regarded as an essential task of CSCL script design. This paper discusses how to support this task, focusing on the joint application of patterns, and describes an evaluation of the proposed approach.

Towards a flexible model for computer-based analysis and visualization of collaborative learning activities

The definition of appropriate interaction analysis methods is a major research topic in Computer Supported Collaborative Learning. Analysis methods can be totally or partially supported by computer-based tools that provide for better and more efficient analysis processes. The current research in this field shows that most interaction analysis tools have been based on unstable prototypes, and are highly dependant on the learning environments and research goals for which they were defined. As a consequence, it is not possible to use them in authentic CSCL settings with real users. The goal of this European Research Team therefore is to utilize the synergies of experience in manual interaction analysis with computer-based analytical methods. In this article we present an approach that embeds standardized computer-supported techniques into a semi-formal analysis process model which can be utilized and adapted in a flexible way according to the cases and environments to be analysed.