Alexey Podlasov

Spatial pedagogy: mapping meanings in the use of classroom space

The positioning and movement of the teacher in the classroom are fundamental to the pedagogical process. Specific spaces in the classroom take on certain meanings because of the nature of pedagogic discourse that occurs on the site and the positioning and distance of the site relative to the students and the teaching resources. Spatial pedagogy is realised through the patterns of positioning and the directionality of movement, as well as the intersemiotic correspondences in the use of space with other semiotic resources (e.g. language, gesture and teaching materials). This paper investigates the different types of space in the classroom and their associated meanings. It also discusses an approach for annotating the teacher’s use of space, and the usefulness of visualising this annotation through digital graphical methods. The notion of ‘structured informality’ in the classroom is proposed through data analysis of two teachers conducting similar lessons, but with apparently different pedagogical styles.

Interactive software for multimodal analysis

This article presents the design and functionalities of interactive software for multimodal analysis currently being developed in the Multimodal Analysis Lab, Interactive Digital Media Institute (IDMI) at the National University of Singapore. The software is being used for the annotation, analysis, search and retrieval of semantic patterns in unified but complex semiotic acts – for example, the interaction of gesture, gaze, intonation, camera angle, and music in a film. In addition to providing a digital platform for multimodal analysis, the software provides the site for further development of multimodal theory as the analytical techniques and tools produce insights into the nature of the multimodal phenomena. The approach is located within the digital humanities paradigm that promotes the use of computer techniques and technologies for humanities, arts, and social science research.

Analysing multimodality in an interactive digital environment: software as a meta-semiotic tool

The present paper discusses issues arising from the use of an interactive digital software tool to analyse multimodal communication. The focus is on the ways in which such technical resources and associated techniques enable the analyst of social semiosis to apply different types of analysis, and provide the site for critical reflection upon the results of such analyses. The aim is to present an argument for the use of an interactive digital software application as a meta-semiotic tool. Three major challenges for scholars engaged in multimodal social semiotics are addressed in pursuing this aim, specifically with reference to the development of digital interactive analytical resources. Firstly, scholars are faced with the task of not only accounting for an increasing range of semiotic resources, but also for the way different semiotic phenomena interact to produce meaning. Secondly, the multimodal semiotician has a broad range of analytical approaches potentially relevant to any task to draw upon. Thirdly, the nature of contemporary media forms offer challenges themselves in terms of access, analysis and presentation of analysis.

Multimodal digital semiotics: the interaction of language with other resources

Abstract A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allens interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Hallidays systemic theory.

Interactive state-transition diagrams for visualization of multimodal annotation

Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are well developed in the literature, and there is progress in the development of annotation software, the interpretation of annotated data still rarely uses algorithmic and data visualization approaches, limiting practical research in multimodal analysis. In the current work we propose how time stamped, tier-based multimodal annotations can be seen as finite state automatons and visualized as state-transition diagrams. We propose an algorithm converting time stamped annotation into a state-transition diagram and discuss how the resulting state-transition diagram can be visualized using freely available graph visualization tools. Then, we discuss common drawbacks of the existing tools, and propose an interactive visualization software tool allowing the analyst to synchronize the annotation state-transition diagram with the original media file. Finally, a case study on how the proposed visualization tools can assist media researchers in detecting patterns in multimodal analyses of news videos is presented.

Developing Novel Multimodal and Linguistic Annotation Software

In this paper we present a collaborative work between computer and social scientists resulting in the development of annotation software for conducting research and analysis in social semiotics in both multimodal and linguistic aspects. The paper describes the proposed software and discusses how this tool can contribute for development of social semiotic theory and practice.