Sabine Tan

Multimodal analysis within an interactive software environment: critical discourse perspectives

Critical discourse analysts are increasingly required to account for multimodal phenomena constructed through language and other resources (e.g. images, sound and music) and to relate high-level critical insights on the social motivations of these texts to their realizations in low-level expressive phenomena, and vice versa. In this paper, we use interactive software resources for critical multimodal discourse analysis (OHalloran, K.L. (2011b). Multimodal analysis and digital technology. In A. Baldry & E. Montagna (Eds.), Interdisciplinary approaches to multimodality: Theory and practice. Readings in intersemiosis and multimedia (pp. 21–34). Campobasso: Palladino; OHalloran, K. L., Tan, S., Smith, B. A., & Podlasov, A. (2010). Challenges in designing digital interfaces for the study of multimodal phenomena. Information Design Journal, 18(1), 2–12; Smith, B. A., Tan, S., Podlasov, A., & OHalloran, K.L. (in press). Analyzing multimodality in an interactive digital environment: Software as metasemiotic tool. Social Semiotics) to help in achieving these aims. The field of critical discourse studies, itself being interdisciplinary, along with its holistic approach, is well placed to take advantage of interactive software, as these resources encourage the application, exploration and correlation of various analytical perspectives at different levels of description. Drawing on the analysis of a short video advertisement, we demonstrate how the interactive (multimodal) digital environment is one in which the discourse analyst can effectively draw upon different traditions of analysis, including ‘mainstream’ and social semiotic traditions, as well as other traditions such as media studies, to interpret dynamic audiovisual media texts in a critically self-reflexive manner.

Analysing multimodality in an interactive digital environment: software as a meta-semiotic tool

The present paper discusses issues arising from the use of an interactive digital software tool to analyse multimodal communication. The focus is on the ways in which such technical resources and associated techniques enable the analyst of social semiosis to apply different types of analysis, and provide the site for critical reflection upon the results of such analyses. The aim is to present an argument for the use of an interactive digital software application as a meta-semiotic tool. Three major challenges for scholars engaged in multimodal social semiotics are addressed in pursuing this aim, specifically with reference to the development of digital interactive analytical resources. Firstly, scholars are faced with the task of not only accounting for an increasing range of semiotic resources, but also for the way different semiotic phenomena interact to produce meaning. Secondly, the multimodal semiotician has a broad range of analytical approaches potentially relevant to any task to draw upon. Thirdly, the nature of contemporary media forms offer challenges themselves in terms of access, analysis and presentation of analysis.

Multimodal digital semiotics: the interaction of language with other resources

Abstract A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allens interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Hallidays systemic theory.

Interactive state-transition diagrams for visualization of multimodal annotation

Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are well developed in the literature, and there is progress in the development of annotation software, the interpretation of annotated data still rarely uses algorithmic and data visualization approaches, limiting practical research in multimodal analysis. In the current work we propose how time stamped, tier-based multimodal annotations can be seen as finite state automatons and visualized as state-transition diagrams. We propose an algorithm converting time stamped annotation into a state-transition diagram and discuss how the resulting state-transition diagram can be visualized using freely available graph visualization tools. Then, we discuss common drawbacks of the existing tools, and propose an interactive visualization software tool allowing the analyst to synchronize the annotation state-transition diagram with the original media file. Finally, a case study on how the proposed visualization tools can assist media researchers in detecting patterns in multimodal analyses of news videos is presented.

Developing Novel Multimodal and Linguistic Annotation Software

In this paper we present a collaborative work between computer and social scientists resulting in the development of annotation software for conducting research and analysis in social semiotics in both multimodal and linguistic aspects. The paper describes the proposed software and discusses how this tool can contribute for development of social semiotic theory and practice.