Kay L. O’Halloran

A Digital Mixed Methods Research Design: Integrating Multimodal Analysis With Data Mining and Information Visualization for Big Data Analytics:

This article demonstrates how a digital environment offers new opportunities for transforming qualitative data into quantitative data in order to use data mining and information visualization for mixed methods research. The digital approach to mixed methods research is illustrated by a framework which combines qualitative methods of multimodal discourse analysis with quantitative methods of data mining and information visualization in a multilevel, contextual model that will result in an integrated, theoretically well-founded, and empirically evaluated technology for analyzing large data sets of multimodal texts. The framework is applicable to situations in which critical information needs to be extracted from geotagged public data: for example, in crisis informatics, where public reports of extreme events provide valuable data sources for disaster management.

Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics

ABSTRACT This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large datasets. The approach is illustrated by a study which aims to analyse how violent extremist groups use language and images to legitimise their views, incite violence, and influence recruits in online propaganda materials, and how the images from these materials are re-used in different media platforms in ways that support and resist violent extremism. The approach developed in this article contributes to what promises to be one of the key areas of research in the coming decades: namely the interdisciplinary study of big (digital) datasets of human discourse, and the implications of this for terrorism analysis and research.

Violent extremism and iconisation: commanding good and forbidding evil?

ABSTRACT This study employs a multimodal social semiotic approach to the analysis of text and image relations in material produced by the violent extremist organisation known as Islamic State in Iraq and al-Sham (ISIS). The study focuses on iconisation, where meanings are condensed and interpersonally charged through ‘bonding icons’ which embody the organisations world view and values. A sample of issues in the online magazine Dabiq produced by ISIS are analysed using Systemic Functional Multimodal Discourse Analysis (SF-MDA). The ISIS world view is shown to be internally cohesive, based on a narrow, fundamentalist interpretation of selected Islamic scripture and violently opposed to any other world view. To represent, synthesise and justify its values and world view, ISIS uses bonding icons constructed from combinations of artefacts, supported by references to selected Islamic scripture. The study is a prelude to more detailed investigation of bonding icons in materials produced by ISIS, the attraction of SIS to potential mujahideen and ‘copy-cat’ jihadist groups and the recontextualisation of materials from Dabiq in different media platforms. Such studies would provide deeper insights into the workings of organisations such as ISIS, and facilitate the further development of multimodal social semiotic approaches to image and text relations.

Performance Evaluation of Students using Multimodal Learning Systems

Multimodal learning, as an effective method for helping students to understand complex concepts, has attracted much research interest recently. Using more than one media in the learning process typically makes the study material easier to grasp. In the current study, students annotate linguistic and visual elements in multimodal texts by using geometric shapes and assigning attributes. However, how to effectively evaluate student performance is a challenge. This work proposes to make use of a vector space model to process student-generated multimodal data, with a view to evaluating student performance based on the annotation data. The vector model consists of fuzzy membership functions to model the performance in the various annotation criteria. These vectors are then used as the input to a multi-criteria ranking framework to rank the students.

Multimodal Digital Humanities

Interactive digital technologies with facilities for developing conceptual frameworks and storing multidimensional data at different levels of description have the potential to advance research in semiotics (i.e., the study of sign systems and processes). Such an approach, called multimodal digital semiotics (O’Halloran et al., Text and Talk: Special Edition for Michael Halliday 33(4–5):665–690, 2013), involves a “digital semiosphere” (http://semioticon.com/semiotix/2011/02/multimodal-digital-semiotics/, http://semioticon.com/semiotix/2010/03/multimodal-semiosis-multimodal-semiotics-digital-technologies-and-techniques-for-studying-multimodal-communication/) for building theory and analytical approaches for semiotics research, borrowing from Lotman’s (Sign System Studies 33(1):208, 2005) notion of the semiosphere as “the semiotic space, outside of which semiosis cannot exist.” A digital semiosphere permits the interaction of sign systems in sociocultural processes to be theorized, analyzed, visualized, interpreted and compared, as demonstrated in this chapter. In this case, the underlying basis of the semiotics research is multimodal analytics, “the study of semantic patterns arising from the integration of language, images, and audio resources in multimodal texts” (O’Halloran et al., The Routledge handbook of multimodal analysis, Routledge, London, p. 386, 2014). Multimodal digital semiotics, the digital semiosphere and multimodal analytics move digital humanities (e.g., Berry, Understanding digital humanities, Palgrave, Hampshire, 2012) into the realm of multimodal digital humanities; in this case. “leveraging the potential of the visual and aural media that are part of contemporary life” (Svensson, Digital Humanities Quarterly, 4(1), 2010; see also McPherson, Cinema Journal 48(2):119–123, 2009) for the development of theory, tools and techniques for semiotics research (O’Halloran et al., Visual communication, Gruyter, Berlin, pp. 565–588, 2014; Podlasov and O’Halloran, Critical multimodal studies of popular culture, Routledge, New York, pp. 71–90, 2014).

Online leadership discourse in higher education: A digital multimodal discourse perspective

As leadership discourses in higher education are increasingly being mediated online, texts previously reserved for staff are now being made available in the public domain. As such, these texts become accessible for study, critique and evaluation. Additionally, discourses previously confined to the written domain are now increasingly multimodal. Thus, an approach is required that is capable of relating detailed, complex multimodal discourse analyses to broader sociocultural perspectives to account for the complex meaning-making practices that operate in online leadership discourses. For this purpose, a digital multimodal discourse approach is proposed and illustrated via a small-scale case study of the online leadership discourse of an Australian university. The analysis of two short video texts demonstrates how a digital multimodal discourse perspective facilitates the identification of key multimodal systems used for meaning-making in online communication, how meaning arises through combinations of semiotic choices (not individual choices), and how the results of multimodal discourse analysis using digital technology can reveal larger sociocultural patterns – in this case, divergent leadership styles and approaches as reflected in online discourse, at a time of immense change within the higher education sector.

Under the shade of AK47s: a multimodal approach to violent extremist recruitment strategies for foreign fighters

ABSTRACT Two notable features of the current conflict in Syria and Iraq are the number of foreign fighters from western countries fighting for Sunni militant organisations, and the use of the Internet and social media by some extremist groups to disseminate propaganda material. This article explores how the group which refers to itself as Islamic State and an affiliated British group, Rayat al Tawheed, deploy combinations of images and text which serve as bonding icons to rally supporters. The data consists of the English language edition of ISIS’s online magazine Dabiq and online materials produced by Rayat al Tawheed. The results suggest that ISIS and Rayat al Tawheed adopt similar but different iconisation strategies. While ISIS adopts a global strategy to present a unified world view utilising a range of ISIS values in its iconisation, Rayat al Tawheed foregrounds jihad using strategies specifically targeting young, English-speaking men of Islamic/Arab backgrounds.

The linguistic and interactional factors impacting recognition and dispatch in emergency calls for out-of-hospital cardiac arrest: a mixed-method linguistic analysis study protocol

Introduction Emergency telephone calls placed by bystanders are crucial to the recognition of out-of-hospital cardiac arrest (OHCA), fast ambulance dispatch and initiation of early basic life support. Clear and efficient communication between caller and call-taker is essential to this time-critical emergency, yet few studies have investigated the impact that linguistic factors may have on the nature of the interaction and the resulting trajectory of the call. This research aims to provide a better understanding of communication factors impacting on the accuracy and timeliness of ambulance dispatch. Methods and analysis A dataset of OHCA calls and their corresponding metadata will be analysed from an interdisciplinary perspective, combining linguistic analysis and health services research. The calls will be transcribed and coded for linguistic and interactional variables and then used to answer a series of research questions about the recognition of OHCA and the delivery of basic life-support instructions to bystanders. Linguistic analysis of calls will provide a deeper understanding of the interactional dynamics between caller and call-taker which may affect recognition and dispatch for OHCA. Findings from this research will translate into recommendations for modifications of the protocols for ambulance dispatch and provide directions for further research. Ethics and dissemination The study has been approved by the Curtin University Human Research Ethics Committee (HR128/2013) and the St John Ambulance Western Australia Research Advisory Group. Findings will be published in peer-reviewed journals and communicated to key audiences, including ambulance dispatch professionals.

A multimodal approach to visual thinking: the scientific sketchnote:

There is a growing interest in the use of visual thinking techniques for promoting conceptual thinking in problem solving tasks as well as for reducing the complexity of ideas expressed in scientific and technical formats. The products of visual thinking, such as sketchnotes, graphics and diagrams, consist of ‘multimodal complexes’ that combine language, images, mathematical symbolism and various other semiotic resources. This article adopts a social semiotic perspective, more specifically a Systemic Functional Multimodal Discourse Analysis approach, to study the underlying semiotic mechanisms through which visual thinking makes complex scientific content accessible. To illustrate the approach, the authors analyse the roles of language, images, and mathematical graphs and symbolism in four sketchnotes based on scientific literature in physics. The analysis reveals that through the process of resemiotization, where meanings are transformed from one semiotic system to another, the abstractness of specialized discourses such as physics and mathematics is reduced by multimodal strategies which include reformulating the content in terms of entities which participate in observable (i.e. tangible) processes and enhancing the reader/viewer’s engagement with the text. Moreover, the compositional arrangement creates clear stages in the development of the ideas and arguments that are presented. In this regard, visual thinking is a form of cultural communication through which abstract ideas are translated and explained using a multimodal outline or summary of essential parts by adapting resources (e.g. linguistic resources and mathematical graphs), using new resources (e.g. stick figures and other simple schematic drawings) and maintaining others from the original text (e.g. mathematical symbolic notation), resulting in a congruent (or concrete) depiction of abstract concepts and ideas for a non-specialist audience.

Hijacking the dispatch protocol: When callers pre-empt their reason-for-the-call in emergency calls about cardiac arrest:

This article examines emergency ambulance calls made by lay callers for patients found to be in cardiac arrest when the paramedics arrived. Using conversation analysis, we explored the trajectories of calls in which the caller, before being asked by the call-taker, said why they were calling, that is, calls in which callers pre-empted a reason-for-the-call. Caller pre-emption can be disruptive when call-takers first need to obtain an address and telephone number. Pre-emptions have further implications when call-takers reach the stage when they are required to deliver the scripted turn ‘tell me exactly what happened’. When there has been a pre-emption earlier on, callers tend to treat the scripted turn as a request for more information and may not repeat their reason-for-the-call. This can occasion delays and important information can be lost. We identified an effective alternative strategy used by some call-takers, pre-emption repeat, which callers treat as a request for confirmation.