Peter Wignell

Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics

ABSTRACT This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large datasets. The approach is illustrated by a study which aims to analyse how violent extremist groups use language and images to legitimise their views, incite violence, and influence recruits in online propaganda materials, and how the images from these materials are re-used in different media platforms in ways that support and resist violent extremism. The approach developed in this article contributes to what promises to be one of the key areas of research in the coming decades: namely the interdisciplinary study of big (digital) datasets of human discourse, and the implications of this for terrorism analysis and research.

Intertextual voices and engagement in TV advertisements

By analysing multimodal TV advertisements, this study aims to show how intertextual voices are exploited in advertising discourse to enhance persuasive power. Taking as their point of departure the assumption that all discourses are intertextual recontextualizations of social practice that draw on external voices from both specific discourses and discursive conventions, the authors identify two types of intertextual voice in TV advertisements: character and discursive voice. This article illustrates the multimodal construction of voices and demonstrates that the choice of voices is closely related to the ‘domain’ of the product. It is argued that the intertexual voices contribute to the advertising discourse through multimodal engagement strategies. Character voice endorses the advertised product through such resources as lexico-grammar, intonation, facial expression and staged narrative, while discursive voice endorses the advertised product through contextualization and intertextual discourse structure. It is hoped that the study will shed light on the understanding of the heteroglossic nature of advertisements, the interaction between intertextual voices and the advertised message, and multimodal construction of voices and engagement.

Violent extremism and iconisation: commanding good and forbidding evil?

ABSTRACT This study employs a multimodal social semiotic approach to the analysis of text and image relations in material produced by the violent extremist organisation known as Islamic State in Iraq and al-Sham (ISIS). The study focuses on iconisation, where meanings are condensed and interpersonally charged through ‘bonding icons’ which embody the organisations world view and values. A sample of issues in the online magazine Dabiq produced by ISIS are analysed using Systemic Functional Multimodal Discourse Analysis (SF-MDA). The ISIS world view is shown to be internally cohesive, based on a narrow, fundamentalist interpretation of selected Islamic scripture and violently opposed to any other world view. To represent, synthesise and justify its values and world view, ISIS uses bonding icons constructed from combinations of artefacts, supported by references to selected Islamic scripture. The study is a prelude to more detailed investigation of bonding icons in materials produced by ISIS, the attraction of SIS to potential mujahideen and ‘copy-cat’ jihadist groups and the recontextualisation of materials from Dabiq in different media platforms. Such studies would provide deeper insights into the workings of organisations such as ISIS, and facilitate the further development of multimodal social semiotic approaches to image and text relations.

Under the shade of AK47s: a multimodal approach to violent extremist recruitment strategies for foreign fighters

ABSTRACT Two notable features of the current conflict in Syria and Iraq are the number of foreign fighters from western countries fighting for Sunni militant organisations, and the use of the Internet and social media by some extremist groups to disseminate propaganda material. This article explores how the group which refers to itself as Islamic State and an affiliated British group, Rayat al Tawheed, deploy combinations of images and text which serve as bonding icons to rally supporters. The data consists of the English language edition of ISIS’s online magazine Dabiq and online materials produced by Rayat al Tawheed. The results suggest that ISIS and Rayat al Tawheed adopt similar but different iconisation strategies. While ISIS adopts a global strategy to present a unified world view utilising a range of ISIS values in its iconisation, Rayat al Tawheed foregrounds jihad using strategies specifically targeting young, English-speaking men of Islamic/Arab backgrounds.

Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL

Multimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of CALL in 3-D virtual worlds, illustrated by the example of Second Life . The multimodal analyses of a conventional face-to-face lesson and three language learning activities in Second Life highlight some of the affordances and challenges presented by 3-D virtual environments. The results suggest that while multimodal resources integrate naturally to facilitate language teaching and learning in an orderly, structured and goal-orientated manner in classroom lessons, the often uncoordinated use (or absence) of avatars’ gaze, facial expression, body posture, gesture, as well as the unclear proxemics and use of space pose problems for effective communication in a 3-D virtual world. In addition, a “technology-oriented” register, alongside traditional instructional and regulative genres and registers, is introduced to help students cope with the demands of learning a language in a 3-D virtual environment. The study raises the issue of the relative effectiveness of 3-D virtual worlds for language teaching and learning. In doing so, a digital approach to multimodal research is proposed in order to address the complexity of multimodal learning environments and the various challenges for CALL.

A multimodal approach to visual thinking: the scientific sketchnote:

There is a growing interest in the use of visual thinking techniques for promoting conceptual thinking in problem solving tasks as well as for reducing the complexity of ideas expressed in scientific and technical formats. The products of visual thinking, such as sketchnotes, graphics and diagrams, consist of ‘multimodal complexes’ that combine language, images, mathematical symbolism and various other semiotic resources. This article adopts a social semiotic perspective, more specifically a Systemic Functional Multimodal Discourse Analysis approach, to study the underlying semiotic mechanisms through which visual thinking makes complex scientific content accessible. To illustrate the approach, the authors analyse the roles of language, images, and mathematical graphs and symbolism in four sketchnotes based on scientific literature in physics. The analysis reveals that through the process of resemiotization, where meanings are transformed from one semiotic system to another, the abstractness of specialized discourses such as physics and mathematics is reduced by multimodal strategies which include reformulating the content in terms of entities which participate in observable (i.e. tangible) processes and enhancing the reader/viewer’s engagement with the text. Moreover, the compositional arrangement creates clear stages in the development of the ideas and arguments that are presented. In this regard, visual thinking is a form of cultural communication through which abstract ideas are translated and explained using a multimodal outline or summary of essential parts by adapting resources (e.g. linguistic resources and mathematical graphs), using new resources (e.g. stick figures and other simple schematic drawings) and maintaining others from the original text (e.g. mathematical symbolic notation), resulting in a congruent (or concrete) depiction of abstract concepts and ideas for a non-specialist audience.

Image and text relations in ISIS materials and the new relations established through recontextualisation in online media

This study takes a systemic functional multimodal social semiotic approach to the analysis and discussion of image and text relations in two sets of data. First, patterns of contextualisation of images and text in the online magazines Dabiq and Rumiyah produced by the Islamic extremist organisation which refers to itself as Islamic State (referred to here as ISIS) are examined. The second data set consists of a sample of texts from Western online news and blog sites which include recontextualisations of images found in the first data set. A sample of examples of the use and re-use of images is discussed in order to identify patterns of similarity and difference when images and text are recontextualised. It is argued that the ISIS material tends to foreground the interpersonal metafunction in combination with the textual metafunction (i.e. the stance towards the content and the organisation of the message for this purpose), while the other data set tends to foreground the ideational metafunction (the participants, processes and circumstances of what is being reported). These inferences indicate that further exploration of a larger data set is worth pursuing. Such studies would provide deeper insights helping to distinguish between online material which supports terrorism and that which opposes it, as well as facilitating the further development of multimodal social semiotic approaches to image and text relations.

From book to stage to screen: semiotic transformations of Gothic horror genre conventions

ABSTRACT This paper adopts a multimodal social semiotic approach for exploring the semiotic changes involved in the transformation of a novel into stage and screen productions. It examines how semiotic resources are deployed in each medium through elements of mise-en-scène, such as speech, music, sound, lighting, props, staging, and cinematographic techniques, and the viewing perspectives that are thus established for audiences. The genre of Gothic horror is selected for this purpose, given how this form of performance has transfixed audiences for centuries and has been adapted for both the stage and the screen. In order to demonstrate how each performance medium has produced its own unique set of foregrounding devices to enthral and captivate audiences, a comparative analysis of excerpts from the novel The Woman in Black by Susan Hill, a videotaped theatrical performance, and the 1989 British television film of the same name is undertaken. The paper discusses the implications of the multimodal semiotic approach for developing a better understanding of the semiotic transformations that horror genre conventions undergo in different media and the viewership positions that are thus re-drawn for audiences. The paper concludes with a view of multimodal recontextualisation processes which form the underlying basis of human sociocultural life.