Sumin Zhao

Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint

Abstract The ubiquitous software PowerPoint has significant influence on evaluations of professional and academic success, and has attracted considerable attention from both social commentators and researchers in various fields. Existing research on PowerPoint considers the software, slideshows created with it, and PowerPoint-supported presentations in isolation from each other and is therefore unable to promote better understanding of the interaction between the softwares design and its use. This article proposes a model for exploring this interaction. Specifically, it introduces a multimodal social semiotic approach to studying PowerPoint as a semiotic practice comprising three dimensions – the softwares design, the multimodal composition of slideshows, and their presentation – and two semiotic artefacts, the software and the slideshow. It discusses the challenges each dimension presents for discourse analysis and social semiotic research, focusing especially on the need to step away from the notion of text and to develop a holistic, non-logocentric, and adaptive multimodal approach to researching semiotic technologies. Using PowerPoint as a case study, this article takes a step toward developing a social semiotic multimodal theory of the relation between semiotic technologies, or technologies for making meaning, and semiotic practices.

Critical multimodal studies of popular discourse

The global impact of demanding environmental concerns is visible in almost all contexts of contemporary communication and across geographical borders. An increasing range of multimodal texts surface continuously in various media in order to facilitate public understanding of irreversible environmental changes, to educate future generations in ecoliteracy, to promote green or disclose greenwashed corporate images and practices, or to entertain and facilitate appropriate actions as well as responses. Simultaneously, research in environmental communication tries to keep up with this rapid pace by examining environmentfocused multimodal texts from the context of journalism (Doyle, 2011; Lester & Cottle, 2009), education (Maier, 2010; Reid, 2007), advertising (Corbett, 2006; Cox, 2010; Hansen & Machin, 2008; Maier 2011; Moschini, 2007), and popular culture (Brereton, 2004; Meister & Japp, 2002; Starosielski, 2011), to mention only a few relevant areas. Although environment-focused music videos have also proliferated in the last decade, and despite their recognized impact upon younger generations, giving expression as they do to the rhythms and visual associations relevant to youth cultures, music videos that deal with the environmental theme have relatively rarely been the subject of research endeavours. The present chapter intends to draw attention to how the analysis of relevant multimodal texts, such as the music video Earth Song, can contribute to a better understanding of the ways by which communication about environmental issues takes place in the context of popular culture. Our analysis will primarily be focused on how the video takes a critical view of human interaction with the environment by questioning the wisdom of traditional national boundaries and notions of time as linear and irreversible. Michael Jackson’s Earth Song is a call to save the planet from the destructive impact that has been wrought upon the earth by humanity and technology. It was recorded in 1995, but never released as a single in the United States, due to events related to perceptions of Michael Jackson’s private life. However, Earth Song won a Grammy nomination in 1997( Jurin, Roush, & Danter, 2010, p. 132), as well as recognition in the form of the Genesis Award in 1996. According to Grant (1998), it was Jackson’s intention to create a lyrical and also melodically simple song, so the whole world, including non-English-speaking audiences, could sing along. Earth Song has a specific synchronization of semiotic modes, orchestrated along four narrative strands and filmed in four different geographic locations across the globe, but presumably occurring at the same time. Each of these strands presents images of deforestation, animal cruelty, pollution, and war, with their disastrous consequences for humanity and Earth. These visual stories, based on shots taken from documentary archives and documentary-like footage filmed in Warwick, New York, the Amazon Rain Forest, Croatia, and Tanzania, are brought together and synchronized with an equally alarming musical accompaniment, insistent lyrics, an iconic presenter, and carefully edited shots of similar actions and gestures performed by the participants. The regular solo appearances of Michael Jackson as the voice of the world are staged against a backdrop of burning forests around New York. The overall spectacular effect is largely achieved through the interplay between its regular musical structure and the chorus-like chant “What about us?” which is coordinated with footage from the four disparate locations of devastation. Earth Song has earned its recognition as a “green anthem” because the broader environmental discourse that underlies it can be found not in the four individual “activity schemas” (Machin, 2010, p. 94), but at the level of the whole video, which reveals Michael Jackson’s critical approach to environmental issues. This book chapter is available at Research Online: http://ro.uow.edu.au/sspapers/418 This chapter illustrates a few of the ways in which this video’s discourse constructs space and time through the interplay of several semiotic modes. Our focus on space and time is motivated by the fact that we consider these to be fundamental coordinates of many environmental discourses. As will be shown below, in this particular video, the multimodal representation of space and time carries the critical green message in multiple ways.

Understanding Semiotic Technology in University Classrooms: A Social Semiotic Approach to PowerPoint-Assisted Cultural Studies Lectures.

In this paper, we propose a social semiotic approach to studying PowerPoint in university classrooms. Our approach is centred on two premises: (1) PowerPoint is a semiotic technology that can be integrated into the pedagogical discourse of classrooms, and (2) PowerPoint technology encompasses three interrelated dimensions of social semiotic practices: the design of the software, the composition of the slides and the slideshow-supported presentations, i.e. lectures. Using this approach, we explore how PowerPoint has been used in seven cultural studies lectures in an Australian university. Our analysis demonstrates how multimodal resources in PowerPoint have been used for pedagogic recontextualisation. More specifically, it shows how different semiotic resources have been deployed and combined to recontextualise two key types of knowledge – signifying practice and subjectivity – in the classroom discursive space, and how different strengths of pedagogic framing are achieved multimodally.

Beyond the self: Intersubjectivity and the social semiotic interpretation of the selfie:

As an iconic image of our time, the selfie has attracted much attention in popular media and scholarly writing. The focus so far has been on the representation of the self or subjectivity. We propose a complementary perspective that foregrounds the intersubjective function of the selfie. We argue that the presence of selfhood is often an assumption. What distinguishes the selfie from other photographic genres is its ability to enact intersubjectivity – the possibility for difference of perspectives to be created and this difference to be shared between the image creator and the viewer. Based on a social semiotic analysis of selfies on Instagram, we identify four subtypes of selfie, each deploying a combination of visual resources to represent a distinct form of intersubjectivity. Our analysis suggests that the potential for empowerment is inherent in the visual structure of the selfie, and that, as a genre, it is open for recontextualisation across contexts and social media platforms.

2.2 Interpreting Websites in Educational Contexts: A Social-Semiotic, Multimodal Approach

This chapter reports on three research projects concerned with websites. All three have adopted a social-semiotic, multimodal approach (cf. Halliday, M. A. K. (1978). Language as Social Semiotic. London: Arnold; Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Oxon: Routledge; Kress G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold, Kress, G., & Van Leeuwen, T. (2006 [1996]) Reading images: The grammar of visual design (2nd edn). London: Routledge) and developed new tools for understanding the complexity of website design and its implications for educational contexts. They have focused, however, on different challenges and types of hypermedia: (i) user orientation within websites for children, (ii) knowledge construction in online educational interactives for children, and (iii) news design in online newspapers and the literacy demands online newspapers present for TESOL and applied linguistics students. This paper opens by outlining the common starting point of all three projects in relation to the data – treating websites as meaningful texts. It then proceeds to consider differences in the questions and types of websites explored in each study, and how these differences have influenced the way social-semiotic multimodal tools for analysing the data have been developed in each study and complemented with other tools for collecting and analysing data (e.g. interviews with website users, designers and media practitioners). The paper concludes with a brief discussion of challenges that websites continue to present for educational research.

Advancing Multimodal and Critical Discourse Studies: Interdisciplinary Research Inspired by Theo Van Leeuwen’s Social Semiotics

As a founder and leading figure in multimodality and social semiotics, Theo van Leuween has made significant contributions to a variety of research fields, including discourse analysis, sociolingui ...

The interplay of (semiotic) technologies and genre: the case of the selfie

ABSTRACT The selfie is one of the most widely publicized, criticized, and debated visual phenomena of our time. However, formulating a definition of the selfie is not straightforward, as visual clues – be they representational or compositional – alone are not sufficient for identification. Recognizing an image as a selfie, rather than a portrait, often requires viewers to interpret the image in relation to the technological and sociocutural context in which the photo was taken and shared. In this paper, we consider the technological conditions that have shaped the evolution of the selfie as a visual genre. Central to our discussion is the premise that the selfie is not simply a genre for self-representation, but means of generating various perspectives: that of the selfie maker, the represented visual participant, and the viewer identification. This unique perspective-generating affordance of the selfie is both facilitated and constrained by the various technologies involved in selfie practices. On the one hand, the technological and physical constraints of the smart phone camera give rise to a specific form of “distorted” look which makes certain types of selfie possible. On the other hand, social media platforms facilitate the sharing of selfies, which results in increasingly stylized and creative ways in which perspectives of the self can be represented, negotiated, and, in the case of selfies manipulated via apps, augmented.