Emilia Djonov

The Semiotics of texture : from tactile to visual

The term ‘texture’ is often applied beyond the tactile, to describe visual and aural qualities. While tactile, visual and aural texture have been studied separately in various fields, the relationships between them remain largely unexplored. To address this gap, this article proposes parameters for describing tactile surface texture and visual texture, and compares their meaning-making potential. The authors argue that, as new technologies increasingly limit the role of tactile experience and expand the importance of the visual, there is a growing need to study the influence of ubiquitous technologies on our use and understanding of the semiotic potential of resources such as texture. They hypothesize about this influence by reviewing the presentation of texture as a fill option for shapes and backgrounds in Microsoft PowerPoint for Windows from 1992 to 2007.

Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint

Abstract The ubiquitous software PowerPoint has significant influence on evaluations of professional and academic success, and has attracted considerable attention from both social commentators and researchers in various fields. Existing research on PowerPoint considers the software, slideshows created with it, and PowerPoint-supported presentations in isolation from each other and is therefore unable to promote better understanding of the interaction between the softwares design and its use. This article proposes a model for exploring this interaction. Specifically, it introduces a multimodal social semiotic approach to studying PowerPoint as a semiotic practice comprising three dimensions – the softwares design, the multimodal composition of slideshows, and their presentation – and two semiotic artefacts, the software and the slideshow. It discusses the challenges each dimension presents for discourse analysis and social semiotic research, focusing especially on the need to step away from the notion of text and to develop a holistic, non-logocentric, and adaptive multimodal approach to researching semiotic technologies. Using PowerPoint as a case study, this article takes a step toward developing a social semiotic multimodal theory of the relation between semiotic technologies, or technologies for making meaning, and semiotic practices.

Critical multimodal studies of popular discourse

The global impact of demanding environmental concerns is visible in almost all contexts of contemporary communication and across geographical borders. An increasing range of multimodal texts surface continuously in various media in order to facilitate public understanding of irreversible environmental changes, to educate future generations in ecoliteracy, to promote green or disclose greenwashed corporate images and practices, or to entertain and facilitate appropriate actions as well as responses. Simultaneously, research in environmental communication tries to keep up with this rapid pace by examining environmentfocused multimodal texts from the context of journalism (Doyle, 2011; Lester & Cottle, 2009), education (Maier, 2010; Reid, 2007), advertising (Corbett, 2006; Cox, 2010; Hansen & Machin, 2008; Maier 2011; Moschini, 2007), and popular culture (Brereton, 2004; Meister & Japp, 2002; Starosielski, 2011), to mention only a few relevant areas. Although environment-focused music videos have also proliferated in the last decade, and despite their recognized impact upon younger generations, giving expression as they do to the rhythms and visual associations relevant to youth cultures, music videos that deal with the environmental theme have relatively rarely been the subject of research endeavours. The present chapter intends to draw attention to how the analysis of relevant multimodal texts, such as the music video Earth Song, can contribute to a better understanding of the ways by which communication about environmental issues takes place in the context of popular culture. Our analysis will primarily be focused on how the video takes a critical view of human interaction with the environment by questioning the wisdom of traditional national boundaries and notions of time as linear and irreversible. Michael Jackson’s Earth Song is a call to save the planet from the destructive impact that has been wrought upon the earth by humanity and technology. It was recorded in 1995, but never released as a single in the United States, due to events related to perceptions of Michael Jackson’s private life. However, Earth Song won a Grammy nomination in 1997( Jurin, Roush, & Danter, 2010, p. 132), as well as recognition in the form of the Genesis Award in 1996. According to Grant (1998), it was Jackson’s intention to create a lyrical and also melodically simple song, so the whole world, including non-English-speaking audiences, could sing along. Earth Song has a specific synchronization of semiotic modes, orchestrated along four narrative strands and filmed in four different geographic locations across the globe, but presumably occurring at the same time. Each of these strands presents images of deforestation, animal cruelty, pollution, and war, with their disastrous consequences for humanity and Earth. These visual stories, based on shots taken from documentary archives and documentary-like footage filmed in Warwick, New York, the Amazon Rain Forest, Croatia, and Tanzania, are brought together and synchronized with an equally alarming musical accompaniment, insistent lyrics, an iconic presenter, and carefully edited shots of similar actions and gestures performed by the participants. The regular solo appearances of Michael Jackson as the voice of the world are staged against a backdrop of burning forests around New York. The overall spectacular effect is largely achieved through the interplay between its regular musical structure and the chorus-like chant “What about us?” which is coordinated with footage from the four disparate locations of devastation. Earth Song has earned its recognition as a “green anthem” because the broader environmental discourse that underlies it can be found not in the four individual “activity schemas” (Machin, 2010, p. 94), but at the level of the whole video, which reveals Michael Jackson’s critical approach to environmental issues. This book chapter is available at Research Online: http://ro.uow.edu.au/sspapers/418 This chapter illustrates a few of the ways in which this video’s discourse constructs space and time through the interplay of several semiotic modes. Our focus on space and time is motivated by the fact that we consider these to be fundamental coordinates of many environmental discourses. As will be shown below, in this particular video, the multimodal representation of space and time carries the critical green message in multiple ways.

“David Byrne really does love PowerPoint”: art as research on semiotics and semiotic technology

This article presents a reading of David Byrnes Envisioning Emotional Epistemological Information, an art work created with MicroSofts presentation software PowerPoint, as an instance of creative research on semiotics and semiotic technology. It reveals commonalities and differences between Byrnes ideas about PowerPoint and related ideas from linguistics and semiotics, and is intended as a contribution to research on PowerPoint, and on semiotic technologies generally, as well as to efforts aimed at developing criteria for evaluating art as research on semiotics.

Notes towards a semiotics of kinetic typography

This paper traces the development of a new semiotic mode, kinetic typography. Kinetic typography began with the experiments of filmmakers like Len Lye and Norman McLaren. Later, film title designers like Saul Bass and Pablo Ferro drew on the shapes of letters with inventive metaphors – serifs, for instance could make letters walk, because they can stand for shoes as they are elongated horizontals on which something stands. In Saul Bass’ titles for Hitchcocks Psycho, the splitting of letters became a metaphor for the split mind of the films main character. Such inventions eventually became part of a lexicon of clichés drawn on by designers across the world. Eventually, researchers and software designers began to formalize and systematize the language of kinetic typography, and the fruit of their work is now widely available, not only to specialists, but also to anyone who uses PowerPoint or Adobe AfterEffects, even though users may not always be aware of the lexico-grammatical rules which underlie the menus they choose from. And computers, being agnostic as to the kind of “objects” their operations operate on, apply these grammars to letters as well as other graphic forms, thus consolidating the multimodality of the language of movement. The second part of the paper discusses these formalizations, drawing on the kinetic design literature. Based on M.A.K. Hallidays transitivity theory, it sketches the outlines of a systemic grammar of movement that can make the meaning potential of kinetic typography explicit. The paper concludes with an analysis of art works created by David Byrne which use PowerPoint as a medium. Using PowerPoints relatively simple movement grammar, Byrne has nevertheless succeeded in using movement creatively, giving us a glimpse of a future of creative writing which has kinetic typography at its very centre.

The Routledge Handbook of Critical Discourse Studies

John Flowerdew is Emeritus Professor at City University of Hong Kong and a visiting professor at Lancaster University. He has authored and co-authored several books including: Advances in Discourse Studies (with V. K. Bhatia and R. Jones, 2008), Critical Discourse Analysis in Historiography (2012), Discourse in English Language Education (2013) and Discourse in Context, (2014). He serves on the editorial boards of a range of international journals, including Critical Discourse Studies, Journal of Pragmatics, Journal of Language and Politics and Journal of English for Specific Purposes.

2.2 Interpreting Websites in Educational Contexts: A Social-Semiotic, Multimodal Approach

This chapter reports on three research projects concerned with websites. All three have adopted a social-semiotic, multimodal approach (cf. Halliday, M. A. K. (1978). Language as Social Semiotic. London: Arnold; Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Oxon: Routledge; Kress G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold, Kress, G., & Van Leeuwen, T. (2006 [1996]) Reading images: The grammar of visual design (2nd edn). London: Routledge) and developed new tools for understanding the complexity of website design and its implications for educational contexts. They have focused, however, on different challenges and types of hypermedia: (i) user orientation within websites for children, (ii) knowledge construction in online educational interactives for children, and (iii) news design in online newspapers and the literacy demands online newspapers present for TESOL and applied linguistics students. This paper opens by outlining the common starting point of all three projects in relation to the data – treating websites as meaningful texts. It then proceeds to consider differences in the questions and types of websites explored in each study, and how these differences have influenced the way social-semiotic multimodal tools for analysing the data have been developed in each study and complemented with other tools for collecting and analysing data (e.g. interviews with website users, designers and media practitioners). The paper concludes with a brief discussion of challenges that websites continue to present for educational research.

Advancing Multimodal and Critical Discourse Studies: Interdisciplinary Research Inspired by Theo Van Leeuwen’s Social Semiotics

As a founder and leading figure in multimodality and social semiotics, Theo van Leuween has made significant contributions to a variety of research fields, including discourse analysis, sociolingui ...

Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice

ABSTRACT This article addresses the need for critical approaches to social media by bridging the focus on language and other semiotic resources that characterises discourse studies with the broader perspective on social media as social, cultural, economic and technological constructs that dominates media and cultural studies. Specifically, we propose a model for analysing how social media as semiotic technologies, that is, technologies designed to enable and constrain meaning-making, may transform social practices. By incorporating Van Leeuwen’s [2008. Discourse and Practice: New Tools for Critical Analysis. London: Oxford University Press] framework for the critical analysis of discourse and social practice, the model extends the social semiotic approach developed in recent critical multimodal studies of software such as PowerPoint to social media, which function primarily to provide platforms for and commodify social practices, rather than to offer rich arrays of semiotic resources for creating multimodal texts and artefacts. Using the academic social network site ResearchGate and the practice of research peer review, we illustrate the model’s capacity to account for the ways the design of social media platforms – through the semiotic resources they make available and the ways these are presented – enables and constrains their users’ ability to perform key social practices and has the potential to transform these practices.

Reading aloud as performance and its representation on television programmes for children

ABSTRACT The social practice of reading aloud picture books to children, or shared reading, has been represented on many televisions programmes broadcast across English-speaking countries. This article views shared reading as a performance, and explores its transformation on two television shows for children and the potential of such shows to promote reading engagement and literacy development. Taking a critical multimodal perspective, we analyse shared reading in real life and on television as a social practice, focusing on the ways the participants talk about the picture book, relate it to exterior texts or activities, and legitimise shared reading through the employment of multimodal and interactive strategies. The analysis reveals significant differences between actual adult–child shared reading and its representation on television. The comparison illustrates the potential benefits and limitations of television shows in which picture books are read to the viewer, in terms of promoting shared reading among families and supporting young children’s emergent literacy development.