Carey Jewitt

The move from page to screen: the multimodal reshaping of school English:

In the move from page to screen a range of representational modes (including image, movement, gesture, and voice) are available as meaning-making resources. This article focuses on the reshaping of the entity ‘character’ in the transformation of the novel Of Mice and Men (Steinbeck, 1937) to CD-ROM (1996). Through detailed analysis the article demonstrates that the shift from written page to multimodal screen entails a shift in the construction of the entity ‘character’. It is also suggested that students’ interaction with the resources of the CD-ROM as a visual text demand that ‘reading’ and the process of learning within school English be thought of as more than a linguistic accomplishment.

Mobile Experiences of Historical Place: A Multimodal Analysis of Emotional Engagement.

This article explores how to research the opportunities for emotional engagement that mobile technologies provide for the design and enactment of learning environments. In the context of mobile technologies that foster location-based linking, we make the case for the centrality of in situ real-time observational research on how emotional engagement unfolds and for the inclusion of bodily aspects of interaction. We propose that multimodal methods offer tools for observing emotion as a central facet of person–environment interaction and provide an example of these methods put into practice for a study of emotional engagement in mobile history learning. A multimodal analysis of video data from 16 pairs of 9- to 10-year-olds learning about the World War II history of their local Common is used to illustrate how students’ emotional engagement was supported by their use of mobile devices through multimodal layering and linking of stimuli, the creation of digital artifacts, and changes in pace. These findings are significant for understanding the role of digital augmentation in fostering emotional engagement in history learning, informing how digital augmentation can be designed to effectively foster emotional engagement for learning, and providing insight into the benefits of multimodality as an analytical approach for examining emotion through bodily interaction.

Screens and the social landscape

The screen may be understood as a designed interface (e.g. television set, computers, information signage) and has a central place in representation and communication of the social landscape, and in the cultural and technological imagination, as well as having economic significance in the digital era of the 21st century. Traditional notions and functions of the screen are continuously shifting – a shift made all the more dramatic by the development of mobile and ubiquitous technologies. In this special issue of Visual Communication on ‘Screens and the social landscape’, we explore the ‘screen’ and related information technologies and ask what its implications might be for how people communicate and interact in public spaces as well as in their display and function in the urban environment, museums and galleries. We set out to explore this question by bringing together professionals and academics working in the broad disciplines of design, computer science, digital technology, linguistics, sociology and cultural studies to focus on the ‘screen’ as a common reference point.

Special Issue: Social media and the visual

Social media are a significant part of contemporary communication. It is estimated that, by the end of 2016, over 2 billion people worldwide will be using social media platforms such as Facebook and Twitter, to communicate, interact, and undertake a range of formal and informal activities and practices (Mccarthy et al., 2014). It is therefore vital that we understand social media platforms and their usage. With a shared interest in the social significance of visual communication in social media (and beyond), the five articles in this special issue contribute to a small body of empirical research on visual meaning making resources in the context of social media environments, including blogs, Tumblr and Instagram. Existing work on social media within the field of multimodal studies includes genre analysis of Facebook publishing (Eisenlauer, 2010) and text compositional practices on Facebook (Bezemer and Kress, 2014), children’s blogging (Abas, 2011), the aesthetic meaning potential of web design (Adami, 2015), the expression of style in Tumblr and Pinterest (Jewitt and Henriksen, 2017, forthcoming) and identity work through repurposing resources (Leppänen et al., 2013), as well as trans-media production across different social media (Adami, 2014). Work focused more specifically on image includes Seko’s (2013) multimodal analysis of self-injury photographs on Flikr, Aiello and Woodhouse’s (2016) multimodal critical discourse analysis of gendered identity in stock photography and Manovich’s (2016a) latest work on style in Instagram photos. 644153 VCJ0010.1177/1470357216644153Visual CommunicationAdami and Jewitt research-article2016

Reshaping Touch Communication: An Interdisciplinary Research Agenda

This workshop aims to generate an interdisciplinary research agenda for digital touch communication that effectively integrates technological progress with robust investigations of the social nature and significance of digital touch. State-of-the-art touch-based technologies have the potential to supplement, extend or reconfigure how people communicate through reshaping existing touch practices and generating new capacities. Their possible impact on interpersonal intimacy, wellbeing, cultural norms, ways of knowing and power relations is far-reaching and under-researched. Few emerging devices and applications are embedded into everyday touch practices, limiting empirical exploration of the implications of digital touch technologies in everyday communication. There is, thus, a real need for methodological innovation and interdisciplinary collaboration to critically examine digital touch communication across social contexts and technological domains, to better understand the social consequences of how touch is digitally remediated. This agenda-setting workshop will bring together HCI researchers and designers with colleagues from sociology, media & communications, arts & design to address key research challenges and build the foundations for future collaborations.

Methodological dialogues across multimodality and sensory ethnography: digital touch communication

There is a significant gap between technological advancements of digital touch communication devices and social science methodologies for understanding digital touch communication. In response to that gap this article makes a case for bringing the communicational focus of multimodality into dialogue with the experiential focus of sensory ethnography to explore digital touch communication. To do this, we draw on debates within the literature, and reflect on our experiences in the IN-TOUCH project (2016–2021). While acknowledging the complexities of methodological dialogues across paradigm boundaries, we map and reflect on the methodological synergies and tensions involved in actively working across these two approaches, notably the conceptualization, categorization and representation of touch. We conclude by homing in on aspects of research that have served as useful reflective route markers on our dialogic journey to illustrate how these tensions are productive towards generating a multimodal and multisensorial agenda for qualitative research on touch.

Conceptualising and researching the body in digital contexts: towards new methodological conversations across the arts and social sciences

The turn to the body in social sciences has intensified the gaze of qualitative research on bodily matters and embodied relations and made the body a significant object of reflection, bringing new focus on and debates around the direction of methodological advances. This article contributes to these debates in three ways: 1) we explore the potential synergies across the social sciences and arts to inform the conceptualization of the body in digital contexts; 2) we point to ways qualitative research can engage with ideas from the arts towards more inclusive methods; and 3) we offer three themes with which to interrogate and re-imagine the body: its fragmenting and zoning, its sensory and material qualities, and its boundaries. We draw on the findings of an ethnographic study of the research ecologies of six research groups in the arts and social sciences concerned with the body in digital contexts to discuss the synergetic potential of these themes and how they could be mobilized for qualitative research on the body in digital contexts. We conclude that engaging with the arts brings potential to reinvigorate and extend the methodological repertoire of qualitative social science in ways that are pertinent to the current re-thinking of the body, its materiality and boundaries.

Methodological Innovation, Creativity and the Digital Body

This chapter contributes to debate on the body and method within qualitative research. It argues that social science engagement with the arts and the relatively unmapped terrain of the digital body has the potential to open up new spaces, questions and methods that can inform social science methodological innovation . I make this argument by exploring how the body in the context of digital technologies was understood and researched across the arts and social sciences, drawing on examples from the MIDAS project. The chapter shows how working across the arts and social sciences can be a catalyst for methodological innovation and explores how working across these boundaries can generate creative thematic synergies for researching the digital body.