Multimodal Semiotics and Collaborative Design

comparatively analyzing the multimodal perspective with the anthropologically informed
ethnographic methodologies, Sarah Pink (2011) argues the tensions and challenges of combining
these two approaches. According to Pink, the main difference between anthropological and
multimodal approaches results from ”their understandings of both the relationships between the
senses and the way a ‘sensory’ approach might be mobilized as a methodology” (Pink 2011: 262).
Pink’s phenomenological anthropology describes ‘affordances’ in relation to perceptual processes
rather than understanding of culture as a set of rules and representations that can be categorized
and differentiated through distinct sensory channels (of sensory information). Taking Pink’s
(2011) comments on the challenges of doing an ethnographically guided social semiotic analysis of
multimodality, I focus on modes as interrelated (and mutually supportive) ways of representing the
rhetorical intentions of sign-maker(s). In doing so, I emphasize the interconnectedness of modes as
representational tools (Ivarsson et al. 2009), and focus on not only the direct sensory channels of
information (such as audio-visual texts) but also social practices in designed places (such as
movement, orientation, spatiality and interactivity). My analytical perspective of what modes can
do, or what designers can do with them, also bears traces of the phenomenological approach; in
that I am also interested in how individuals experience affordances and limitations in their social
semiotic experiences in multi-sensory (and multimodal) environments.
I consider social semiotics as and epistemological bridge between the structuralist roots of
traditional semiotics and the critical socio-cultural perspective on production of sign systems and
their social functions. The sociocultural tradition in communication research considers
communication as a dynamic process in which meanings, identities and social structures are coproduced
by the participants of communicative social processes (Craig and Muller 2007, Griffin
2009). The symbolic interactionist perspective focuses on the meanings that arise out of social
interaction, through which humans act towards other people or things (Griffin 2009). For socialpsychologist
George Herbert Mead (2007 [1934]), it is the social process that gives birth to thought
and communication, as mutually empathetic responses to the message of communication; and
social control operates as a reference to ‘organized social processes of experience and behavior’
through awareness of the other. In this framing, resources for communicating cannot be taken
separately from their possible social uses, and discourses of various actors use them. Therefore,
defining a tool essentially requires construction of a complex nexus of co-production practices. One
can associate Mead’s sociocultural perspective with the notion of affordances, as the trajectory of
affordances and constraints are also determined by the potential for meaningful action through use
of objects (Gibson 1986, Norman 1988).
As a prominent constructivist perspective to classical sociology, Bruno Latour’s (2005) Actor-
Network Theory (ANT) approach also questions the notion of social, and puts forth an intricate
69