Multimodal Semiotics and Collaborative Design

how they will be appropriated to the social environment, would change from one culture to
another; as “modes are the result of a social and historical shaping of materials chosen by a society
for representation” (Kress 2010: 11).
Although epistemological foundations of the social semiotic framework originates from linguistics,
the its systemic functional approach is extensively applied to other modes, often being modified to
study the meaning potentials in images and text (Kress and van Leeuwen 2001, 2006 [1996), film
(O’Halloran 2004), mediated and non-mediated spaces (i.e. Stenglin 2008, McIlvenny and Noy
2011, Lymer et at. 2011, Wang and Heath 2011; also see O’Toole [1980] for representation of spatial
experience in narrative), or interactive digital software (i.e. Lemke 2005, van Leeuwen 2005, Wade
2008). The multimodal perspective bears a significant importance for my analytical purposes, as
design of virtual places and artifacts inherently synthesize the meaning potentials of various modes
as resources, including images, texts, sounds and music, and interactive 3D objects. Among this
wide trajectory of multimodal approaches, I find O’Toole’s (1994, 2004) model for systemic
analysis of architecture and built spaces most helpful, particularly because of its potentials for
adaptability to virtual places and artifacts in order to emphasize their spatial characteristics as
semiotic meaning potentials.
For Hodge and Kress (1988: 124), “signs in the system of modality have their own history, which
contributes to their meaning and effect.” They acknowledge Halliday’s recognition of modality
within the domain of social relations, which bears traces from not only the context but also
activities of social actors, their prior contestations and the state of representations. . In a social
semiotic system, modality determines the respective positions (stances) of participants in the
semiotic process by organizing and categorizing the social actors, places and sets of relations within
the context of culture. In doing so, every semiotic entity would also always bear traces of modality,
amplifying the various voices –embodied in semiotic resources- that contributed the making of the
sign Modality, in this perspective, is not an internal characteristic of the objects themselves, but it
is realized and practiced during the actual process of collaboratively making meanings.
Multimodal social semiotics theorizes meaning in 3 semiotic instances: (1) semiosis
(communication as meaning complex), (2) multimodality (modes and their relations) and (3)
specific mode (specificities, material affordances, historical and cultural origins of elements)
(Kress 2010: 61).
I consider the process of making multimodal arrangements by using (including copy and pasting,
remixing, etc.) semiotic resources as a form of (bric-)collage (Kress 2010) , where the
resourcefulness and aptness of the modes are chosen according to the semiotic requirements of the
context of situation. Multimodal arrangements are a form of syntagms. Designers frame the chosen
48