Multimodal Semiotics and Collaborative Design

aspects of multimodal entities for representation, intended for their users/interpreters to perceive
the design as a coherent semiotic whole. “The form and character of sub-textual units, the
modules, are derived from their functions and uses within the text“(Kress 2010: 148).
Arrangements are made as spatial and temporal ensembles in the social world. Through
orchestration of modes and movement in time and space, multimodal arrangements offer their
interpreters possible paths/routes for meaning-making (Jewitt 2009). Kress (2010) mentions
division of semiotic labor in making of the ensembles of meaning, and their orchestrations. In the
introduction to his edited handbook on multimodality, Carey Jewitt (2009) describes the two
central scopes of the multimodal approach as description of semiotic resources and investigation of
intersemiotic relations between modes. Van Leeuwen (2005) explains the unity of internal
orchestration of signs as multimodal cohesion, which consists of simultaneous organizations of
rhythm (time), composition (space/layout), information linking (cognition) and dialogue
(communication). Orchestration, as an analytical keyword and a metaphor, has central importance
for my purposes in understanding the social semiotics of collaborative design (in VWs), in two
particular ways. First, the orchestration of modes and modal configurations (of semiotic resources)
in composition of multimodal texts is implied. The second form or orchestration is a way of
organizing and managing the social practices through which signs are co-produced.
In fact, all texts are considered as material artifacts, which are always already multimodal in terms
of textual composition (Lemke 2009a, Kress 2010). In other words: “the world of meaning has
always been multimodal” (Kress 2010: 174) Following this logic, I take the position proposed by the
multimodal approach to social semiotics, and deem that “fundamentally, all semiosis is
multimodal” (Lemke 2002: 302). I consider (written and/or spoken) language as only one of these
communicative resources; while drawings, music, video, gestures, 3D artifacts and architectural
constructions can also be used as modes in representation and communication. Each mode is used
in specific contexts and affords certain social actors and constrains others. A particular message,
information of emotion may be communicated with a particular mode may not be effective with
others or it may need modal translation. Similarly, the indexicality of visual modes (such as images,
movies and interactive graphics) can be used to enhance the affordances of the semiotic
arrangements for communicating messages. For instance, in Elements of Semiology, Roland
Barthes (1967: 26-27) mentions clothes as written about, clothes as photographed, and clothes as
worn to emphasize the effects of modality as semiotic resource. Within the multimodal perspective,
different ways of experiencing an architectural space (such as through actual physical presence,
browsing images or watching videos of it, or navigating in an interactive virtual representation)
would certainly affect the construction of spatial experiences. In terms of the design of virtual
places, as any other spatial composition, syntagmatic relations within the layout of the overall
49