Multimodal Semiotics and Collaborative Design

semiotic practices, in all kinds of human society at all periods of human history (Hodge and
Kress 1988: 261).
However, the systemic functional model for linguistic sign systems cannot be directly adopted and
applied for the analysis of multimodal phenomena, such as audio-visual presentations, 3D artifacts
and places. Although these entities often bear verbal signs in combination with other multisensory
information in multimodal arrangements, the analysis requires consideration of the use of various
modes to afford different types of interaction and contribute to the overall meaning by their
situated meaning potentials. For instance, design of virtual places and artifacts in SL often include
linguistic signs to inform visitors, draw attention to a particular theme or guide their movements to
certain locations, while verbal information is placed within a 3D virtual environment where
avatars, objects and surfaces (textures) are organized to facilitate spatial navigation. Within the
multimodal paradigm, Kress (2010) focuses on the three functions by questioning the semiotic
potentials of modes in communication and representation. For the ideational function, he asks if
certain modes can represent meanings about the world of states, actions and events. The main
question for the interpersonal function is to inquire how multimodal signs represent meanings
about the social relations of those who are engaged in communication. Finally, the textual
dimensions is concerned with how ‘message-entities’ which are internally coherent and which
cohere with their environment are formed (Kress 2010: 88).
Lemke (2002, 2009) reconsiders Halliday’s linguistic meta-functions in accordance with the
multimodal conception of digitally mediated communication, and generates a new terminology to
study social functions of multimodal texts. In the course of analyzing how these functions are
realized within multimodal composition, Lemke also offers a comprehensive analysis of
multimedia and virtual environments. In Lemke’s formulation, the ‘ideational’ becomes the
‘presentational function, which (re-)presents a state of affairs, or enacts an activity; the
‘interpersonal’ becomes the ‘orientational’ function, which addressed the (real or imagined) Other,
constructing interpersonal relationships; and finally, the ‘textual’ becomes the ‘organizational
function’ which constructs relationships of parts to wholes, and strands of continuity based on
similarity within difference (Lemke 2002: 304).
As mentioned by Halliday, the context of situation and its structural elements have a relationship
with the semiotic functions and their transformations on the semiosic plane 15 . His functional
sociolinguistic approach to meaning considers not only the semiotic functions (ideational,
interpersonal and textual) as contextual elements in making of signs but also takes the structural
15
The influence of Polish-born British anthropologist Bronisław Malinowski on social semiotic theory is evident in
Halliday’s (1978) theorization of ‘context of situation’ and ‘context of culture’.
65