Multimodal Semiotics and Collaborative Design

structures and analyzing the context of situation in relation to habitus and social roles of signmakers.
A synthesis of Saussure’s and Peirce’s perspectives were developed and elaborated into a frame of
‘second-order signification’ by Roland Barthes, who also illustrates the socio-cultural turn of
semiotics in the second half the 20 th century. Barthes’ earlier structural analyses of particular
cultural sign systems (i.e. products, places, advertising images) in terms of metalanguages
(Barthes 1967) and myths (Barthes 2009 [1957]) and his ideas on the role of jouissance 3 and
multiplicity of voices (Barthes 1977) are also important in terms of marking the socio-cultural
dimensions of multimodal sign making (Lemke 2009a). Another significant influence is the critical
linguistic approach of Mikhail M. Bakhtin (1981, 1986) 4 , a Russian philosopher and literary
theorist who was also influential in later formulation of theories on dialogic approaches to
communication, proposed important theoretical concepts, including ‘actively responsive
understanding’. Through this way of engagement with semiotic systems, social actors are both
respondents (interpreters) and makers of social meanings. Bakhtin’s critique of Saussure, and later
structuralist social scientists, also lies in his emphasis on the importance of extreme heterogeneity
of speech genres and the difficulty of determining the complete nature of any utterance.
While traditional semiotics may provide the theoretical foundation to deconstruct various
meanings potentials through an interpretative reading of signs and sign-systems, it lacks “an
adequate explanation of why certain meanings get attached to certain symbols at certain historical
times” (Griffin 2009: 337). For van Leeuwen (2005a), the problem with Peirce’s formulation of
motivated signs (icons and indices) is that in structuralist semiotics meaning was still formulated
as ‘objectively existing’ within the sign system, rather than being constructed in the ‘act of sign
production and interpretation’. Social semiotics aims to close this gap by focusing on the signmaker’s
perspective with a rhetorical approach to communication.
Social semiotics formulates communication in terms of semiotic resources and meaning potentials
in particular contexts, rather than signs and their corresponding meanings. The sociolinguistic
roots of social semiotics are specifically apparent in Halliday’s functionalist approach, which aims
to critically discuss the structuralist models on semiotic texts. According to Halliday, the object of
social semiotic inquiry is not structures of particular signs, but sign systems:
With the notion of system we can represent language as a resource, in terms of the choices
that are available, the interconnection of these choices, and the conditions affecting their
3 Loosely translated as ‘pleasure of the text’
4
The so-called ‘Bakhtin group’ includes the Marxist (early-) poststructuralist critique of Voloshinov (1929), who
describes Saussurean structuralism as a form on ‘abstract objectivism’.
43