Multimodal Semiotics and Collaborative Design

object’s consumption refers to its sign value. Baudrillard’s social functions of object systems in
production of discourses can also be related to Henri Lefebvre’s (1991 [1974]) social constructivist
notion of social space, as well as Csikszentmihalyi ‘s (1998) framing of ‘design and order in
everyday life’. In this view, the interaction between social action (such as production or
consumption) and symbolic communication takes place in the socio-cultural domain, and it is
largely based on circulation - and transformation - of signs. Social semiotic theory of
communication attempts to rebalance these power and attention issues by putting equal emphasis
on both the initial maker and the interpreter of signs, and by sketching out a framework for
communication that begins with interpretation of and response to a prompt, and transformation of
messages by multimodal means (Kress 2010). Meanings are made in two ways: inwardly
productive (interpretation and transformation) and outwardly productive (sign-making). On one
hand, the design process provides the semiotic grounds for communication by the production of
the message. On the other hand, the interpreter’s engagement, and active attention provide
another translation 18 as a new, inwardly focused sign system (Kress 2010).
The multimodal perspective is also challenged about the intermedial aspects of media convergence
and ‘culture of appropriation’. In his study on digital media production and ‘culture of
appropriation’, Øystein Gilje (2008) integrates terms from socio-cultural theory and social
semiotics to study student-designers’ engagements with literary practices/events through mastery
and appropriation as mediated action. Gilje analyzes how his students ‘take culture apart’ through
literacy events (Gilje 2008: 32), for which he argues that a perspective both on learning and signmaking
is necessary. He analyzes not only the observation of semiotic resources and their
perceived affordances for specific social actions, but also the transformative relationships between
resources and the means by which they are mediated. Gilje’s most important argument, for the
purposes of this thesis, is on the necessity to supplement social semiotics with socio-cultural
theories, which is resulted from the theory’s lack of interest in exploring relationships between
social agents and cultural tools. When Gilje’s (2008) perspective is considered with respect to
Sarah Pink’s (2011) phenomenological anthropology oriented critique of the approach, the
possibility and necessity to supplement the systemic functional multimodal framework with
ethnographic methods of inquiry become more evident.
18
In general terms, translation means the movement of meaning across different modes, modal ensembles and cultures.
Semiotic translation is theorized by two terms: transduction (movement across modes) and transformation (staying
within the mode but re-ordering syntagms) (Kress, 2010). These two practices of translation depict the central aspects for
semiotic change, in that they both reflect and shape cultural classifications and representations.
72