Multimodal Semiotics and Collaborative Design

Ferdinand de Saussure, the triadic semiotic model of Charles Sanders Peirce (1998 [1894]), and
the relatively poststructuralist and multimodal semiology of Roland Barthes (1967, 1977).
However, I build my arguments primarily on the social semiotic approach, which is marked by the
systemic functional socio-linguistics of Halliday (1978), and its contemporary multimodal
extensions theorized by Kress (2010) , van Leeuwen (2005a) 4 and Lemke (2009a, 2009b), among
several others. In this chapter, I also revisit the theoretical debates on the notions ‘space’ and
‘place’ in relation to ‘virtual place’ and lay out an experiential semiotic framework to study places as
social constructs. I exemplify my theoretical arguments by analyzing the various design strategies
that are employed by the VW designers, and outline the systemic functional framework of
experiential, interpersonal and textual meta-functions for multimodal analysis (O’Toole 1994).
In the fourth chapter, which is also the final section of the theoretical framework, I review design
research literature to support my theoretical arguments on the socio-semiotic nature of coproduction
activities in VWs. In my review, the central terms that shape the analytical framework
are affordances and constraints, and their relations to the socio-semiotic notion of meaning
potentials. I refer to theories on and socio-cultural ecologies of design, design methods and
product semiotics to discuss certain epistemological and methodological frameworks in design
research in relation to the main research question. I argue that an in-depth understanding of
certain ‘designerly ways of knowing’ (Cross 2007) supports the social semiotic framework and
allows the analysis to expand beyond the conceptual limitations of socio-linguistics. This chapter
also includes the discussions on collaborative design processes, creative problem-solving with
distributed stakeholders (Arias et al. 2000), and emergence of design solutions (Grossen 2008).
In the fifth and sixth chapters, I outline and discuss the analytical and methodological frameworks.
My emphasis here is on the combination of multimodal analysis with a participatory ethnographic
approach, and the poststructuralist framing of design as both context and content of multimodal
discourses. In the analytical framework, I present and discuss the epistemological, ontological and
methodological motivations for the construction of the social semiotic analysis, and I outline the
particular ways in which I found it useful to support the multimodal approach with relevant social
theories. Here, I outline two central methodological perspectives: multimodal (systemicfunctional)
analysis (i.e. Kress and van Leeuwen 2001, van Leeuwen 2005, Jewitt 2009, Kress
2010); in combination with MDA (i.e. Scollon 2001, Scollon and Scollon 2003, 2004, Norris and
Jones 2005), which is not a separate field of study but an expansion of the social semiotic theses. In
the Methodology chapter, I substantiate the discussions on the analytical framework, and present
the multiple-case study approach, methods of data production and strategies for sampling the
4 For Kress and van Leeuwen’s perspectives on the analysis of multimodal discourse, also see Kress and van Leeuwen
(1996, 2001).
12