Multimodality is used to refer to the object of a field of research (i.e. multimodal texts) as well as to the interdisciplinary study of multimodal texts and a set of theories about multimodal semiosis.
As an interdisciplinary study, multimodality examines the semiotic resources people use to communicate and interact in social settings. Multimodality can thus be seen as a broader definition of the study of language, since it studies language, which traditionally is understood as writing and/or speech, as an act of meaning-making which also involves non-linguistics semiotic resources, e.g. typography, color, sound, gesture.
Multimodality also refers more specifically to a theoretical framework in social semiotics about the social use of semiotic resources in representation, communication and interaction. Social semiotic multimodality theory uses concepts from SFL (among other sources) in its description of multimodal semiotics: This includes the concepts of metafunctions, semiotic system, meaning potential, stratification, register, text and context.
Following Jewitt (2009), there are four basic assumptions of social semiotic multimodality theory: 1) language is always embedded in meaning-making in combination with non-linguistic resources, 2) in a multimodal ensemble (i.e. text), each mode can perform different kinds of communicative work/function in a text depending on the affordances of the modes, 3) people create meaning through the selection and combination of resources from modes available to them, and 4) multimodal meaning-making is shaped by the interests of people as social actors in social contexts.
Multimodality is also studied within other theoretical frameworks, e.g. cognitive semantics, media studies, rhetoric and psychology. In general semiotics, the concept is labeled inter-semiosis and has been addressed by key thinkers such as Peirce, Jakobson, Griemas, Barthes and Eco (see Nöth 2000 for overview).
Multimodality is also a term used in perception psychology (van Leeuwen 2011) and Human-Computer-Interaction (e.g. Bernsen 1997), but in these contexts the concept denotes the interaction of sense modalities (i.e. sense integration). It would be interesting to examine possible connections between semiotic and sensorial studies of multimodality (for preliminary work see Boeriis 2009).
In a general perspective, multimodality is often conceptualized as either an interaction between semiotic modes or an integration of semiotic modes (resources) (Poulsen 2014). These perspectives are present in both social semiotic and cognitive literature and should be thought of not as mutually excluding but complementary (see entry on multimodal meaning).
Citing this entry
Poulsen, Søren Vigild. 2015. “Multimodality.” In Key Terms in Multimodality: Definitions, Issues, Discussions, edited by Nina Nørgaard. https://multimodalkeyterms.wordpress.com/
References
Bernsen, N. O. (1997). “Defining a taxonomy of output modalities from an HCI perspective”. In Computer Standards & Interfaces, 18(6), 537-553.
Boeriis, M. (2009). Multimodal socialsemiotik og levende billeder. Ph.D. dissertation, University of Southern Denmark, Odense.
Jewitt, C. (ed.) (2009). The Routledge Handbook of multimodal analysis. London: Routledge.
Nöth, W. (2000). Handbuch der Semiotik. 2. Ausgabe. Stuttgart: Metzler.
Poulsen, S. V. (2014) Mod en analysemetode for webstedet som multimodal tekst. Ph.D. dissertation, University of Southern Denmark, Odense.
van Leeuwen, T. (2011). “Multimodality and multimodal research”. In E. Margolis & L. Pauwels (eds.), The Sage handbook of visual research methods (pp. 549-569). Los Angeles, London, New Delhi: Sage.