modes and multimodality

In this week’s first video, Gunther Kress identifies modes as semiotic resources. By this, he means that they are different ways that we make meaning. We can also think of modes as the different forms that semiotic signs can take. On a stop sign, the color red signifies a warning. In this case, color is a mode because it has meaning. A second mode, writing, is used on the sign as well, as the word “STOP”, also conveys meaning. The stop sign uses two semiotics resources, or modes, to coordinate two messages. The writing mode conveys a directive while the color mode conveys the warning that is would be dangerous to not follow that directive.

Image, shape, gesture, moving images, speech, and layout are other modes that we use to communicate with. Each mode has its own affordances, meaning that each is more effective at conveying certain types of messages or at conveying them in certain ways. Understanding these affordances allows us to choose those modes that will best convey our intended message(s). Multimodality is the coordination of a number of modes to convey a message or compound set of messages. The stop sign is a simple example of multimodality. An iBook could be an example of a more complex use of multimodality.

In this week’s reading, Jewitt points out that different modalities can be at work in the same object. An obvious example of this is a video with sound. A more interesting example may be in writing. As we discussed earlier this semester, the written word signifies based on our understanding of symbolic relationship between the markings and certain concepts (or perhaps between the markings and certain sounds that themselves have symbolic relationships with certain concepts). However, the appearance of those markings may also signify. The signification of “what are you doing?” is different from “WHAT ARE YOU DOING?” In this case, the sentence signifies both as a set of words and as an image, as the appearance it self has significance.

Multimodality has been a part of human communication at least since we began to combine gestures and words. Digital technology, offers opportunities to combine a wider variety of modes than was previously possible within a single frame. As creators within digital media, however, our challenge is not just to see how many modes we can use to express the same idea, but to explore ways of coordinating these different modes to construct complex and compelling messages in ways that would not be possible in a non-digital space.

Videos and Readings for September 28

“Multimodality, “Reading”, and “Writing” for the 21st Century / Carey Jewitt / Find at – Academic Search Complete