Get e-book Analyzing Multimodal Interaction: A Methodological Framework

Free download. Book file PDF easily for everyone and every device. You can download and read online Analyzing Multimodal Interaction: A Methodological Framework file PDF Book only if you are registered here. And also you can download or read online all Book PDF file that related with Analyzing Multimodal Interaction: A Methodological Framework book. Happy reading Analyzing Multimodal Interaction: A Methodological Framework Bookeveryone. Download file Free Book PDF Analyzing Multimodal Interaction: A Methodological Framework at Complete PDF Library. This Book have some digital formats such us :paperbook, ebook, kindle, epub, fb2 and another formats. Here is The CompletePDF Book Library. It's free to register here to get Book file PDF Analyzing Multimodal Interaction: A Methodological Framework Pocket Guide.

A few years ago, I set out to ethnographically study the everyday identity construction of two women living in Germany. As a result, I developed the framework that I explicate in this book. Thus, this framework for multimodal interactional analysis is a practice-based methodology. I would like to particularly express my gratitude to Ron Scollon for the hours of discussion, his encouragement, and comments on the manuscript; Gunther Kress for his insightful comments; and Heidi Hamilton, Theo Van Leeuwen, and Ruth Wodak for their fruitful conversations.

I spoke with numerous people at universities, conferences, and also via email about my framework, and all of these people had some impact on the development of my thoughts. I am grateful to Tom Randolph for his detailed attention to wording. A multimodal framework for the analysis of multimodal interaction can only be explicated multimodally. Norris, Kevin P. Shapiro, and Suzanne S. Williams, DMD, who generously agreed to be videotaped and to have their pictures incorporated in the book to help me explicate the methodological framework.

My family has taken part in the book in countless ways, some visible and others not visible in the book at all. My sons, Kevin and Luke, have spent hours reading in the same room that I was writing in, and at other times they have made sure that I escaped from my desk. My partner, Alan, has spent hours reading and rereading the manuscript, giving me comments about wording. While I am indebted to all of these people, as well as many others who helped in my thinking, the statements made and the views expressed are solely the responsibility of the author.

Imagine, for example, a simple two-person interaction, a conversation with a friend. If there is music playing in the background, even though you are not focusing on the music, you are aware of it. All of these elements play a part in this conversation. You may speak quickly or slowly, depending on the music playing in the background or the given environment that the interaction takes place in. Intuitively we know that we draw on all of these communicative channels or modes when interacting with others. We also know that we are aware of many things that surround us while we interact with others.

Let us keep thinking about a conversation. No matter where it may take place, you are certainly aware whether other people are present in close proximity. Thus, if your conversation takes place at a table in a cafeteria, you are aware of others talking, eating, or passing by your table. You may not take much notice of these other people, because you are focused on your conversation, but you are aware of them nevertheless. Interactional meaning Generally it is assumed that we can communicate best through our use of language.

Language seems to have the most informative content, which can easily be employed without a need for other channels. We may speak on the phone, write emails, or go to chat-rooms. In each case, we use language, either spoken or written, to communicate. But when thinking about TV or the Internet, it is clear that we also communicate through images. Often, viewing an image may carry more 2 Multimodal interaction 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 communicative meaning than reading a description of the very same thing.

This realization questions the notion that the process of communicating is dependent upon language. Just as moving images or still photos can communicate meaning to the viewer, nonverbal channels such as gesture, posture, or the distance between people can — and do — carry meaning in any face-to-face interaction. All movements, all noises, and all material objects carry interactional meaning as soon as they are perceived by a person.

Previously, language has been viewed as constituting the central channel in interaction, and nonverbal channels have been viewed as being subordinated to it. While much valuable work on the interplay between the verbal and nonverbal has been established, I believe that the view which unquestionably positions language at the center limits our understanding of the complexity of interaction. Therefore, I will step away from the notion that language always plays the central role in interaction, without denying that it often does.

Language, as Kress et al. In this view, gesture, gaze, or head movement may be subordinated to the verbal exchanges going on as has been shown in much research. However, gesture, gaze, and head movement also may take the superior position in a given interaction, while language may be subordinated or absent altogether. Alternatively, sometimes many communicative channels play an integral part in a given interaction, without one channel being more important than another. While we all intuitively know that people in interaction draw on a multiplicity of communicative modes, and that people in interaction are aware of much more than just what they are focused upon, an analysis of such multimodal interaction brings with it many challenges.

While spoken language is sequentially structured, gesture is globally synthetically structured, which means that we cannot simply add one gesture on to another gesture to make a more complex one. With gestures, this is not possible, since gestures that are linked to language inform about global content or intensity. Gaze, however, may be sequentially structured, and during conversation it often is. But, during other interactions, gaze can be quite random. For example, when you walk through the woods with a friend, your gaze Multimodal interaction 3 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 may wander randomly, focusing on a tree, a rock, or nothing at all.

Then there are other communicative modes, which are structured even more differently. Chairs are usually located around a table, or a reading lamp is located next to an easy chair. Thus, different modes of communication are structured in very different ways. Another challenge for the analysis of multimodal interaction is the fact that different communicative modes possess different materiality.

For example, spoken language is neither visible nor enduring, but it does have audible materiality. The mode of print has more visible materiality and is also enduring; and the mode of layout, thinking about furniture, for example, has highly visible materiality and is extensively enduring. When conducting multimodal analysis of human interaction, we need to consider the human mind. I am partial to the notion of a duality of mind, as discussed in great detail by Chalmers Second, Chalmers speaks of the phenomenal concept of mind. This is the concept of mind as conscious experience or simply the state of awareness.

While both concepts of mind often if not always co-occur, this division allows us to focus on just one part of consciousness. Chalmers proposes that: A good test for whether a mental notion M is primarily psychological is to ask oneself: Could something be an instance of M without any particular associated phenomenal quality?

If so, then M is likely psychological. If not, then M is phenomenal, or at least a combined notion that centrally involves phenomenology. Chalmers In interactional multimodal analysis, we are not much concerned with the perceptions, thoughts, and feelings that people are experiencing, but we are concerned with the perceptions, thoughts, and feelings that people are expressing. We can surmise that some perceptions, thoughts, and feelings 4 Multimodal interaction 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 that are expressed by someone are also somewhat experienced by that person, even though the actual experience and the expression of the experience should not be viewed as a one-to-one representation and may be as diverse as to contradict each other.

We can also surmise that not every perception, thought, and feeling that a person experiences is expressed. A part of conscious experience, namely interactional awareness, can be analyzed qualitatively. In order to do this, we analyze not only the messages that an individual in interaction sends, but also how other individuals in the interaction react to these messages. Thus, there is a constant tension between what a person consciously does and what that person expresses.

Interaction, then, is the exchange of communicated expressed, perceived, and thereby interpreted experiences, thoughts, and feelings of participants. Let us return to our example in a cafeteria, and imagine that there is a big window to the right of the table. When you gaze out of the window during the conversation, your friend may interpret this action as a sign of either boredom or deep concentration, or may not take any notice of it.

In multimodal interactional analysis we are only concerned with what individuals express and others react to. We are not concerned with what people are actually perceiving, thinking, and feeling, which may sometimes be different and more complex than what they express. Multimodal interactional analysts set out to understand and describe what is going on in a given interaction. Furthermore, it is of utmost importance to realize that one and the same action — like looking out of a window — can have many different meanings, intentional or unintentional.

While we react to the words, gestures, facial expressions, etc. For example, when you visit a retail store to buy a present for your mother, and you talk to the sales person, you are not interested in what the sales person is thinking, but whether or not the sales person is showing you items that your mother would like. Considering what others are thinking is rare and usually comes about only in intimate situations.

Yet, what we do consider is an aspect of the phenomenal concept of mind. We consider whether a person is kind to us or shows anger, whether a person we interact with reacts in an appropriate manner or not, and whether or not somebody is paying attention to us.


  1. Analyzing Multimodal Interaction A Methodological Framework by Sigrid Norris Information;
  2. Stray (Shifters, Book 1).
  3. JPod: A Novel.
  4. Prof. Sigrid Norris – Webinar on Multimodal Discourse (Sept 9th).

As humans, we are excellent at noticing whether others are paying attention, and we learn from early childhood onwards that we are supposed to realize whether we can approach another human being or whether the other person is occupied. Chaplin, ; emphasis is my own Awareness and attention can be used somewhat interchangeably, even though there are slight differences in their meaning.

Titchener a state of sensory clearness with a margin and a focus. Let me refer once more to the Dictionary of Psychology: attention level: the degree of clarity of an experience ranging from unconsciousness total lack of awareness to focal attention vivid awareness. All participants in a given interaction react to the expressed aspect of the phenomenal mind of others, and it is this aspect of mind that is observable by the analyst. We can be phenomenally conscious of something without attending to it. Schoolchildren are gathering on the corner of the sidewalk close to the school bus.

When focusing on the crossing guard, we see her communicating with the drivers and the children at the intersection in various ways, paying attention to them in varying degrees. Simultaneously, the crossing guard is aware of the children, who are gathering on the sidewalk, although she pays little attention to them. She is pointing to the front, motioning with her left hand to communicate to the drivers that they may now turn. In fact, we can tell when she is shifting her attention from one section of the intersection to the next.

This shifting always happens a moment before she turns to give the next part of the intersection her full attention. While such shifts of attention are discussed in detail in Chapter 6, I would like to point out that the shifts are of particular importance. You may cross now! The second image in the third row shows the children crossing the street. Structure of the book Interaction is an everyday occurrence, and in each day-to-day interaction we use a multiplicity of communicative channels. In order to analyze interactions in their complexity, a theoretically grounded methodological framework is essential.

Before setting out to delineate such a framework, in Chapter 2 I give an overview of a multiplicity of communicative modes. I introduce each mode separately by referring to some background literature and then explicating the mode with a real-time example. In Chapter 3, I discuss practical issues of multimodal transcription. I talk about the method of transcription and video analysis and give a step-by-step guide for accomplishing such transcription. Then I move on to the methodological framework, which makes it possible to incorporate numerous communicative modes in an analysis and links interaction to cognition.

Chapter 6 illustrates the semantic structuring devices that we all utilize in order to structure our own focused actions. We express these to the others present, which indicates that they have an interactive as well as a psychological function. In Chapter 8 I mull over the notion of interaction and conclude with what the proposed framework allows us to do.

These are useful as a study guide and as a quick reference. I also give some ideas of assignments that teach the student how to analyze interactions from a multimodal perspective. Interactional meaning: All movements, noises, and material objects carry interactional meaning when they are perceived by an individual. Structure of communicative modes: Modes of communication may be sequentially, globally synthetically, or functionally structured, or they may appear to be quite random.

Awareness and attention: In multimodal interactional analysis, we consider only the awareness and attention that individuals in interaction express and others react to. Actual experiences, thoughts and feelings and expressions of experiences, thoughts and feelings are not viewed as one-to-one representations. Multimodal interactional analysis grew out of interactional sociolinguistics, mediated discourse and nexus analysis, and multimodality in combination with the technology of video cameras and computers.

I take from interactional sociolinguistics its focus on realtime interaction and language in use, from mediated discourse and nexus analysis their emphasis on mediated action, and from multimodality its highlighting of the importance of taking into account other semiotics such as music, color, and gesture. In this book, I purposefully cross the boundaries between linguistics, nonverbal behavior, and the material world, in order to show that all three directions of research come together when we think about human interaction.

Each action is mediated. Lower-level action: The smallest interactional meaning unit. Frozen action: Higher-level actions, which are entailed in material objects. We can say that layout is a mode, which would include furniture, pictures on a wall, walls, rooms, houses, streets, and so on. But we can also say that furniture is a mode. A mode has no clear boundaries. Modes such as proxemics, posture, head movement, gesture, gaze, spoken language, layout, print, music, to name several, are essentially systems of representation.

A system of representation or mode of communication is a semiotic system with rules and regularities attached to it Kress and Van Leeuwen, I like 12 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Communicative modes to call these systems of representation communicative modes when I emphasize their interactional communicative function. A communicative mode is never a bounded or static unit, but always and only a heuristic unit. A system of representation — a writing system, for example — is usually thought of as a given system that exists in and by itself once it is developed.

While such a system changes over time, we can describe the system in the form of dictionaries and grammars, showing the rules and regularities that exist. Taking this thought further, we could describe systems of representation like gesture, gaze, layout, etc. Communicative modes and interaction When observing an interaction and trying to discern all of the communicative modes that the individuals are utilizing, we soon notice that this is a rather overwhelming task. People move their bodies, hands, arms, and heads, and while the observer may try to understand the content of what is being spoken, they have already missed many important messages which each speaker is sending — intentionally or not — and the other speaker is reacting to through other modes.

Yet, a multimodal interactional analysis is not as impossible as one may think. First, the analyst needs to become skilled at distinguishing one communicative mode from others. Then the analyst is ready to investigate how modes play together in interaction.

In this chapter, I give an overview of an array of selected communicative modes. These selections are by no means meant as a complete list, as there are many communicative modes that I do not address here, like facial expression, dress, object handling, and color. Each selected mode is described in a separate section, giving the reader the opportunity to concentrate on one mode at a time.

Then, in the second part of each section, I draw on our knowledge about the system of representation, and illuminate it with a real-time example. When working with real-time interaction, we discover that there is constant tension and contradiction between the system of representation and Communicative modes 13 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 the event.

Individuals in interaction draw on systems of representation while at the same time constructing, adopting, and changing those systems through their actions. In turn, all actions that individuals perform are mediated by the systems of representation that they draw on. Unit of analysis As I mentioned in the introduction, the differing structures and materiality of modes were challenges that needed to be overcome, as an integrative multimodal approach required a single unit of analysis that allowed for the communicative modes to be structurally and materially different.

In multimodal interactional analysis, the mediated action is the unit of analysis, and since every action is mediated, I will simply speak of the action as the unit of analysis. The action as unit of analysis, however, is still a complicated issue, because there are smaller lower-level and larger higher-level actions.

Now, take a meeting among three friends, which can be called a conversation, a moment in time, or a social encounter. The meeting is taken to be the higher-level action. This higher-level action is bracketed by an opening and a closing of the meeting and is made up of a multiplicity of chained lower-level actions. All intonation units that an individual strings together become a chain of lower-level actions. All gesture units that an individual performs become a chain of lower-level actions. All postural shifts that an individual completes become a chain of lowerlevel actions.

All gaze shifts that an individual performs become a chain of lower-level actions, and so on. Consequently, all higher-level actions are made up of multiple chains of lower-level actions. The chains of lower-level actions are easily understood when talking about embodied communicative modes like gaze, gesture, or spoken language. But disembodied modes can play just as important a role in interaction as do the embodied modes. Modes like print — a magazine that participants are reading or that is just lying on a table for anyone to see; or layout — the furniture in a room, pictures hung on a wall, or a busy street with signs, buildings, and walkways, are disembodied modes.

These modes can also be analyzed by using the unit of analysis, the mediated action. However, here the unit of analysis is the frozen action. Frozen actions are usually higher-level actions which were performed by an individual or a group of people at an earlier 14 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Communicative modes time than the real-time moment of the interaction that is being analyzed.

These actions are frozen in the material objects themselves and are therefore evident. When we see a magazine lying on a table, we know that somebody has purchased the magazine and placed it on the table. Thus, the chains of lower-level actions that somebody had to perform in order for the magazine to be present on the table are perceptible by the mere presence of the magazine itself. The same is true for furniture, pictures on walls, houses in cities, or a CD playing.

Material objects or disembodied modes, which we are concerned with here because individuals draw upon them in interaction, necessarily entail higher-level actions which are made up of chained lower-level actions. Each lower-level action is mediated by a system of representation which includes body parts such as the lips, etc. Every higher-level action is bracketed by social openings and closings that are at least in part ritualized. When the three friends get together for their meeting, the higher-level action of that meeting is opened up by the physical coming together of the friends and by ritualized greetings.

Similarly, this overarching higher-level action will be ended by ritualized greetings and a parting of the individuals. To understand this concept, we can think about ice. Similarly to the freezing of water, actions are frozen in the material objects present in interaction. When you are sitting in a cafeteria with a plate of food in front of you, and a friend comes by, your friend will know that you went through a chain of actions for the plate of food to be there.

source link

Join Kobo & start eReading today

Thus, your actions of getting a plate of food in a cafeteria are frozen in the very plate of food that is now standing in front of you. However, these actions are visible in frozen form in the materiality of the plate and the food on it. There is usually more than one higher-level action frozen in a material object. These higher-level actions are the most visible frozen actions that are entailed in the magazine, because they are the most recent actions that an individual had to perform for the magazine to be present on that table.

There are also other frozen actions entailed in that magazine, one of which would be the production of the magazine itself. This illustrates that the frozen actions which are closest in time and place to the ongoing interaction are the ones that individuals usually focus upon. How are these actions mediated? Which actions are lower-level actions, and which are the higher-level actions? Spoken language: Generally, sequentially structured, but often there is also simultaneity of language, i.

Simultaneous utterances are also sequentially structured. Talk in interaction Spoken language is the mode that has received the most attention, developing rapidly in the last 30 years, and it is impossible to touch on its many concepts in such a short section as this. Therefore this is a fragmentary introduction, which includes the intonation unit, and three levels of discourse as viewed from a mediated discourse approach: three levels of higher-level actions.

Spoken language, although generally a sequentially organized mode, in which the smaller parts add up to larger parts, is not always and only utilized sequentially, but also simultaneously. As Chafe has pointed out, spoken language is uttered by speakers in intonation units. These units are physiologically conditioned by the need to breathe. Thus, each intonation unit is bracketed by the inhaling of air. Speakers do not speak in complete sentences, and speaking is often interrupted by false starts, pauses, and breaks.

Every intonation unit with all of the other sounds that a speaker may utter, has interactive meaning, and therefore, needs to be noted down in transcripts. Sequencing of these lower-level actions intonation units creates, in part, the higher-level actions that individuals in interaction construct.

Thus, people use language interactively on many different levels and for many different reasons. First, let us think about only three of the many different levels at which people can communicate by using language. To make this notion of different levels clear, let us think about questions. Such questioning may be seen on a city street, where somebody is asking a passer-by for directions.

Now think about a party at which a group of people are engaged in conversation. Suddenly, one person in the group asks a question that has nothing to do with the topic at hand. Now, the question is functioning as the initiation of a topic shift. When a question functions as a topic shift, the interaction is not limited to the completion of a sequenced adjacency pair, but is rather extended in other directions.

Questions may be used in still other ways. Two people may be in a situation which may or may not involve conversation. Here, the question functions as an opening to a conversation. This opening to a conversation has the potential to construct an even higher level of action than the topic shift in the example above, and the two people may be engaged in a long conversation following this conversation-opening question. Communicative modes 17 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Spoken language, with its high degree of informative content, has been investigated from many different angles and from many different theoretical and methodological perspectives.

In most of these perspectives, language is viewed as the primary mode of communication. While language certainly is a very important mode, it is not always the case that it plays the primary communicative role. Language is one of many modes that people draw upon in interaction, and the actual role that language takes in a given interaction has to be determined through analysis.

Then, it plays a subordinate role for a while, before again taking on the primary role. Dentist—guardian interaction In this interaction, the dentist is giving a child dental treatment. Background The child had sat down in the chair, while the dentist was exchanging greetings with the child and the mother.

Then the dentist had engaged the mother in some conversation about dental care for another child. Excerpt After a while, the following exchange occurred, in which the dentist addresses the mother see transcription conventions on p. With this question, the guardian invites the dentist to depict which conversational direction the dentist was opening up with her topic-shifting question in line 1 , displaying curiosity. Here, she speaks of Brenda, who used to be her assistant and who recently had moved to Frankfurt. In line 16 , the guardian overlaps with the dentist in line 15 , showing that she now knows where the conversation is going as she also knew Brenda.

This is a clear case of cooperative overlap, as both speakers become more animated and keep speaking without interruption. Communicative modes 19 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 While I only analyzed the spoken discourse in this section, I take up this same example on pp. Assignment Tape-record a conversation and transcribe a short section, which has a beginning and an end. Proxemic behavior: Culturally conditioned and integrated with the higher-level actions performed as well as the environment in which the interaction takes place.

Gives insight into the kind of social interaction that is going on. Utilizing space Proxemics is the study of the ways in which individuals arrange and utilize their space. We are concerned with the distance that an individual takes up in relation to others as well as to relevant objects. The interest then lies in the relationship between the higher-level actions performed and the distances taken up by the participants.

As an example, think of meeting a friend who is standing at a bus stop. Meeting a friend is a higher-level action which is comprised of a multiplicity of chained lower-level actions. At the moment we are only concerned with the chains of proxemic lower-level actions.

Thus, one chain of lowerlevel actions is made up of your proxemic behavior, and another one is made up of the proxemic behavior of your friend. This initial physical positioning of you towards your friend consists of a chain of lower-level actions that you are performing. The space that you have taken up in relation to your friend indicates your social relationship and is appropriate for the higher-level action of meeting this friend at the bus stop. This step towards you is now a lower-level action that your friend performs in order to talk to you in private.

The new space that your friend is taking up towards you is then the appropriate distance to tell a secret. Proxemic behavior is tightly integrated with the higher-level actions that are being performed, and at the same time, proxemic behavior indicates social relationships.

Hall distinguishes four distances: intimate distance, personal distance, social distance, and public distance. He notes that proxemic behavior of this sort is culturally conditioned and entirely arbitrary, but is binding on all concerned. Intimate distance, personal distance, social distance, and public distance are never set values, but always and only general notions. We speak of ranges from close to far phases, when speaking of any one of these four distances to emphasize the range of possible spacing.

This nevertheless depends on the culture and also upon the environment that the interaction takes place in. Generally, in closed spaces — a subway or a bus, for example — personal and social distance is much closer than in open space. In interactional analysis, working with real-time, we discover that the individuals in interaction draw on the system of representation of proxemics that they have learned through socialization. However, the individuals are rarely — if ever — aware of such a system of representation being available to them.

Individuals construct the mode of proxemics through their own actions, and we can learn much about the interaction going on if we are aware of the general proxemic behavior of the particular individuals under study. When analyzing the mode of proxemics in interaction, we always need to remember that proxemic behavior is culturally habituated; and that, overall, while closeness tends to indicate intimacy and distance tends to indicate formality, the actual distances are completely arbitrary. Proxemic behavior is different in different cultures and subcultures, and it may be somewhat different for each individual.

Accordingly, we can observe video clips of one individual in many different circumstances interacting with many different people. For an informal sociolinguistic interview, the interviewer prepares some open-ended questions, but is also prepared and even inclined to cross the boundary between interview and conversation. In the following example, 22 Communicative modes 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 I interviewed a German—Italian woman, whom I call Anna, about her self-perceived identity. The time of the interview was set by the interviewee, as I had let her know that I was available whenever she would have the time to talk for a while.

She chose one mid-morning, during which her husband was at work, her oldest son in school, and her younger son in preschool. Her youngest child, whom I will call Katie, was the only other person present besides Anna and myself. When I walked into the apartment, Anna was ironing.

She had placed bundles of clothes on most of the chairs and pointed out that she had left one spot on the sofa for me to sit in. I placed the taperecorder on the chair closest to Anna, and then sat down in the spot that she had reserved for me. Plate 2. When focusing on the participants present during this interview, it is noticeable that the distance between Anna and Katie is much shorter than the distance between Anna and the interviewer. While Anna and Katie are positioned back-to-back, their close proximity indicates a mother— daughter Anwesenheit. Communicative modes 23 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 The distance between the interviewer and Anna is rather large.

As we recall, Hall distinguishes between personal and social distance. For Anna, who is German— Italian far personal distance is exhibited between her and her daughter, while the distance that Anna takes up to the interviewer demonstrates far social distance. The social distance that Anna takes up to the interviewer demonstrates that she views this informal sociolinguistic interview as a formal occasion, which is even more apparent when investigating other communicative modes.

Nevertheless, Anna is talking freely about her self-perceived identity, and the interview is not strained or strictly memorized, either. As I mentioned above, Anna chose the time and place for the interview. Therefore, she also chose the actions of ironing and watching TV. She had positioned the ironing board at a certain distance from the TV, so that she would be able to look up from her action of ironing to watch the program.

The distance that Anna takes up with respect to the ironing board is predetermined by her actions of ironing, and the distance she takes up with respect to the TV is predetermined by her actions of ironing and by visual space. The fact that Anna has positioned herself at these distances to these objects allows her to perform the actions she is performing. In turn, the actions that she is performing communicate information about her family and national identities. People taking a walk in a park, standing at a bus stop, and walking in a busy shopping street or mall would be perfect situations.

Then you want to look at enclosed spaces, like hallways, elevators, buses, or a room full of people. People may display open or closed postures, and they display directionality through posture. Postural behavior: Gives insight into the involvement of participants with others.

Body position Posture is the study of the ways in which individuals position their bodies. Dittman has described the open—closed position of arms and legs as hands and arms apart and knees separated for open, and arms crossed or folded and legs crossed for closed Dittman, However, there are other open and closed positions that an individual can take up. The interest for the interactional analyst lies in the relationship between the higher-level actions performed and the postures taken up by the participants.

Open and closed postures come in many varieties and degrees. A person may be standing in a fully open posture with limbs apart, or a person may express a completely closed posture with arms and legs crossed. Thus, the complete body of an individual has to be considered in order to analyze the degree of any open or closed positions. Similarly, directional positions of the body come in many varieties and degrees.

Imagine that your friend is facing the street and you are coming towards your friend from the right. As you approach and your friend recognizes that you are approaching, your friend will, most likely, move their body towards you. On the other hand, such a directional move may also be minute, so that your friend only turns the head and one shoulder to the right. Certainly, your friend can also take up any body position that lies in between these two.

Thus, the friend at the bus stop, who only moves the shoulder and head to the right, opens the posture at the very moment that they direct their body towards you as you are approaching. When a person is standing during an interaction, the position of the feet may give insight into the main focus of the participant. In interactional analysis, working with real-time, we discover that the individuals draw on the system of representation of posture that they have learned through socialization. Individuals construct the mode of posture through their own actions, and we can learn much about the interaction going on if we are aware of the general postural behavior.

When analyzing the mode of posture in interaction, we need to remember that postural behavior is also culturally habituated. Postural behavior may be different in different cultures and subcultures and it may be somewhat different for each individual. In order to analyze the mode of posture in interaction, a view from within the culture is very valuable, i. This view from within a culture can best be gained through ethnographic research methods linked with native interaction intuition, since we all have native interaction intuition about the meaning of behavior in our own cultures and subcultures.

But, one can also elicit native interaction intuition from the participants through playback methodology. Generally, we can surmise that the directional positioning of the body towards others indicates an engagement — however remote — in an interaction, while the positioning of the body away from others communicates a disengagement — again, however remote — in a given interaction.

However, when an individual turns the body away from others, displaying disengagement through posture, it does not necessarily mean that the individual is unaware of the ongoing interaction. While they may signal disengagement, they may strongly focus on the interaction, which would be visible through other communicative modes.

An example of this would be a parent watching several young children, sitting alone at a table eating dinner. While the parent may turn the body away from the children, they may pay special attention to all the ongoing sounds. Higher-level actions: Sandra constructs two higher-level actions that are central to the analysis of this interaction. First, she constructs the action of speaking on the phone, and, second, she constructs the action of having coffee with a friend. Both higher-level actions have a clear opening and closing sequence; and the higher-level action of speaking on the phone is inserted into the higher-level action of having coffee with a friend.

The following example is taken from a long-term ethnographic study that I conducted in Germany. Here, a woman, whom I call Sandra, and her friend were sitting at an outdoor table of a coffee shop, when her cell phone rang. Sandra took the phone out of her purse, took up a new posture, and then spoke for about 15 minutes. Her right leg crosses her left leg, turning her torso to the left and away from the other person at the table.

She holds the phone in her left hand, and her left shoulder is slightly bent backwards, and aligned with her torso. The second image in Plate 2. Here, Anna displays a closed posture with her forearms and shoulders close together and slightly bent forward. Yet, Anna has raised and slightly turned her head, gazing directly at Sandra, who is sitting across from her at the dinner table.

Assignment First, observe postural behavior by looking at one individual at a time. How is the person standing or sitting? How are the extremities positioned? Is the person displaying an open or a closed posture? You can do this exercise anywhere. Note down how the two individuals in this interaction are positioning their bodies. Metaphoric gestures: Possess a pictorial content, however they present the invisible: an abstract idea or category.

These abstract concepts are given form and shape in the imagery depicted in the motion and the space of the gesture. Deictic gestures: Point to objects or people in the physical world or to abstract concepts and ideas as if they had a physical location. Beat gestures: Look like beating musical time. Gesture is a global-synthetic mode, which means that gestures are constructed from whole to part. McNeill notes that the whole determines the meaning of the parts and one gesture can combine many meanings.

However, gestures produced together do not combine to form a larger, more complex one. Thus, a gesture often has elements, including trajectory, but the parts always depend upon the meaning of the whole. The parts do not have an independent meaning as opposed to the parts of a language which are meaningful, such as morphemes and words.

Hand and arm movements are often interdependent and concurrent with spoken language, slightly preceding the spoken discourse — to realize imagery. It is probably not often useful to separate these kinds of gestures from the language with which they co-occur. However, there are also gestures that do not coincide with spoken language, and others that co-occur but do not depict the same message that is conveyed in the occurring language.

A Multimodal Analysis of Advertisement of Online Marketplace Shopee | KnE Social Sciences

These are the gestures which have received much less attention. While they are composed just like gestures supplementing language, they accompany other communicative modes or depict meaning that differs from the language used. Before highlighting how gestures interact with other modes see pp. Iconic gestures usually consist of three movement phases: the preparation, the stroke, and the retraction phase.

Iconic gestures depict pictorial content and generally mimic what the individual communicates verbally. Metaphoric gestures usually consist of three movement phases: the preparation, the stroke, and the retraction phase. While metaphoric gestures also depict pictorial content, they portray abstract ideas or categories. Such abstract notions are given form and shape in the imagery portrayed in the motion and the space of the gesture. Deictic gestures also usually consist of three movement phases: the preparation, the stroke, and the retraction phase.

Such gestures often point to people or objects in the physical world, but they can also point to events in the past or the future, or point to ideas and notions as if they had a physical location in the world. They look as if the performer is beating musical time in quick succession. By viewing the gesture alone, it is usually not possible to come to a conclusion, because the two modes of gesture and language are here so closely linked that an analyst needs to refer to one mode to be able to understand the message in the other mode.

When looking at deictic gestures, however, we can often understand the message by viewing the gesture alone. When a person is pointing to 30 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Communicative modes someone, we often follow the stretched out arm with our gaze, even if we are not in earshot and cannot know what the person is saying. Therefore, while deictic gestures often occur with spoken language, these gestures can actually be understood without understanding the mode of language — at least to a certain extent.

Here, then, we may say that deictic gestures are not necessarily as dependent upon language as are iconic or metaphoric gestures. Deictic gestures may also be performed without language altogether, for this reason.

Beat gestures are again different. Some beats are highly dependent upon the mode of language, taking on a function of emphasizing a certain word or notion, and thereby coherently integrating the overall discourse, while other beats indicate a higher-level communicative coherence which will be discussed in detail in Chapter 6.

McNeill 83 , building on Kendon , describes gestures as occurring in a gesture phrase, which consists of one or more of the following movement phases: 1 2 3 Preparation, during which the limb moves away from the rest position to a position in gesture space where the stroke begins. Pre-stroke hold is the position that is reached at the end of the preparation. Stroke is the peak of the gesture, at which point the meaning of the gesture is expressed. Retraction is the return of the hand to its rest position. While the stroke is obligatory, all other phases are optional.

Nevertheless, most iconic, metaphoric, and deictic gestures consist of three movement phases, the preparation, stroke, and retraction. The accountant has let go of the piece of paper and is stretching his arm out towards the form lying on the desk. The second image of the second row shows the stroke of the gesture. The second image in this row displays the retraction phase of the gesture. Assignment Observe gestures that individuals around you perform. At the beginning of your observations you will realize that it is not always easy to distinguish which movement can be termed a gesture and which movement cannot.

Take your time and observe carefully.

Refine your editions:

Once you feel comfortable seeing gestures as units made up of parts, you will want to observe a dyadic interaction. Now, try to note down some gestures that are especially prevalent. Simple head movement: Complex head movement: Types of head movements: Lateral, sagittal, or rotational. Overlapping of two or more patterns. Directional shifts often one—two shifts or deictic head movement often three movement phases. Postural shifts, which are discussed on pp. Gaze shifts, which are discussed in pp. Communicative modes 33 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Positions of the head Head movement is the study of the ways that individuals position their heads.

Altorfer et al. While Altorfer et al. Individuals move their heads in many positions, and head movement in interaction has a range of functions from conventional to novel. Such movements would be simple sagittal or simple rotational movements. The head may be lowered or raised to close or open a posture, mainly through sagittal movement. Such movements are often complex, with many positions intertwined. Thus, head movement may be simple, meaning a clear lateral, sagittal, or rotational movement, or it may be a complex movement, in which two or all three movement patterns overlap.

The extent of head movement differs in any culture and subculture, and even differs from one individual to another. As with any mode in interaction, it is necessary to understand the movements of the people that one is studying, in order to make any claims of meaning. Simultaneously, it is important to keep in mind that the same head movement may have a different meaning in different situations, as interactional meaning is always dependent upon the individuals performing the movement and the individuals interpreting it.


  • Product details.
  • BreMM17 | Zhaofen | Bremen Conferences on Multimodality;
  • Combining Multimodal Analysis from International Perspectives?
  • Citas por año!
  • Ссылок за год?
  • Meaning is always co-constructed, and unintentional actions may be just as communicative as intentional ones. Conventional head movements are important in interaction, and they are also the easiest to study. These movements have a clear one-to-one verbal counterpart: yes or no, and are often performed instead of a verbal message. Meaning for such conventional head movements can be assigned by analyzing their position within a conversation, where they often take the place of the second part of an adjacency pair.

    Such movements do not have a verbal one-to-one counterpart, and it is not always easy to distinguish head movement from other communicative modes. The movement phases of the head beats or head tosses are similar to manual beat gestures, so that they only consist of two quick stages. The position and phases of head movements vary to a great extent, but one can determine whether a person is performing a head beat or a deictic head movement, because all head movements have clear boundaries. There are always pauses between head movements, even though they may be short. The same is true for deictic head movements, and, of course, also for conventional head movements.

    Thinking of conventional head movements, we can often determine the strength of the message by the number of times that a person shakes or nods the head. Usually, the more often an individual shakes or nods between actually resting the head in one position, the stronger the message. Strength of novel head movements also presents important information, as I will discuss in detail in Chapter 6. Only when analyzing head movement in interactional terms, is it possible to determine its meaning. The function that a head movement has for the individual performing it is no more important then the function that others in the interaction perceive, i.

    Interaction in a retail store The following interaction between owner and designer was videotaped in a gift basket retail and distribution store. Both women were working behind a counter on different tasks. The women were working side-by-side on their own tasks for the most part of about half an hour. At another time, the owner asked a question about an item that needed to be shipped. In Plate 2. The designer is mostly hidden behind gift baskets, while we have a clear upper body view of the owner.

    In the second image of this row, the owner has bent her head even lower and rotated it to the far left. Throughout the retraction phase, the designer moves her head, lifting it slightly to better see what the owner had been pointing at. Assignment In order to discern head movements, you should begin by observing one person at a time. First, you should observe a person who is sitting or standing alone. You will see that there are long rest periods between movements.

    Try to use these rest periods to learn to jot down the movements as exactly as possible. Next, look at one person in a dyadic interaction and again try to note down the head movements. Do not worry yet about the functions of these movements. Initially you need to learn to see head movements when they are being performed and then note them down quickly. Gaze may play a subordinate role in interaction when people are conversing and are not engaged in other activities. Communicative modes 37 2 3 4 5 6 7 8 9 1 2 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 Gaze may play a superordinate role when people are simultaneously engaged in other activities while conversing.

    Structure of gaze: Gaze can oscillate between sequentially structured and random, as it is integrated with the higher-level actions performed and the environment in which the interaction takes place. Generally, gaze is more structured when the interaction is also more structured. Organization and direction of looking The organization of gaze in interaction has received substantial attention. Among the authors who do not adopt the view that modes are easily distinguishable and discrete meaning-making units is Norris, who actually takes a firm stance against that view.

    She writes: One of the problems with using speech and writing as apparently transparent categories is that, historically, there have been many changes in how notions of speech and writing have been viewed, resulting in a complex picture. Writing is typically space-bound, contrived, visually decontextualised, factually communicative, elaborately structured, repeatedly revisable, and graphically rich.

    Crystal 28 This, for Goddard, is true only of particular genres of speech and writing. Blackwell Publishing Ltd. Although, as Goddard argues, language as both writing and speech is extremely difficult to define absolutely, this difficulty has not precluded the widespread deployment of the term as a concrete and easily defined concept, in both lay and professional contexts. But the application of linguistic concepts to non-linguistic and non-verbal semiotic modes, as Bateman et al.

    Theories tested against data in one field linguistics do not permit the wholesale transfer of the theory to other fields image, sound, etc. Bateman et al. They continue: The analytic procedures for establishing to what extent [the analytic principle of Given and New] could be a reliable property of layout rather than an occasionally plausible account are unclear. As such, a delicate balance between the adoption of and rejection of linguistics theories to visual analysis and intersemiotic processes must be maintained.

    In her words: Modes are unavoidably construed as distinct entities. A mode is a loose concept of a grouping of signs that have acquired meaning in our historical development. Jewitt makes her position clear up front: modes are means of representing, and media, means of disseminating. Yet, reading path is as determined equally by the mode of written language in this case, written English , not only by the medium of book.

    It might be unfair to take the writers to task for this, as the express purpose of those chapters is to further some other aspect of a systemic-functional theory of multimodal Blackwell Publishing Ltd. Media As outlined above, Norris in Analyzing Multimodal Interaction and Kress and van Leeuwen believe that a mode is an abstract resource for semiosis, not a physical or material one. As Jonathan Sterne argues in his book The Audible Past, most studies that engage in the process of mediation and its products are let down by a rather murky and tangled conception of medium, which for him is a recurring set of contingent social relations and social practices, and contingency is the key here.

    As the larger fields of economic and cultural relations around a technology or technique extend, repeat and mutate, they become recognisable to users as a med- ium. A medium is therefore the social basis that allows a set of technologies to stand out as a unified thing with clearly defined functions. Nicholas Cook , in Analysing Musical Multimedia, meditates in his last chapter on the question: What is a medium?

    He finds value in a definition Blackwell Publishing Ltd. Jewitt is not wrong to use these examples, but she has missed an entire other dimension of media and mediation and let her definition perpetuate the notion that media are entirely physical means. On this basis it would seem that the books are only comparable in a very general sense. Given the different function of each book, it is difficult to compare them in an evaluative sense, though the merits and shortfalls of each can be discussed.

    That said, I would have liked to have seen more of the chapters cover the first perspective on discourse and technology: the impact of technology on meaning making. If we are to understand and theorise the relationship of technology and semiosis, a broader conception of technology is desirable here, as is the scope for the inclusion of media production as well as reception technologies. Understandably, a small percentage of the work included in the book attends to the impact of technology on semiosis directly though, as mentioned above, many papers do focus on the place of technology in multimodal discourse analytic research.

    Furthermore, and more generally, the theme of the relation- ship of semiosis and technology should also encompass the notion of constraint; that is, how semiosis and the semiotic resources within a particular context of practice are shaped by and shape the concerns, methods and resources of the producer s in a specific setting. While some papers included in Discourse and Technology and indeed in the other works under review do attend to this notion, most do not; and while the idea of constraint may not be at all relevant to many papers, to others it would contribute a welcome in my opinion necessary dimension of analysis.

    Meaning making, as Fairclough et al. Fairclough et al. Many of the analyses and frameworks are extremely technical, with little to help the SFL novice along the way. Kress and van Leeuwen would be a typical example in their work. The level of technicality referred to above, though difficult, is also an asset to Multimodal Discourse Analysis: theories and frameworks are outlined in great depth and with a high level of skill. Without its technicality, perhaps, there would be little to commend this strand of research into multimodality, as connections with theories outside the SFL sphere are few.

    Although this lack is also true of the other books under review, it is Multimodal DiscourseAnalysis in particular that aims to represent new sites of semiotic analysis. Presently, in the area of the semiotics of sound design and music in multimodal contexts we have only van Leeuwen , Iedema , and my own work on sound in TV and film Constantinou , forthcoming. It would be unfair to evaluate this book solely on this basis, given its pedagogical and practical function, yet it remains a bugbear of mine that reflexivity in this Blackwell Publishing Ltd.

    In this sense, it could be argued that Analyzing Multimodal Interaction should also aim to be used to teach researchers, via demonstration, how to construct their own frameworks, not only how to use the present one. Rather, she aligns her own project with the strongly political aspects of multimodality cf. Lemke ; Kress and Van Leeuwen : that language, in both written and spoken forms, need not be considered central in any explanation of human interaction, and that the plethora of other communicative modes are always mobilised in every communicative and material moment.

    Essentially, this is a question of materiality: different communicative events have different materialities associated with them, some enduring, like print media, others fleeting, such as live music. What is at issue in her consideration of consciousness absent from much of the previous work on multimodality is the intentionality behind the communica- tive events of humans. In the more durable materialities such as those involved in the production of prevalent media forms e. Only by appeal to the human mind can this be achieved. Finally, the book is extremely well-organised and the argument, for the most part, clear and concise.

    Its application in the classroom would be simple to implement, especially as a complement to other, more contemplative and reflexive works on multimodality, such as Multimodal Discourse Kress and Van Blackwell Publishing Ltd. It deploys clear illustrations and exercises that are well thought out and relevant. The title of that book, however, suggests far more than even a surplus of theoretical connections.

    Lemke and various others. But it narrows the scope of multimodality, and invites the question: Are the concerns of multimodality necessarily oriented towards discourse, or to the interests of discourse analysis? In fact, the original title of Kress and van Leeuwen was Multimodality. Consequently, the book found its home among other discourse-oriented works, and its title suggests it is aimed at the DA community. Yet this is only a mild criticism of the way the academic publishing industry works. Of course, where there are funds to recoup, it can be expected that there are professionals whose task it is to reify or perhaps create academic markets.

    My intention here is not to reprimand those who facilitate the non-electronic distribution of original and interesting theory. Academic marketing is a necessary evil and the ins and outs of the marketing machine are not the issue here. Which leads me to the future of the field of MDA. Those who do pursue this line of inquiry will most likely derive their fundamental methodologies from Kress and van Leeuwen , There are fewer researchers, however, concerned to widen their analytic and methodological lenses to include analysis of the extra-semiotic, constraining features of a production Blackwell Publishing Ltd.

    This strand is extremely important to pursue, and is recognised as such not only outside of the MDA field e. Cottle ; Fairclough et al. Specifically, those interested in this pursuit should consider how the dynamics of various media production and interactional processes, and the practical and technological constraints of the media involved, play a part in establishing the parameters for meaning making.

    The other concerns the representation of truth through multiple semiotic means see Kress and van Leeuwen for discussion. Van Leeuwen, personal communication. Multimodality and empiricism: Methodological issues in the study of multimodal meaning-making. A model of genre in document layout.

    Customers who bought this item also bought

    Sound constructs. Constantinou, Odysseas. Audiovisual organisation: Synchronicity and continuity in broadcast news. Cook, Nicholas. Analysing Musical Multimedia. NewYork: Oxford University Press. London: Sage. Crystal, David. Cambridge: Cambridge University Press. Fairclough, Norman, R. Jessop and A. Critical realism and semiosis. Iedema, Rick. Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic practice.

    Kahn, Douglas. Cage and phonography. London: Routledge. Kress, Gunther and Theo van Leeuwen. Front pages: The critical analysis of newspaper layout.