Multmodality

Semiosis, or meaning making, is a complex but important process, which defines human experience. The increase in research interest in the field of communications, media studies, and semiotics has brought a renewed attention to the multifarious nature of meaning in a text. Baldry (2000), Kress (2003) and Kress and van Leeuwen (2001) draw attention to the multimodal nature of contemporary society where meaning is often made through a co-deployment of a combination of semiotic resources. In other words, images, gestures, and sounds often accompany language in semiosis. This recognition of semiotic resources operating independently and interdependently on different levels in a text has ushered in a necessary transformation in the research landscape pertaining to the study of semiosis. The process of meaning making in text is both multifaceted and multimodal. As such, understanding the meaning made in a specific text cannot equate merely to an analysis of the semiotic resource of language, but has to take into account the complex nature of semiosis through both the deployment and co-deployment of various semiotic resources. What this recognition implies is that in order to understand meaning making, it is no longer sufficient to deconstruct and analyze the use of language in a text only. Instead, the operation of other semiotic resources must also be factored in as contributing to the total meaning made in the text.

Based on an analysis accounting for the whole body as a possible articulator in the depiction of actions, this chapter argues for an expansion of the notion of ‘character viewpoint gestures’ to a notion of ‘multimodal action depiction from a character viewpoint’. Our study shows that speakers may deploy only single articulators, providing a semantically reduced depiction of the action, or they may deploy more bodily articulators and give a semantically rich picture of the event narrated. Our findings suggest a continuum of semiotic complexity, capturing the range of bodily involvement from less pantomimic (single articulators involved) to pantomimic depictions (more articulators involved) of actions. The paper closes by discussing our observations with respect to the notion of ‘constructed action’ and ‘role shift’ in sign languages and by giving some general remarks on the multimodal analysis of narrations.

Le présent article se propose d'adopter une approche micro-analytique basée sur des enregistrements vidéo pour étudier les différents cadres de participation émergeant en classe et de dégager la complexité de leur dynamique multimodale, en faisant émerger la manière dont des discours appartenant à différentes sphères de légitimation émergent, s’influencent et se croisent (cf. Mondada, 2003). Il prend plus spécifiquement
pour objet central les moments d'émergence et de transition entre cadres de participation pour montrer, au
moyen d'une analyse prenant en compte aussi bien les éléments proprement linguistiques que les gestes, regards et orientations corporelles, comment ces moments sont négociés
et construits localement par les participants de manière collaborative. Ces analyses permettront à leur tour de réfléchir au potentiel de ces
moments de transition tant pour la co-construction des contenus disciplinaires que pour la mise en œuvre et la pratique de ressources complexes sur le plan interactionnel.

As práticas de linguagem em ambientes midiáticos se realizam, crescentemente, sob o impacto da transmídia (JENKINS, 2008), ou trânsito de conteúdos entre mídias. Tal migração acontece incessantemente entre ambientes como a televisão e a internet. Assim, nos questionamos como ocorre a construção de sentidos em gêneros audiovisuais que sofrem mudança de suporte. Trata-se, aqui, de traçar um paralelo entre recursos de língua/linguagem voltados para o discurso multimodal. Apropriamo-nos de reflexões como a de Kress (2010) para identificar o potencial significativo das semioses num corpus composto por três vídeos do site YouTube. A análise se baseia na categoria de metafunção (KRESS; van LEEUWEN, 2006). Observamos que a mudança de suporte ocasiona um reordenamento dessas macrofunções.