Multimodality and embodied interaction

• Applicant: Dr. Cornelia Gerhardt
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
• Term: from 2012 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 221933637

• Final Report Year: 2020

Final Report Abstract

The research network “Multimodality and Embodied Interaction” was concerned with communicative practices in social, face-to-face interaction from a multimodal perspective. Taking the perspectives of Conversation Analysis, Interactional Linguistics and / or Multimodal Analysis, the projects assembled in the network aimed to analyse how interactants use vocal and verbal resources situated in a larger semiotic context, i.e. in interdependence with visual signals, such as e.g. gaze, gesture, or body posture. Developing a methodology which takes into account this semiotically rich construction of social interaction, the network combined interdisciplinary as well as cross-linguistic and -cultural aspects in its research, drawing on data from various social settings and constituted through different technologies, i.e., video cameras or eye-trackers. The results of the network, its meetings, and the member’s individual projects were disseminated in one edited volume, a monograph as well as 20 individually published papers by network members which reflect the following main findings: Taking a bottom-up perspective and drawing on a wealth of different data sets (e.g. care-taker interaction; parliamentary debates; or handball timeouts), the analysis revealed that the use of embodied resources is guided by the displayed understanding of the social activity under way, oriented to by and made visible through the participants’ use of phonetic-prosodic, lexicosyntactic, and visuomaterial resources in interaction. Following on from this, the key finding of the joint research conducted in the network, expressed in the edited volume, is that it is the overall activity or course of action (and not necessarily an embodied resource, practice or turn) which forms the basic unit of embodied interaction. An analysis of how embodied coordination is accomplished in the production of turns and in the interaction between participants must put a focus on the situated use of the resources involved in the construction of activities. The analyses revealed that these resources often encompass practices of visibility, i.e., practices that shape and manipulate the physical context to make it meaningful for the co-interlocutors and the activity at hand. The potential resources at the disposal of the participants are not assembled in arbitrary ways, but are finely attuned to the specific constraints and goals intertwining the language and body into meaningful multimodal gestalts both in the individual formation of actions and in interaction with participants. While activities may have different levels of organisation, i.e., more language based activities and activities engaged in through nonverbal, practical actions, there is no general primacy of one resource. As to the organisation of visuo-material resources, they can be deployed to implement non-verbal actions organised in sequences, e.g. in courses of actions which are achieved through physical actions only as well as be simultaneously organised with ongoing speech, e.g., for stance-taking or turn-taking tasks. Although in its original design, the network was intended to address questions of basic research in multimodality and embodied interaction, its findings are implicative of prospective applications, specifically with regard to the design and programming of communicative artifacts, i.e., e.g., virtual avatars or (social) robots. We can conclude from our research within the network that it is still too early to draw a schematic taxonomy of the resources used in embodied activities. Specifically, some modalities, especially relating to the face, are completely understudied. While several papers point out the relevance of facial expressions, empirical studies of their use in interaction seem practically non-existent.

Publications

• 2017. Discourse and organisation. In: Wolfram Bublitz & Christian Hoffmann (eds). Handbook of Pragmatics 11: Pragmatics of Social Media, 245-274. Berlin: de Gruyter Mouton
  Frobenius, Maximiliane & Cornelia Gerhardt
  
• 2019. Embodied Activities in Face-to-face and Mediated Settings - Social encounters in Time and Space, London: Palgrave MacMillan, 459 + xv pages
  Reber, Elisabeth & Cornelia Gerhardt (eds)
  
• 2019. Embodied activities. In: Cornelia Gerhardt & Elisabeth Reber (eds). Embodied Activities in Face-to-face and Mediated Settings: Social Encounters in Time and Space, 3-27. London: Palgrave MacMillan
  Gerhardt, Cornelia & Elisabeth Reber
  
• 2019. Epilogue. In: Cornelia Gerhardt & Elisabeth Reber (eds). Embodied Activities in Face-to-face and Mediated Settings: Social Encounters in Time and Space, 437-452. London: Palgrave MacMillan
  Gerhardt, Cornelia & Elisabeth Reber
  
• 2019. Punch and Judy politics? Embodying challenging courses of actions in parliament. In: Elisabeth Reber & Cornelia Gerhardt (eds). Embodied Activities in Face-to-face and Mediated Settings: Social encounters in time and space, 255-297. London: Palgrave Macmillan
  Reber, Elisabeth
  
• 2019. Showing as a means of engaging a reluctant participant into a joint activity. In: Cornelia Gerhardt & Elisabeth Reber (eds). Embodied Activities in Face-toface and Mediated Settings: Social Encounters in Time and Space, 137-175. London: Palgrave MacMillan
  Gerhardt, Cornelia
  
• Quoting in Parliamentary Question Time. Exploring recent change (Studies in English Language). Cambridge: Cambridge University Press, 2021. XX, 344 S.
  Reber, Elisabeth