Project Details
Projekt
Quantification of Gesture Form Analysis (quantitative GFA) Application of GFA to gestures within sign language and development of an interdisciplinary coding system
Applicant
Dr. Julius Hassemer
Subject Area
General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
Term
from 2015 to 2017
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 281272438
Gesture studies, a young but dynamic interdisciplinary area of research of interest to psychology, anthropology, linguistics and informatics, has not yet settled on a common typology of gestures, nor even on the principles on which such a typology might be constructed. It is agreed that co-verbal gestures convey meaning through the physical form and movement of the hand, often by means of iconically representing aspects of objects or imitating practical actions (e.g., Calbris 2003; Cienki 2005; Efron 1941; Kendon 2004; Kita et al. 2007; Mittelberg 2010; Müller 2008; Streeck 2009). But existing categorisations fundamentally disagree on the number of types and on their relations to one another. The number of gesture type categories ranges from 4 (Müller 1998) to 12 (Streeck 2008) to 84 (Sowa 2006), often making cross-studies comparison difficult.The theory of Gesture Form Analysis (GFA), proposes a systematic, form-based categorisation framework, combining insights from the above-mentioned scholars. Basic assumptions of GFA have already been empirically supported in a motion-capture study (Hassemer, dissertation).Still lacking, however, are (1) application of GFA to the gestural elements of sign language and (2) standardisation of a practical coding procedure that allows quantifying gesture data by type across different corpora. These are the two primary foci of this proposal. A third, exploratory goal is to begin testing the viability of GFA as a model for automatic gesture recognition.In support of these goals, we propose to continue the study at the University of São Paulo, with Prof. Leland McCleary, a linguist who is familiar with GFA and who, with his colleague Evani Viotti, has been studying gestural elements of Brazilian Sign Language as well as methods of sign transcription; and whose colleague Felipe Barbosa does research on phonetic variation in sign language, using VICON motion-capture technology. The proposed situation will also permit the initial investigation of GFA applications to machine-learning methods together with a specialist in that area, Sarajane Peres. All of these researchers at the University of São Paulo have agreed to collaborate.Specifically in support of the second goal of implementation of a practical coding method, we bring to the project prior experience of having successfully collaborated (with Han Sloetjes, MPI Nijmegen) on the creation of a gesture annotation plugin for the ELAN transcription software. In support of the second and third goal, we also bring a motion-capture corpus, collected during the dissertation, containing multiple-camera (high-speed) video recordings and three-dimensional motion-capture data on verbal-gestural descriptions that can be used as a show-case corpus for quantitative GFA and for the exploratory machine-learning investigation.
DFG Programme
Research Fellowships
International Connection
Brazil
