Prehensile Interaction: User Interaction Concepts based on Prehensile Hand Behavior

• Applicants: Professor Dr.-Ing. Lars Linsen; Dr. Dimitar Valkov
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2020 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 436291335

While interactive 3D environments have gained increasing popularity in the last years, 3D user interfaces still remain rather complex and in many cases require special skills and training. Interaction with hand gestures has been an active research area in the 3D user-interfaces field, but research from the related fields of psychology has rarely been taken into consideration when designing the hand-based interfaces. Investigations on reach-to-grasp actions in various domains of psychology have consistently shown that the natural kinematics of prehension allows for predicting the object a human is going to grab and sometimes even the subsequent actions taken on that object. Many studies investigated different factors affecting the prehension, e.g., the object's size or shape, its specific affordances or intended use, as well as the effect of obstacles and distractors. Some studies also evaluated differences in prehensile behavior when using physical, memorized, or virtual objects. Furthermore, it was shown that the computational complexity of grasp kinematics can be greatly reduced, since the fingers' dynamics are highly correlated in natural prehension. These insights promise great opportunities for substantially improved interaction in 3D environments, provided that the hand kinematics information is extracted and evaluated on the fly and instantaneously incorporated in the interface.In this project, we will establish a general framework for the analysis of prehensile behavior in the context of human-computer interaction and we will explore the applicability of prehensile information to design natural user interfaces for the interaction with computer-generated virtual environments. We plan to perform a series of experiments, in which the users' hand kinematics will be evaluated in different application scenarios, for different tasks, and with different methods.The aim is to gather sufficient data, develop effective interactive visual methods to analyze them, and examine the benefits and limitations of early prediction of intended objects and tasks based on hand kinematics. The expected insight will allow us to explore the design space for future interactive systems based on natural hand interactions, i.e., without requiring special gestures or skills.

• DFG Programme: Research Grants