Erkennung von Mehrfingergesten mit einem Fingerring: Ubiquitäre Interaktion für das Internet of Things
Final Report Abstract
Finger rings are a widely used and accepted article of jewelry in our culture, carrying symbolic and mythical meaning. Due to their small size and form factor, they can be worn all day without being obtrusive or drawing much attention. Equipping a ring with sensors can realize a ubiquitous and unobtrusive gesture interaction device. Current approaches to ring-based interaction lack a broad interaction space, which limits the variety of applications. This is reasoned by the fact that those rings can only sense motion or bend angle of a single finger. Possible solutions to this problem are the use of multiple rings or rings spanning multiple fingers. Both reduce the obtrusiveness and comfort and thus the suitability for daily use. In this project, we developed PeriSense, a single ring enabling a broad range of multi-finger interactions. The basic concept of PeriSense is the application of four electrodes measuring the capacitance between the electrodes and the fingers and the hand. This allows detecting the motion of the finger wearing the ring and at least the adjacent fingers. Additionally, it is equipped with a motion sensor allowing tracking the hand position and orientation. In comprehensive evaluations, we studied the suitability of capacitive proximity sensing as sensing technology for a ring enabling multi-finger gesture interaction. We showed its resistance to environmental noise and determined the effective resolution at different distances. Based on different interaction techniques such as finger gestures and postures, pinch gestures, and unistrokes drawn above the ring and with the thumb drawn on the palm, we studied how well these gestures can be detected and distinguished and where the limits currently are. In user-dependent tests, convincing results for all interactions except pinch gestures were achieved. However, the detection rate in the user-independent test dropped, in some cases. Besides the study of gesture interactions, we developed a long short-term memory (LSTM) algorithm to map the capacitive measurements to ten finger angles (middle joint and palm close joint). The remaining five angles (the fingertip close joints) are approximated by the predicted joints' angle ratio. We revealed a mean absolute error of 13.02 degrees over all finger angles in a leave-one-user-out cross-validation test. The motion of the little finger and the thumb are out of the capacitive measurement range or covered by the index and ring finger, respectively. With natural finger movements, the LSTM even estimates the little finger and thumb's angles based on the movement pattern of the other fingers. Finally, we developed a length normalized complexity estimate measure with a linear memory and time complexity to filter non-gesture signals. Filtered signal parts are not sent. This increases the battery runtime because the Bluetooth-based data transition consumes about 90 % of the energy. 13 % of the data could be blocked without influencing the gesture recognition accuracy. In this manner, the battery runtime can be extended by about one hour in case of our latest prototype. In general, more electrodes could increase the sensitivity and enhance the recognition rate significantly, which, however, has still to be studied. Smaller electrodes additionally enable smaller ring forms. This would foster wearing comfort and flexibility of the wearing finger as the current prototype limits the finger bend flexibility depending on the finger length. A trade-off between electrode size and its sensor range has to be taken though. The evaluation results reveal the promising potential of capacitive proximity sensing to be a sensing technology enabling diverse and versatile multi-finger gesture interactions with a single ring. Due to the many interaction possibilities, the ring can be used in many different ways. It can be applied as an input device for AR and VR glasses. Since no extra controller has to be held in hand and the ring can be used ubiquitously, the immersion is increased. Furthermore, the ring can also serve as an extended control device for smartphones, e.g., control the music player, take photos, reject calls, or write short texts without having to pick up and unlock the phone. The ring can also be used as a control device for connected environments, e.g., to switch on the light or to lower the blinds.
Publications
- A Gesture Control System for Drones Used with Special Operations Forces. In: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI’20 Companion), March 23-26, 2020, Cambridge, United Kingdom. ACM, New York, NY, USA, 1 page
M. Montebaur, M. Wilhelm, A. Hessler, S. Albayrak
(See online at https://doi.org/10.1145/3371382.3378206) - Demonstration of Finger Tracking Using Capacitive Sensing with a Ring. In: 25th International Conference on Intelligent User Interfaces Companion (IUI ’20 Companion), March 17-20, 2020, Cagliari, Italy. ACM, New York, NY, USA, 2 pages
M. Wilhelm, J.-P. Lechler, D. Krakowczyk, S. Albayrak
(See online at https://doi.org/10.1145/3379336.3381475) - Ring-based Finger Tracking Using Capacitive Sensors and Long Short-Term Memory. In: 25th International Conference on Intelligent User Interfaces (IUI ’20), March 17-20, 2020, Cagliari, Italy. ACM, New York, NY, USA, 5 pages
M. Wilhelm, J.-P. Lechler, D. Krakowczyk, S. Albayrak
(See online at https://doi.org/10.1145/3377325.3377535) - “PeriSense: Ring-Based Multi-Finger Gesture Interaction Utilizing Capacitive Proximity Sensing,” Sensors, vol. 20, no. 14, p. 3990, Jan. 2020
M. Wilhelm, D. Krakowczyk, and S. Albayrak
(See online at https://doi.org/10.3390/s20143990)
