WASEDO: Wearable-federated, weakly-supervised Activity Sensing through Egocentric Detection of Objects

• Applicants: Professor Dr. Kristof van Laerhoven; Professor Dr. Michael Möller
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 506589320

Body-worn sensor systems bare a great potential in analyzing our daily activities with minimal intrusion yielding various applications, ranging from the provision of medical support to supporting complex work processes. With (deep) neural networks representing the state-of-the-art technology for the automatic analysis of such data, a key bottleneck becomes the annotation of data for the underlying training, specifically because the acquired non-visual data is difficult to interpret in hindsight, and visual data represents a significant intrusion in the privacy of any participant in such a study. Therefore, the goal of this proposal is to study a federated learning approach, in which clients use both wrist-worn sensors and camera glasses, where the latter deliver visual data that is merely used to locally supervise the training of a network analyzing the wrist-worn sensor signals. By following this weakly-supervised federated learning approach, we are able to avoid both the necessity for manual annotations, as well as the submission of visual data to a central server. Our goal is to conduct fundamental research on the weak supervision as well as the collaborative training to combine our results in a practical solution for wrist-worn sensor data analysis.

• DFG Programme: Research Grants