Brain Dynamics in Cyber-Physical Systems as Measure of User Presence

• Applicant: Professor Dr. Klaus Gramann
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 462163815

Cyber-physical systems (CPS) allow for controlling or monitoring physical processes utilizing computer-based algorithms. CPS are transforming work places as well as daily interactions with technical systems. Virtual environments (VEs) are one example of advanced CPS that situate users in diverse environments including human-machine interaction settings that require dynamic and precise interaction with objects or other agents. Precise interactions in CPS, like in the real world, depend on the users’ presence as well as their ability to deal with uncertainties of dynamic changes in the environment. Only when users behave in VEs as they would do in the real world, they experience presence, the subjective feeling of being in the virtual place even though they are physically located in another place. The users’ ability to precisely interact with a technical system requires near-natural sensory feedback to reduce uncertainty regarding the action outcome in complex and dynamic environments by means of predictive processing. Until now, presence in VE applications was assessed using self-report questionnaires. This method interrupts the feeling of presence and is prone to problems associated with introspection. Alternatively, peripheral physiological measures are used that require emotional stimulation to trigger a measurable response while providing only limited temporal resolution (in the range of seconds) about the ongoing cognitive and affective processes. After decades of research, no standardized and objective presence metric has been identified to assess user presence without interrupting the ongoing VE experience. We aim to overcome existing limitations of presence metrics by investigating brain dynamics in combination with peripheral physiology accompanying multisensory integration during naturalistic interaction with a virtual system. We propose to develop a new method to unobtrusively and continuously assess the user’s experience of presence using a Mobile Brain/Body Imaging (MoBI) approach. We will use electroencephalography (EEG) synchronized with electrocardiography (ECG) and measures of electrodermal activity (EDA) plus motion capture and virtual reality technology while using data-driven analyses approaches to develop a multimodal presence metric. We will investigate event-related physiological activity related to deviations of predicted and real interaction outcomes and how these correspond to established questionnaire metrics of presence. We will further investigate how noise in different sensory systems influences presence experience and how multisensory integration of visual and vibrotactile feedback during a precision task in VE is realized on a neuronal level. The results of this project aim at establishing a new presence assessment that will allow us to continuously assess user presence in CPS without disrupting the interaction itself while quantifying the importance of different sensory modalities for experiencing presence.

• DFG Programme: Research Grants
• International Connection: USA
• Cooperation Partner: Professor Dr. Pedro Lopes