Localizing individual steps of stimulus-response transformations in the human brain with highly parameterized models

• Applicant: Professor Dr. Hannes Ruge, since 12/2023
• Subject Area: General, Cognitive and Mathematical Psychology ; Human Cognitive and Systems Neuroscience ; Cognitive, Systems and Behavioural Neurobiology
• Term: since 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 445383113

In order to survive in complex and dynamic environments, animals are required to transform high-dimensional sensory information rapidly and accurately into appropriate motor responses. Humans show a particularly high degree of flexibility in transforming complex sensory information into motor actions, as expressed for example by their ability to operate a broad range of different machines and computer programs. How does the human brain transform high-dimensional sensory information into motor responses rapidly and reliably? According to the two-stream hypothesis, the ventral stream is associated with detailed visual representations and specifically with object recognition and categorization, whereas the dorsal stream is involved in transforming visual information into motor actions. While the ventral stream and early visual cortex have been extensively investigated with functional imaging in combination with advanced modeling approaches like multivariate pattern analysis and encoding models, stimulus-response processing along the dorsal stream has not yet been characterized with such modeling approaches due to methodological limitations. Recent progress in machine learning now makes it possible to analyze high-dimensional stimulus-response transformations via deep neural networks (DNNs). Here, I propose to conduct two studies on stimulus-response processing in the human brain. In both studies, the participants will be asked to train several computer games before being scanned and then to perform these computer games inside an MRI scanner. The resulting fMRI datasets will be analyzed via DNNs to provide a fine-grained characterization of individual stimulus-response processing steps in the human brain. Specifically, the activation output of the DNNs, i.e. the activations of the artificial neurons, will be used to model fMRI activation patterns along the dorsal stream. To this end, the encoding model framework, which has been employed before to characterize visual processing in early visual cortex via DNNs, will be extended to allow investigating not only visual representations but also the transformation of visual representations into motor responses. While the proposed project primarily aims at providing a more detailed functional characterization of stimulus-response processing along the dorsal stream, it also promotes the transfer of the most recent methodological progress from machine learning to cognitive neuroscience. Both the collected data and the developed software tools will be made publicly available. Given the ongoing rapid progress currently being made in machine learning, other experimental tasks might be investigated as well via encoding models and DNNs in future projects building on the modeling framework developed here.

• DFG Programme: Research Grants
• International Connection: Netherlands
• Co-Investigator: Professor Dr. Radoslaw Martin Cichy
• Cooperation Partner: Professor Dr. Marcel van Gerven
• Ehemaliger Antragsteller: Dr. Holger Frimmel, until 11/2023