The Architecture of Attentional Processes in Active Vision
Final Report Abstract
Our gaze visits new places several times per second, actively providing high-acuity vision of a subset of the data available in our visual environment. These swift movements of our eyes, heads, and bodies (called saccades) cause massive motion across the entire retina. Moreover, objects that have fixed places in the world incessantly slip across the sensory surface, covering new locations on the retina with each fixation. Given that their sensory consequences are so drastic, movements of gaze are perceptually strikingly unconspicuous. Why is vision continuous rather than erratic? The premise of this project was that perception – as we experience it – is largely determined by attentional processes. Understanding these processes in active observers, therefore, would elucidate the question of perceptual continuity. We used a combination of eye and motion-tracking, visual psychophysics, and computational modeling to investigate the role of attention in visual perception before rapid gaze shifts, and to reveal its significance for trans-saccadic continuity. In particular, we aimed for a better empirical and mechanistic understanding of the dynamics of spatial attention in the presence of saccadic eye movements and large eye-head gaze shifts, and of the role of feature-based selection before and across saccades. Over its funding period, this Emmy Noether project contributed a large number of empirical studies and a computational account of the relation between attention to locations and features. In particular, we have obtained the following results: 1. We have developed experimental setups that combine simultaneous eye- and head-motion tracking for gaze-contingent display control. Using one of these setups, we implemented the first experiments that studied localization performance across large-scale eye-head gaze shifts in complete darkness. We provided evidence that head movements are taken into account in the updating of spatial representations across gaze shifts and that the mechanisms underlying this updating rely on extra-retinal signals . 2. Spatial and feature-based attention interact in their effects on visual sensitivity only in the presence of several stimuli competing for resources. We were able to account for both additivity and interaction with a computational model that assumed complete independence between the mechanisms implementing these two types of attention. 3. Saccades inadvertently prioritize features at their targets for visual processing and support the transition from fragile sensory memory to robust visual short-term memory storage. Inadvertent processing of feature information at the saccade target informs future attentional selection and immediate post-saccadic oculomotor behavior. 4. Whereas saccade preparation does not entail an automatic deployment of attention to features at locations remote from the saccade target, pre-saccadic attention does spread to locations that are perceptually grouped with the target itself. 5. Both trans-saccadic feature-based attention and pre-saccadic updating of attended locations (in retinotopic coordinates) underly the continuous deployment of attention to objects of interest in the world as the eyes move about. Together, these results converge towards the idea that attentional processes play a key role in visual processing when observers actively engage with their environment. Non-spatial and spatial forms of attention combine to give rise to our experience of perceptual continuity across saccadic eye movements.
Publications
- (2013). Allocation of attention across saccades. Journal of Neurophysiology, 109, 1425–1434
Jonikaitis, D., Szinte, M., Rolfs, M., & Cavanagh, P.
(See online at https://doi.org/10.1152/jn.00656.2012) - (2015). Attention in active vision: A perspective on perceptual continuity across saccades. (Yarbus-100 Special Issue) Perception, 44, 900–919
Rolfs, M.
(See online at https://doi.org/10.1177/0301006615594965) - (2015). Attentional tradeoffs maintain the tracking of moving objects across saccades. Journal of Neurophysiology, 113, 2220–2231
Szinte, M., Carrasco, M., Cavanagh, P., & Rolfs, M.
(See online at https://doi.org/10.1152/jn.00966.2014) - (2015). Stimulus competition mediates the joint effects of spatial and feature-based attention. Journal of Vision, 15(14):7, 1–21
White, A.L., Rolfs, M., & Carrasco, M.
(See online at https://doi.org/10.1167/15.14.7) - (2016). Oculomotor inhibition covaries with conscious detection. Journal of Neurophysiology, 116, 1507–1521
White, A.L. & Rolfs, M.
(See online at https://doi.org/10.1152/jn.00268.2016) - (2016). Presaccadic motion integration between current and future retinal locations of attended objects. Journal of Neurophysiology, 116, 1592-1602
Szinte, M., Jonikaitis, D., Rolfs, M., Cavanagh, P., & Deubel, H.
(See online at https://doi.org/10.1152/jn.00171.2016) - (2016). Remapping attention pointers: Linking physiology and behavior. Trends in Cognitive Sciences, 20, 399–401
Rolfs, M. & Szinte, M.
(See online at https://doi.org/10.1016/j.tics.2016.04.003) - (2017). Saccadic eye movements do not disrupt the deployment of feature-based attention. Journal of Vision, 17(8):4, 1–15
Kalogeropoulou, Z., & Rolfs, M.
(See online at https://doi.org/10.1167/17.8.4) - (2017). Selective enhancement of orientation tuning before saccades. Journal of Vision, 17(13):2, 1–11
Ohl, S., Kuper, C., & Rolfs, M.
(See online at https://doi.org/10.1167/17.13.2) - (2017). Setting and changing feature priorities in visual short-term memory. Psychonomic Bulletin & Review, 24, 453–458
Kalogeropoulou, Z., Jagadeesh, A.V., Ohl, S., & Rolfs, M.
(See online at https://doi.org/10.3758/s13423-016-1094-6) - (2018). All is not lost: Post-saccadic contributions to the perceptual omission of intra-saccadic streaks. Consciousness and Cognition, 64, 19-31
Balsdon, T., Schweitzer, R., Watson, T. L., & Rolfs, M.
(See online at https://doi.org/10.1016/j.concog.2018.05.004) - (2019). Pre-saccadic motion integration drives a predictive postsaccadic following response. Journal of Vision, 19(11):12,1-19
Kwon, S., Rolfs, M., & Mitchel, J. F.
(See online at https://doi.org/10.1167/19.11.12)
