Informed Exploration in Reinforcement Learning via Intuitive Physics Model Reasoning

• Applicant: Professor Jan Reinhard Peters, Ph.D.
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 516414603

In the near future robots will perform a variety of tasks such as directing tourists or assisting the elderly. The scientific challenges of these applications is to cope with the large diversity of environments and scenarios that our physical world presents. Given this formidable diversity, programming a robot to cover in foresight all possible turns of events is doomed to failure if the robot is not able to adapt and self-improve. To foster robots’ autonomy, task-specific expertise should be accompanied by general purpose learning algorithms that ensure generalization and autonomy. These algorithms should explore and interact with the physical world in a meaningful way to constantly adapt to their changing (sub-)goals. Reinforcement learning (RL) is one such learning framework for an agent to adapt through interaction with its environment. In RL, an agent explores its environment by applying an action, gathers information by observing a state and is incentivized to adapt by collecting rewards. In recent years, RL has become an extremely powerful tool for decision-making as exhibited by its successes at surpassing human level at playing Atari video-games or defeat grand masters in the board game of Go. These agents were all trained on specific simulators designed for each task, and in each case the given simulator represented the entire reality of the agent. Moreover, these agents have been shown to break down when small changes are made in the environment that are considered insignificant to humans. Developing an RL framework that is able to quickly adapt to changes in the environment largely remains an open question, and is the core focus of this project. To improve adaptability of RL to changes in the environment, we propose to study its integration with intuitive physics models (IPMs). IPMs, or what is also referred to as common sense physics, have been a long-standing topic in AI and machine learning but their integration to RL has not been fully realized yet. In the new framework we propose in this project, the goal will be to autonomously discover and learn salient characteristics of the environment, constituting the IPMs, to explore in an informed way by means of reasoning and planning using the IPMs. The main departure from traditional model-based RL is that IPMs will not seek to equally capture all the information of the environment, but rather to learn more abstract and schematic models that will adapt better to changes in the environment. For example, an IPM will predict if a falling glass will break, but will not seek to model the exact position of glass shards after the glass broke. Identifying these key events in the environment, learning models thereof from data, and using these models in RL will be our main research directions. Developing RL algorithms that can generalize to all such task variations will be a major contribution to both the machine learning and robotics communities.

• DFG Programme: Research Grants