Stochastic Optimal Control based on Gaussian Processes Regression

• Applicant: Professor Dr.-Ing. Uwe D. Hanebeck
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: from 2017 to 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 349395379

In stochastic control, optimal decision making in continuous domains under statistically modeled uncertainty is usually addressed via Dynamic Programming (DP). The goal consists in finding policies that map the information available to the controller to a control input in such a way that a performance criterion, often defined in terms of costs, is optimized. Usually, using nonlinear filtering methods, this information is condensed into a probability distribution that represents the state estimate of the system to be controlled, and the policies map these distributions to control inputs.Unfortunately, DP is intractable except in a few very special cases. Therefore, approximate but tractable approaches are of interest. One such approach is the point-based value iteration algorithm, where each point is a probability distribution. In this approach, the controller maintains the optimal costs for a set of representative state estimates instead of trying the impossible task of maintaining the costs for all state estimates as it would be required in classical DP. Then, it uses this information in order to obtain an approximation of the optimal costs at a state estimate that is needed for decision making. As we see, point-based value iteration requires approximation methods for functions defined over general probability distributions. However, state-of-the-art approaches either restrict the class of possible state estimates or assume finite sets of control inputs and measurements. Although workarounds for continuous control inputs and measurements exist, they usually require additional approximations. For this reason, we propose a novel approach to stochastic control of nonlinear dynamical systems with continuous states, control inputs, and measurements that is based on Gaussian Process (GP) regression. Classical GP regression only allows for deterministic vector-valued inputs. For this reason, we propose a novel extension of the GP framework to inputs given in form of probability distributions. By doing so, we extend the GP framework to infinite-dimensional inputs. Our approach is based on the idea to define the covariance functions that determine the GP in terms of the distance between the probability distributions provided as inputs to the GP.In the course of the project, we plan to develop a solid framework for GPs defined over general probability distributions and to derive stochastic control algorithms that use such GPs to compute the policy. We believe that the proposed project will substantially contribute to research on stochastic control. Furthermore, the presented idea for defining GPs with inputs given in terms of probability distributions can also be used in machine learning research in order to derive other non-parametric Bayesian regression and classification methods over probability distributions.

• DFG Programme: Research Grants