Self-adjusting Model-based Processing of Declarative Forecast Queries in Data-Warehouse-Systems

• Applicant: Professor Dr.-Ing. Wolfgang Lehner
• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: from 2009 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 114523986

Forecasting of time series data is crucial for decision-making processes in many domains. Within the DFG project FFQ (Flash-Foreward Query Framework) we developed a concept for integrating automatic time series forecasting in relational database management systems. In contrast to flash-back queries, which allow a view on the data in the past, we provide declarative flash-forward queries, which allow a view on the data in the future. Complex internals like automatic model selection, usage and maintenance are hidden from the database user. The automatic and transparent processing of such forecast queries is based on a similarity-based model selection approach, logical ensemble models and a configuration advisor, which selects a configuration of forecast models for multi-dimensional data sets and stores it in a model pool.Due to the large search space, long parameter estimation runtime and possible large data sets in typical data warehouse environments, the maintenance of such model configurations exhibits a major challenge. Synchronous maintenance of all models and configurations after each new insert is infeasible and might not even be necessary if the model accuracy is still sufficient or if a model is not referenced by a query. Asynchronous maintenance techniques were already applied for materialized views and usually delay maintenance to free cycles or until a view is referenced by a query. However, in contrast to materialized views, forecast models always produce estimated results and, thus, provide more freedom but also more challenges in model maintenance. On the one hand, maintenance can be delayed or even omitted if the accuracy is still sufficient (depending on the requirements of a forecast query). On the other hand, speculative maintenance might provide even better solutions and lead to higher accuracy. In this follow-up project we aim to develop a flexible scheduling mechanism for maintaining forecast model configurations, which allows high accuracy for arbitrary forecast queries without affecting query runtime. Maintenance tasks have to be scheduled based on data characteristics, model accuracy, the query workload, and available resources. However, efficient maintenance is only possible if the underlying hardware environment is optimal used and parallelization opportunities are exploited. This requires techniques for joint execution and parallelization of maintenance tasks and novel cost models that enable the generation of maintenance jobs and the distribution of resources.

• DFG Programme: Research Grants