Visual Segmentation and Labeling of Multivariate Time Series
Final Report Abstract
The goal of VISSECT was to develop new approaches for visually and interactively supporting the segmentation and labeling of multivariate time series (MVTS) through segmentation and labeling. The segmentation and labeling of MVTS is an important task in many domains. In the medical domain, algorithms that segment and label the important sequences depending on interdependent and time-dependent sensor variables can support a more sophisticated analysis and visualization of electrocardiographs. Similarly, data from human motion tracking is very time-consuming if not supported by algorithms that can segment, label, and filter out specific movements before visualizing the situation to the user for further analysis. All MVTS examples have in common that it is challenging to select and parametrize algorithms for an adequate segmentation and labeling and to compare the produced segmentation results. Furthermore, the results of segmentation and labeling are typically uncertain in various ways. For example, it is difficult to pinpoint the exact time point when a tracked human starts raising her arms or starts a rotation. Inaccurate sensor data or the segmentation algorithm itself can also induce uncertainty in the analysis process. The project team of this German-Austrian collaboration has therefore targeted three particular challenges in the process of segmenting and labeling MVTS: the algorithm selection (TU Darmstadt), the parametrization of algorithms (Universität Rostock), and the uncertainty analysis (TU Wien). The innovative approach of VISSECT was to combine these three challenges within one joint research approach. Accordingly, a major result of VISSECT is the definition of an integrated segmentation and labeling pipeline for MVTS. This pipeline was established based on four design goals. First, the pipeline is general and can be applied to various use cases and application domains. Second, it supports the definition of individual algorithmic routines specific for individual data, users, and tasks. Third, parameters are disclosed and can be defined externally, e.g., initiated from a visual analysis environment. Fourth, VISSECT explicitly incorporated concepts to systematically record and propagate uncertainty information with the algorithmic routines and segmentation results. Overall, this pipeline makes the huge design space and the number of possible configurations of segmentation pipelines (and thus different segmentation results of various uncertainty), but also the target points for new solutions for the analysis of MVTS, more apparent and manageable. Based on the overarching insights, VISSECT also pursued each of the three particular research foci. First, the project strongly advanced the research on algorithm selection, on the interactive coordination of these algorithms, and user workflows for the segmentation and labeling pipeline. It is now possible to visually and interactively create segmentation and labeling pipelines and choose various supervised and unsupervised algorithms that can be analyzed with the technologies developed by VISSECT. This includes several visualizations for single and multiple segmentation and labeling results. Second, the project also developed a sophisticated approach for examining parameter settings. The resulting correlation calculation is very flexible and considers arbitrary subspaces of the parameter space. The user can switch the way of how to compute the correlations, apply different sorting mechanisms, and utilize different analysis strategies. In particular, it is now possible to estimate the parameter influence on a subrange level to support the sampling of the parameter space. Third, pursuing the challenge on uncertainty VISSECT identified several insights. It is now possible to externalize uncertainty from pre-processing (and subsequently data quality), a previous gap in the research on MVTS. This first step led to a better quantification and evaluation of various sources of uncertainty: value, result, aggregation, and cause & effect uncertainty, a clear structure for future research on uncertainty of MVTS. This also helps in understanding how different sources of uncertainty influence the SL pipeline and the uncertainty visualization. Overall, VISSECT was able to demonstrate significant advances in each of the three challenges of algorithm selection, parametrization of SLA, and uncertainty. These advances were only possible through the approach of these challenges in this joint project. The new segmentation and labeling pipeline for MVTS also helps in structuring and categorizing the various past contributions in this field of research. It is furthermore the basis for a joint reference system to demonstrate the combination of all three aspects of VISSECT. All the results have been published at renowned conferences and workshops, and are available to the wider VA community. The three partners at TU Darmstadt, Universität Rostock, and TU Wien are continuing their research based on the VISSECT results in new proposals and initiatives. This includes a project in the medical domain to segment and label extremely long MVTS (> 50 years) in chronic diseases.
Publications
- Cycle Plot Revisited: Multivariate Outlier Detection Using a Distance-Based Abstraction. In: Computer Graphics Forum 36.3, pp. 227-238, 2017
Bögl, M., Filzmoser, P., Gschwandtner, T., Lammarsch, T., Leite, R. A., Miksch, S., and Rind, A.
(See online at https://doi.org/10.1111/cgf.13182) - Interactive Lenses for Visualization: An Extended Survey. Computer Graphics Forum, Vol. 36, No. 6, 2017
C. Tominski, S. Gladisch, U. Kister, R. Dachselt, and H. Schumann
(See online at https://doi.org/10.1111/cgf.12871) - Visual-Interactive Semi- Supervised Labeling of Human Motion Capture Data, In Proceedings of Visualization and Data Analysis (VDA 2017), 34-45, San Francisco, CA, USA, 2017
Bernard J., Dobermann E., Vögele A., Krüger B., Kohlhammer J., Fellner D.W.
(See online at https://doi.org/10.2352/ISSN.2470-1173.2017.1.VDA-387) - Combining the Automated Segmentation and Visual Analysis of Multivariate Time Series. EuroVis Workshop on Visual Analytics (EuroVA), Brno, Czech, 2018
Bernard, J.; Bors, C.; Bögl, M.; Eichner, C.; Gschwandtner, T.; Miksch, S.; Schumann, H. & Kohlhammer, J.
- Quantifying Uncertainty in Time Series Data Processing, Vis-In-Practice Symposium, IEEE VIS, Berlin, Germany, 2018
Bors, C., Bögl, M., Gschwandtner, T., Bernard, J., Miksch, S.
- Sketching Temporal Uncertainty - An Exploratory User Study, 20th EG/VGTC EuroVis Conference on Visualization. The Eurographics Association, pp. 67-71, 2018
Schwarzinger, F., Roschal, A., and Gschwandtner, T.
(See online at https://doi.org/10.2312/eurovisshort.20181080) - Quantifying Uncertainty in Multivariate Time Series Pre-Processing, In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA), Porto, Portugal, 2019
Bors C., Bernard J., Bögl M., Gschwandtner T., Kohlhammer J., Miksch S.
(See online at https://doi.org/10.2312/eurova.20191121) - Visual‐Interactive Preprocessing of Multi-variate Time Series Data, In: Computer Graphics Forum 38.3, pp. 401-412, 2019
Bernard J., Hutter M., Reinemuth H., Pfeifer H., Bors C., Kohlhammer J.
(See online at https://doi.org/10.1111/cgf.13698) - Making Parameter Dependencies of Time-Series Segmentation Visually Understandable. In: Computer Graphics Forum 39.1, pp. 607-622, 2020
Eichner, C.; Schumann, H. & Tominski, C.
(See online at https://doi.org/10.1111/cgf.13894)
