Subjective measurement and instrumental estimation of mobile online gaming quality based on perceptual dimensions
Electronic Semiconductors, Components and Circuits, Integrated Systems, Sensor Technology, Theoretical Electrical Engineering
Final Report Abstract
Due to the DFG-founded project, for the first time a model predicting the QoE of cloud gaming services was developed and submitted to the ITU-T. Such a quality prediction model could only be achieved due to our fundamental research on a great number of influence factors as well as research about the assessment of gaming QoE. The project objectives are in line with three work items at ITU-T, where due to the project, ten ITU Contributions were submitted of which two resulted in openly available recommendations. These recommendations, ITU-T Rec. G.1032 about influencing factors and P.809 about subjective methods, offer the research community guidelines to conduct subjective tests assessing gaming QoE which will lead to more comparable research in this domain. Based on many subjective tests we can conclude that in respect to user factors, a player’s gaming skills, preferences and normally used systems are of high importance. Regarding the system factors, network and encoding parameters but also the game content has a high impact on gaming QoE, whereas the playing device and display have a rather low impact, if the playability (i.e., gaming usability) is not harmed substantially. The investigated environment factors did not show any significant impact. Based on these finding we argue that the developed model is, in its scope, usable for mobile and traditional cloud gaming. While the ITU Recommendation are helpful to ensure reliable and valid results from subjective tests, it is important to cover a broad range of quality ratings and investigated parameters to develop a prediction model. Therefore, we also applied our work on pseudo-objective video quality metrics for the selection of parameter values, especially for bitrates. A parametrization of the game content remains an important and challenging research topic. However, we identified many relevant game characteristics and used experts to categorize games in respect to their delay sensitivity, frame loss sensitivity, and encoding complexity. These classifications improved the performance of the prediction model significantly. With respect to quality dimensions of gaming QoE, based on our explorative research carried out, we can conclude that we could not find important aspects that are missing in the taxonomy, which was used as a starting point. The project led to the ITU-T Rec. P.809, which gives guidelines for the assessment of many aspects of the taxonomy, including details about test paradigms, test structures, participant instructions, and stimulus selection. However, there are also limitations: a) it is not possible to assess all aspects of the taxonomy with a single questionnaire, b) many influence factors such as social interactions or game preferences cannot be influenced by service providers, and c) there is not just one ideal questionnaire for the assessment of gaming QoE, as the construct is highly multidimensional. Nevertheless, we proposed a compact composition of short questionnaires to assess many relevant player aspects but also quality dimensions. The questionnaire is described in detail in ITU-T C.293 and contains also a questionnaire assessing the input quality (consisting of the sub-dimensions controllability, immediate feedback, and responsiveness), which we developed using the method of crowdsourcing and suitable factor analyses. An ITU-T Recommendation for the opinion model G.OMG is foreseen to be completed in the upcoming SG-12 meeting in November 2019. Agreement on its structure and implementation based on its good performance (RMSE of 0.292 and PLCC of 0.94) was already reached during the SG-12 meeting in May 2019. The model is composed of impairment factors related to the video transmission and compression, as well as impairments on the input quality due to network parameters and framerates. Whereas the player experience dimensions can also explain the gaming QoE well, a prediction of concepts such as immersion did not turn out to be very accurate as arguably these concepts are strongly influence by user factors or system factors which are not controllable by network and service providers. For the latter, the developed model can be of great use for deriving ideal resource allocations and to design appropriate networks to offer a satisfying cloud gaming experience to their customers. In the future, an extension of the model for multiplayer, online, and VR gaming including higher resolutions and framerates, would be an interesting research. Also the development of a monitoring model and more research on the method of crowdsourcing for gaming research would be of high interest.
Publications
-
(2017). Towards the delay sensitivity of games: There is more than genres. 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1–6
Schmidt, S. and Zadtootaghaj, S. and Möller, S.
-
(2018). A Classification of Video Games based on Game Characteristics linked to Video Coding Complexity. 16th Annual Workshop on Network and Systems Support for Games (NetGames). IEEE, 1–6 (Best Paper Award)
Zadtootaghaj, S. and Schmidt, S. and Barman, N. and Möller, S. and Martini, M. G.
-
(2018). A Comparison of Interactive and Passive Quality Assessment for Gaming Research. 11th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1–6
Schmidt, S. and Zadtootaghaj, S. and Möller, S.
-
(2018). GamingVideoSET: A Dataset for Gaming Video Streaming Applications. 16th Annual Workshop on Network and Systems Support for Games (NetGames). IEEE, 1–6
Barman, N. and Schmidt, S. and Zadtootaghaj, S. and Martini, M. G. and Möller, S.
-
(2018). Modeling Gaming QoE: Towards the Impact of Frame Rate and Bit Rate on Cloud Gaming. 11th International Conference on Quality of Multimedia Experience (QoMEX). IEEE
Zadtootaghaj, S. and Schmidt, S. and Möller, S.
-
(2018). New ITU-T Standards for Gaming QoE Evaluation and Management. 11th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1–6
Möller, S. and S., Steven and Zadtootaghaj, S.
