Evaluating the Quality of Experience of VR Applications in a Charade Game Environment

Polina Orhunova, Wenhan Tao, and Mylene Farias

The popularity of applications in augmented reality (AR) and virtual reality (VR) has increased considerably in the past decade [1], [2] with the creation of new imaging technologies and the development of compact and affordable head-mounted displays (HMDs). Current devices offer better motion tracking, higher resolution displays, better graphics, lower latency, and higher refresh rates. However, the representation of immersive content requires large volumes of data. Therefore, data streaming and viewing of this type of data requires high computing power and fast Internet, leading to compromises that can reduce the quality, comfort, and acceptance of VR applications and services.

The research is part of the collaboration with the Immersive Media Group of the Video Quality Expert Group (VQEG). More specifically, we are part of the VQEG Immersive Media Test that includes test paradigms to estimate several aspects of quality of experience (QoE) in eXtended Reality (XR) communications. The main objective of the studies performed in this Test Plan is to answer the following research question: How do changes in system factors (such as device and application differences) affect user’s experiences of audio-visual content and the performance of communication tasks? To answer this question, we are performing a series of experiments focusing on the visual quality of VR communication applications. In our first experiment, conducted in the fall of 2024, participants played a charade game using either Horizon Workrooms (a VR platform) or Microsoft Teams (a communication benchmark). Figure 1 shows a screenshot of the Horizon experiment, showing the virtual environment and the two players using the HMDs. We used a between-subject protocol, with half of the participants playing the charade game in Horizons using the Meta Quest 2 HMD and the other half playing charade in Microsoft Teams using an LCD display.

Figure 1. Experimental setup: VR environment and participant interaction. (Photo courtesy of Polina Orhunova, Wenhan Tao, and Mylene Farias.)

The experiment was divided into five stages: pre-questionnaire, instructions, training, main session, and post-questionnaire. From the pre-questionnaire, we collected personal and health data, the last being collected using the simulator sickness questionnaire (SSQ) [3] that is a standard method used to assess any nausea, discomfort, and disorientation that may be caused by the VR environment. During the instructions, the experimenter explained the experimental task (the charade game) to the participants, while during the training, the experimenter explained how the devices worked and performed a mock experimental session. From the post-questionnaire, we again collected health data with the SSQ, but also mental workload with the NASA-TLX questionnaire [4] and social interaction and presence.

During the main session, the participants played a charade game, which is a game in which participants act out a word or phrase without speaking, typically using gestures, facial expressions, and body movements. The aim is for other players to guess the word or phrase correctly within a time limit. It is a popular party game that encourages creativity, communication, and quick thinking. In the experiment, two participants played, one of the players acting out the words (participant A) and the other trying to guess the words (participant B).

We are currently analyzing the collected data from the pre- and post-questionnaires, comparing factors like comfort, presence, task load, and quality of interaction to evaluate if the VR (Horizons) and non-VR (MS-Teams) platforms influence the user experience. We will also analyze the overall performance in the Charade task using the metric: guessed words/minute. Finally, in the near future we will perform two additional experiments where other VR platforms will be analyzed.

References

1. A. Osborne et al., “Being social in VR meetings: A landscape analysis of current tools,” in Proc. ACM Designing Interact. Syst. Conf., Jul. 2023, pp. 1789–1809.
2. N. Wenk et al., “Effect of immersive visualization technologies on cognitive load, motivation, usability, and embodiment,” Virtual Reality, vol. 27, no. 1, pp. 307–331, Mar. 2023.
3. A. Cao et al., “NASA TLX: Software for assessing subjective mental workload,” Behav. Res. Methods, vol. 41, no. 1, pp. 113–117, Feb. 2009.
4. R. S. Kennedy et al., “Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness,” Int. J. Aviation Psychol., vol. 3, no. 3, pp. 203–220, Jul. 1993.