Evaluating Collaboration experience in Spatial VR

Created by Dave Song Spring 2023

The purpose of Project 2 was to explore and develop an evaluation tool for VR collaboration software. Inspired by the experiment design described in the work of Wu, Y., Wang, Y., Jung, S., Hoermann, S., & Lindeman, R. W., rounds of charades were utilized as a part of the in-class activity to evaluate collaborative features and experience quality of Spatial VR.

The class was divided into groups of two. For each group, participants were asked to create SpatialVR accounts and join the shared “classroom" created using Spatial VR’s server. After all participants successfully joined the virtual classroom, they were asked to perform three tasks.

In-person charades
VR charades using Spatial VR
Collaborative SARS-Cov-2 Analysis.

In-Person Charades

During each round of charades, each participant was asked to describe 5 words that were sent via email before the activity time. They were asked to describe the given words without speaking — ideally communicating their idea and information through pen and paper or non-verbal communication. For the evaluation purpose, the time taken for each word was recorded. After the in-person charades activity, the In-person collaboration and communication evaluation form was filled out by the participants.

VR-Charades

After providing some time for the users to explore the VR environment and to use some of the tools and features in the Spatial VR environment, participants were asked again to perform another round of charades with different words. Similar to the first task, the participants recorded time and minimized verbal communication between partners. The main reason why the experiment strongly encouraged the participants to minimize the amount of verbal communication was to evaluate the efficacy of non-verbal communication quality since verbal communication can be easily achieved by simple audio communication functionality.

VR SARS-CoV-2 Analysis

After the participants completed the first two tasks, SpatialVR’s collaborative tools were also evaluated by asking the participants to analyze Sars-Cov-2 protein structures. Specifically, each group was asked to identify RNA, 5 Envelope proteins, 3 membrane proteins, and 7 spike proteins. Additionally, the participants were asked to annotate the structural proteins with a medium of their choice.

After the two VR tasks, the participants were asked to fill out evaluation feedback.

Results

In-Person vs. VR Charades

Average Time per round

In-person: approximately 67.3 seconds
Spatial VR: 202.52 seconds

Insights: generally, in in-person and spatial VR, participants displayed somewhat similar time required to guess the words correctly with few exception words where the team took significantly longer than other words. The charade words used in the experiment attempted to control the difficulty of the provided words by deliberately choosing them from the “medium” and “difficult” levels from The Game Gal.

When evaluating collaboration quality, it was important for the organizer to get a general sense of how demanding the actual in-person collaboration can be compared to the task load of the vr system in interest. Therefore, the post-activity questionnaires first evaluated the task load of in-person and Spatial VR collaboration.

Task Load Evaluation of In-person vs. Spatial VR collaboration

For Task Load Comparison, participants were asked to answer questions regarding task load. Used Likert Scale from 1 - 10 where 1 = Not at all and 10 = Very

*graphs on the right column represents answers regarding Spatial VR experience

Mental Demand:

On average, participants agreed that VR collaboration was more mentally demanding the one in-person.

Physical Demand:

On average, participants gave 4.43 for in-person and 6 for VR. Additionally, the distribution of the answers tells shows that the participants, overall, agree on the fact that VR collaboration requires similar or more physical effort.

Temporal Demand:

As displayed in the two graphs, the participants felt more hurries or rushed when using Spatial VR to collaborate. This is largely due to time constraints the class had for in-class activity and the fact that VR collaboration took place near the end of the class time.

Performance Evaluation:

Performance survey results were interesting since more participants felt they were more successful in accomplishing the goal of the tasks. On average, participants gave 4 for in-person collaboration and 5.67 for VR collaboration. This can be explained by individual's expectations of VR collaboration, Spatial VR's collaborative tools, and the fact that VR collaboration took place after in-person collaboration.

Overall Difficulty:

However, participants felt VR collaboration as more demanding option.

Frustration:

Qualitative Feedback/Evaluation:

Besides the quantitative evaluation, participants also shared some of the challenges they noticed when collaborating in VR. Although this list is not complete because the number of participants for the evaluation was only 12, the list below served as a foundation when developing the VR collaboration evaluation rubric.

lack of dexterity when it comes to portraying things
Lack of detailed hand tracking
Motion detection malfunction
Full Body motion capture

Usability Evaluation:

Besides task load, participants also evaluated the VR system in terms of usability.

In general, users found the system somewhat complex and hard to use mainly due to:

lack of accurate full body motion detection
hand dexterity

However, most users also highlighted that the system was easy to user with features:

Sticky Notes
Reactions
3D pen
Overall interaction design

Participants also provided individuals' definition of an "ideal collaboration experience." From the evaluation and feedback, key attributes are:

clear communication
tools without bugs and glitches
Full body tracking
emotion expression
realism
environment where people can measure and analyze data in real time
hand and finger tracking

Tools and Modes of Communication

Participants also identified some of the tools in Spatial VR that they utilized and enjoyed using:

3D Pen
Sticky Notes
Avatar Emoji

Conclusion

Overall, SpatialVR and its collaborative features generally supported smooth collaboration when it comes to non-verbal communication and scientific rendering analysis. However, the major issue with the software was the lack of full body and finger motion tracking. 3D pens and sticky notes where users can share their ideas in a more conventional way in VR space were utilized. One of the key highlights from the activity and feedback analysis is the higher value of performance given to the VR system over real-life collaboration. While future work will be focusing on if the performance differences exist in controlled experiments.

Notes on body and motion tracking provided enough feedback and served as a great foundation when researching and developing evaluation rubrics for the VR collaboration experience. One thing I realized as I learn more about VR collaboration is that realism is at the center of collaboration experience quality. Knowing that human interaction includes more than just highly expressive rendering motion tracking, towards the end of the research phase, I became more interested in the potential of an AR collaborative system where collaborators are co-located in a traditional collaboration manner with assist or augmented visualization of the dataset of renderings. Future work will explore performance, task load, and social presence differences in AR and VR collaborative systems.

Page updated

Google Sites

Report abuse