Advanced Evaluation of VR collaboration

Spring 2023, Dave Song

Towards Advanced Evaluation of Collaborative XR spaces (2022)

Motivation: After reading several VR research papers and more on the evaluation of VR/AR collaboration systems, it became clear to me that there are a set of measuring and evaluation methods that are utilized commonly. Interested to find more non-intrusive ways of evaluating natural collaboration in the xr environment, I decided to learn more about advanced evaluation methods proposed last year.

<Key Ideas>

• practical adoption of XR technologies → real-world evaluation methods need to be developed to measure those VR and AR systems in real-world settings

• Proposes potential list of wearable-based data measurement for VR and AR system evaluation

<Evaluating XR Environments>

• Evaluated via surveys, observation, and performance measures

• However, these methods are not applicable when we try to measure the performance of the systems in real-world settings

• distortion of the evaluation

  • primacy and recency effects

    • “the tendency to remember first and last items better while recalling a previously presented list of items”

  • Can be prevented by presenting questionnaires right after the single experiment.

    • However, it decreases the immersive experience

  • Also, rating and measures done by participants can be unreliable considering individuals have a tendency to rate too low or too high

  • Performance review → not applicable for exploratory tasks

• wearables that can support data collection in real-world situations without interruption to the users.
  
  

→ different approach to measure the user acceptance of the system by focusing more on

• temporal, cognitive, and physical effort.

  • the cognitive effort of a user

    • pupil diameter

      • Proposed with an index of pupillary activity. Focus on detecting abrupt and sudden changes in pupil size to avoid noise from dilation from light sources.

    • heart rate variability

      • Can be obtained using Photoplethysmography(PPG) sensors in the form wristbands

  • physical effort

    • muscle activity and physical stress could be measured by electromyography(EMG)

  • Flow

    • Theory of Flow → XR system should not minimize effort to zero, but match effort required to produce the optimal flow and task performance

  • Task Technology Fit

Source: https://link.springer.com/chapter/10.1007/978-3-030-98388-8_40