2018 Research Findings

The "2018 Research Findings" page was written to design a user study on currently available VR software, hardware, and media. We gathered data to understand where to improve in terms of getting people interested and excited about VR and to help formulate an assessment of currently available VR software, hardware, and media. We documented our process and results here. We also summarized our findings in a Medium article.

Table of contents

I. Audience / Samples

II. Questions

III. Activities

IV. Notes from David Whitney

V. Ideas for Data Representation

VI. Workshop Details

VII. Workshop Details (cont.)

VIII. Data Representation (Brainstorming)

IX. Data Consolidation

X. VR@Brown Diagrams

Progress Report

I. Audience/ Samples

@Linda + Ankita

• (No/Beginner/Intermediate) CS background

• Brown students

• High Schoolers

• Professors/”older folks” 

II. Questions

@Linda + Ankita

• Background

  • Are you a CS concentrator?

  • What do you study?

  • Have you had any experience with VR before?

• Awareness

  • Define VR

  • List any VR softwares or hardwares (i.e. devices) that you know of.

  • What do you think VR can be used for? What are its current applications?

  • Do you think VR will become more popular in the future? Will more people be aware? Vs. VR being a fad...

  • Can you identify a problem that could be solved using VR?

  • Any words you would associate VR with? (Use word cloud p.6 of Google’s white paper linked below)

• Experience

  • How did you feel? (excited, nauseous, etc.)

  • If you have no experience with VR, do you know why?

  • If you have experience, what got you interested in VR?

    • How did you hear about it?

    • What have you done with VR?

• Accessibility

  • Was there any difficulty in the process of setting up?

  • Is there anywhere near you you can rent the device?

  • How expensive is it?

• Difficulty

  • Is it intuitive?

  • Do you think that the system easily does what you want it to do?

• Enjoyment

  • Tell us about the process.

• Interest

  • What is your field of interest or subject?

  • Favorite applications using VR?

• Retention

  • How likely would you do this again?

  • If we were to hold another workshop, would you attend?

• Time/Availability

  • How long does it take you to setup?

  • Would you be normally available to try VR in any given normal day?

• Perception

  • What do you think of VR in the world right now?

  • Do you believe it to be (male-dominant, difficult, etc.)?

• Evaluation

  • If you could improve anything related to your VR experience, what would you change about it?

  • Why might be an entry barrier to getting involved with VR?

  • What was your favorite VR technology/activity?

  • What was your least favorite VR technology/activity?

  • Changes to user interface to improve performance!

• [Brown Specific]

  • Are you aware of the Cave in Granoff?

  • Are you aware of the YURT?

  • Do you know where to find an HTC Vive?

  • Where did you hear about VR?

  • Have you ever considered taking VR classes?

  • Are you taking VR classes? For what purpose? (Interest, Requirement, etc.)

  • Were you aware that Brown CS has research positions and classes available for VR?

III. Activities

• Hardware - @Ankita

  • Setting up an HTC vive?

  • Playing a game without any instruction? Using the oculus/hololens/vive

• Software - @Linda

  • Google WebVR/ A-Frame mini tutorials/Blender

• Installation - @Ankita

  • Unity, minVR, OpenGL/Blender

• “I’m running into so and so problem” - How would you fix it? - @Linda

  • Evaluation of troubleshooting documentation

• Can you tell me what these various applications are used for? Time limit. - @Linda

• Media Lab Experiment - @Ankita

  • Flesh & Sand, NYT apps

  • https://drive.google.com/file/d/1D1f9Yl_7sihmy9qL-ww1aB4HAM4nSihf/view?usp=sharing

IV. Notes from David Whitney

@Linda + Ankita

Process:

• IRB review

  • Confirm study is ethical

  • Takes about 10 days

  • They typically send revision suggestion

  • Talk to David!

• Worked with psychologist in HCRI (8th floor of scili)

Type of Study

• Within: when everyone does every part of interface/modality 

• Works for less people? 30 people is a good number

• Remember to randomize order to reduce bias

Recruiting

• Posters in several places in CIT, faunce

• Emails (listserv or Today@Brown)

• Offered money for incentive

Relevant Info Taken into Consideration

• Reported users’ age (Year in school?)

• Background info about their experience with VR

• Gender; make the response text area

• Did you have corrected normal vision? (Contacts, glasses, color vision issues i.e. I am color blind: yes or no) 

• In your prime, how many video games did you play?

Objective Measurements

• Time to complete task

• Accuracy of completion

  • Time to quickly hit the green/red button

  • Measures how many questions they get right/wrong 

  • Different types: true positives are things that are true and were labeled true; true-neg was true but labeled false. False positives; false negatives likewise. https://en.wikipedia.org/wiki/Precision_and_recall

• Resources

  • NasaTLX https://en.wikipedia.org/wiki/NASA-TLX#/media/File:NasaTLX.png (way of measuring how difficult a task is

  • (SUS) system usability score https://www.measuringux.com/SUS.pdf

    • http://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf

Looking Back

• Randomization 

  • Make sure to have equal sizes for both groups.

  • Having a larger sample size (changed from 18 to 32). 

  • Recruit cap sized people for diversity (e.g. 5 people)

Representation of Data

• Graphs

  • (i.e. bars, cat and whisker style plots, tables, pie charts). 

• Differences in time and accuracy. 

Evaluation/Forms of Assessment/Questions

• Google form to fill out subjective questions in the end

  • highly suggested to have them digitized in a form! 

  • Most of the questions were likert scale https://www.simplypsychology.org/likert-scale.html 

    • rank / rate in a range

  • Did you have experience with VR; Scale of 1-7 how much do you agree with this statement - 

  • Which of these did you prefer or enjoy the most? 

Location of Findings

• In “Discussion or Analysis” mention anecdotes you got from free-response answers to open-ended questions

  • Evidence of an explanation (this condition didn’t perform as well as it would. Our hypothesis was...our results suggest that…users reported…in such and such evaluation...)

Accessibility

• Ask users if they are color blind, have had any hand injuries

• Take into account hand sizes/head sizes

V. Ideas for Data Representation

@Linda + Ankita

VI. Workshop Details

@Linda + Ankita

All this info is on this doc! 

Logistics

• 30 people - 6 groups of 5 people!

• Science Center

• Tentative Date - 5/7

  • 2 hours: all groups do all 6 activities

  • 1 hour: randomly assign 3 activities to each group

TODO

• Should get the required approvals for user study soon!

• Look at VR Media doc

  • Obtain Google Cardboards

• Book Science Center 

• Ask for volunteers for this workshop/setup

• We should get fooood!

• Start putting together comprehensive surveys for each activity

  • Put them in a place that people can access

• Sign-up sheet:

  • Concentration

  • School

  • Year

  • Experience with VR

  • Allergies

  • Email

• People should know what groups they are in before they come to the workshop

• Schedule a dry run -- make sure this is a feasible workshop in terms of time!!! 

  • Confirm locations for each activity

  • We should do each activity and come up with final documentation/instructions for each activity

  • Make sure people have enough time to do each survey after each activity well

Available Tutorials

• Hack@Brown A-Frame - located in Home/Workshops

• Hello World Google Cardboard in Unity - located in Home/Tutorials

• Blender (might be too much time)

Advertising

• Facebook Event

  • Hack@Brown + Mosaic collaboration?

  • Need a fun facebook banner

• Today@Brown

Activities

• Understand the people we are studying: 15 min + Fun Music :) + Food!!!

  • Time where they fill out a quick survey telling us about themselves

• Software: 30 min - 3 volunteers + 2 instructors for each

  • Hello World Google Cardboard in Unity

  • VR scene in web browser using A-Frame

• Hardware: 50 min - each person would get 10 min - 2 volunteers (or hopefully 30 min if 2 people can go at a time/ we use more than one vive)

  • Play a game without any instruction

    • Vive - Graphics Room! + Robotics Room!

    • Would be awesome if we could have two people go at a time!

• Troubleshooting: 15 min - 3 min for each problem!

  • Identify a few (5) common problems

    • We can crowdsource for this

• What are these applications used for?: 15 min - 5 min for each application

  • Unity

  • Blender

  • Vive

• Media Lab Experiment: 15 min - 5 min for setup, 10 min to watch the film

  • https://drive.google.com/file/d/1D1f9Yl_7sihmy9qL-ww1aB4HAM4nSihf/view?usp=sharing

  • We should try to have all of this set up for them to just pick up and use!

Format

• Hardware + Activity below it can be paired together

VII. Workshop Details (cont.)

@Linda + Ankita

Linda and I felt it would be easier to organize our thoughts in a Google Drive folder. 

VIII. Data Representation (Brainstorming)

@Linda + Ankita

1. Understand the people we are studying

• Pi chart: Background, School, Year, Gender, Age

• Bar graph: Granoff, YURT

• Pi chart: How many of our participants that go to Brown considered or are taking VR classes? 

---

• Workshop demographics: what schools did our participants come from, age, etc.

---

• Score participants on how “perfect” they are for VR (to calculate ideal user of VR)

  • Scale, from least likely to enjoy VR (has hand injuries, wears glasses etc.)

---

• Word cloud representing textual data (Awareness)

2. Software 

• Pi chart: Experience with Unity and A-Frame

• Word cloud: frustrations, things that were not intuitive with software - also helpful for findings

• “How easy was it for people to use this software?”

• Pi chart: Retention - would people use this software again - how great was the experience 

3. Hardware

• Pi chart: Experience with HTC Vive

• How easy was it to use? (Most participants found it to be...)

• Word cloud: frustrations

• Overall experience - helpful for findings -- how people approach VR, what are people trying to achieve

  • How successful were you in accomplishing what you were asked to do

• Retention - would people use the Vive again

• How easy is it to use?

  • How many people found it complicated to use

  • How many people imagine that it is quick to use

• Do people feel safe?

4. Troubleshooting 

• How easy was it for people to resolve commonly encountered problems

• What is the common pattern people take to fix the problems

• Word cloud to describe the process of troubleshooting

• Suggestions on improvement to enhance experience overall

5. What are these applications used for 

• Was the purpose/typical use clear? (bar graph)

• Would you consider using this for the field you’re interested in? (bar graph)

• Why do you think you were unaware of the use of each app? (word cloud)

• How should the industry improve to make it more accessible for people? (word cloud)

6. Media Lab Experiment

• How did you feel & would you do this again (x,y axis with x = yes/no and y = range of emotions from bad to good)

• How would you improve this experience (word cloud)

• Did you feel like you were physically there, was it comfortable, similar to viewing a documentary film (bar graph)

• Journalist, theater spectator images and showing which ones people are more leaning towards (maybe a scale and weights labeled each?)

IX. Data Consolidation

@Linda + Ankita

We also summarized our results here :)

X. VR@Brown Diagrams